Many records to JS TypedArray

Hi. I need fetch many (100k+) rows from table, in row only one column type - int4.

const result = query('SELECT "int4_value" FROM generate_series(1, 100001) int4_value');
const typedArray = Int32Array(result.rows.length);

// This over used memory and utilization proc
result.rows.forEach((row, index) => {
  typedArray[index] = row.int4_value;
});

I want a more effective solution. So as not to load data into the JS array, get the whole result as a ArrayBuffer

Something like this:

const result = query({
  text: 'SELECT "int4_value" FROM generate_series(1, 100001) int4_value',
  type: 'arrayBuffer',
});

const typedArray = Int32Array(result.arrayBuffer);

Is this possible?

[sehrope]

The Int32Array constructor treats each element in the Buffer as a value so you’ll get data.length elements (not 4-bytes at a time). I don’t think there’s a direct way to convert it without some extra module (would be really easy in C…). You can do it manually via:

const int32Array = new Int32Array(data.length / 4);
for(i=0;i<arr.length;i++) {
    int32Array[i] = data.readInt32BE(i * 4);
}
console.log(int32Array);

This outputs:

Int32Array [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]

Not sure how slow that’d be at scale though.

You’re better off allocating it once and reading the results via a cursor. That way it’ll scale out to any number of entries and node only needs to allocate memory for a fixed number of rows. Try something like this:

const pg = require('pg');
const Cursor = require('pg-cursor');

function readCursor(cursor, count) {
    return new Promise((resolve, reject) => {
        cursor.read(count, (err, rows, result) => {
            if (err) {
                return reject(err);
            }
            return resolve(rows);
        });
    });
}

async function main() {
    const NUM_ROWS = 250000;
    const CHUNK_SIZE = 1000;
    const started = Date.now();
    const client = new pg.Client(process.env.DATABASE_URL);
    await client.connect();
    try {
        const sql = 'SELECT x FROM generate_series(1, $1) x';
        const cursor = new Cursor(sql, [NUM_ROWS]);
        client.query(cursor);

        const array = new Int32Array(NUM_ROWS);
        let index = 0;
        do {
            const rows = await readCursor(cursor, CHUNK_SIZE);
            if (rows.length === 0) {
                break;
            }
            rows.forEach(row => {
                array[index++] = row.x;
            });
        } while (true);
        const elapsed = Date.now() - started;
        console.log('Elapsed=%s ms ; Last Entry=%s', elapsed, array[array.length - 1]);
    } finally {
        client.end();
    }
}

main()
    .then(() => {
        process.exit(0);
    })
    .catch((err) => {
        console.error('%s', err.stack);
        process.exit(1);
    });

I’m getting about .5 seconds to run that for 250K entries:

Elapsed=465 ms ; Last Entry=250000

You can play with NUM_ROWS and CHUNK_SIZE to see how it effects memory and speed.

[charmander]

I suspect that the problem is with offset or aligning bytes, but more than an hour of googling a working solution has not given.

The Int32Array constructor treats each element in the Buffer as a value so you’ll get data.length elements (not 4-bytes at a time).

.buffer is the backing ArrayBuffer, and .byteOffset and .length are used correctly. It’s the right approach, just for the wrong endianness.

Is the manual loop not better? There’s also the option of:

const data = res.rows[0].data;

data.swap32();

const int32Array = new Int32Array(
  data.buffer, data.byteOffset, data.length / Int32Array.BYTES_PER_ELEMENT
);