Pass in an array of data [bulk insert] using an insert statement

Hi,

How can I pass in an array of 1,000 elements and have pg do a bulk insert into the database?

I am trying to solve this problem brought up here, but using postgres instead

var sql = 'insert into '+table+
    ' (Month, Merchant_Id, Merchant_Descriptor, Network, Qualification_Code, Transaction_Type, '+ 
    ' Issuer_Type, Card_Type, Txn_Count, Txn_Amount, ' +
    'Interchange) values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)';

psql.connect();
reduceArray(large_array, function(small_array){
    db(small_array);
});

var reduceArray = function(arr, cb){
    var size = 1000,
     small_array = []
     ;

    for (var i=0; i<arr.length; i+=size){
        var chunk = arr.slice(i,i+size);
        small_array.push(chunk);
    }
    cb(small_array);
};

db insert:

var db = function(small_array){
    async.times(small_array.length, function(n, next){
        var data = small_array[n]; 
            psql.query(sql, [data], function(err,result){
                next(err, result);
            });
    }, function(err){
        console.log(err);
    });
};

[brianc]

Yeah if you really need maximum speed you can string concatenate all the values into a single query - but absolutely do not do that for untrusted input. String concatenation is 99% of the time bad, bad, bad for sql.

You could also look at https://github.com/brianc/node-pg-copy-streams but it also does not sanitize the input. It doesn’t suffer from the same sql injection attacks because it doesn’t use a sql statement to do inserts, but there could still be issues w/ malicious input.

On Tue, Oct 27, 2015 at 4:18 PM, John Skilbeck [e-mail] wrote:

Ok sounds good. Yeah, the way I did it was just using .map … I thought that this might create unnecessary overhead however (single insert statement for each record)…

Was thinking there might be savings if you could just have one insert statement with an array of records. An insert statement for each record also works though; was just trying to save fractions of a second.

data.map(function(item,index){ psql.query(sql, item, function(err,result){ if(err) console.log(err); }); });

— Reply to this email directly or view it on GitHub https://github.com/brianc/node-postgres/issues/880#issuecomment-151647004 .

[skilbjo]

Ok sounds good. Yeah, the way I did it was just using .map … I thought that this might create unnecessary overhead however (single insert statement for each record)…

Was thinking there might be savings if you could just have one insert statement with an array of records. An insert statement for each record also works though; was just trying to save fractions of a second.

data.map(function(item,index){
  psql.query(sql, item, function(err,result){
    if(err) console.log(err);
  });
});