Multi statement bulk insert

I’ve been wondering what the best way to achieve a bulk insert as part of a multi statement transaction is.

I have 3 tables that I need to update all at once, and here is the SQL query I use for that.

INSERT INTO `FAB_BCGH` (`OCNO`, `CUSTNAME`, `STYLENO`, `STYLDESC`) VALUES(1, 'FATFACE', 'w2526', '12t1g13g')
on duplicate key update BCGUNIQ = LAST_INSERT_ID(BCGUNIQ);
SELECT LAST_INSERT_ID() INTO @last_inserted_id_var_1;
INSERT INTO FAB_BCGD(BCGUNIQ, ITEMCODE, ITEMDESC, TTLROLLGEN) VALUES (@last_inserted_id_var_1, '2i2ht12t', 'white', 1000)
on duplicate key update BCGLINE = LAST_INSERT_ID(BCGLINE), TTLROLLGEN = TTLROLLGEN + VALUES(TTLROLLGEN);
SELECT LAST_INSERT_ID() INTO @last_inserted_id_var_2;
INSERT INTO FAB_BCGR (`BCGUNIQ`, `BCGLINE`, `ROLLNO`, `RLENGTH`, `RWIDTH`, `PONO`) VALUES (@last_inserted_id_var_1, @last_inserted_id_var_2, '123456', 100, 50, '21t8h12h'), (@last_inserted_id_var_1, @last_inserted_id_var_2, '12367', 100, 50, '21t8h12h')
on duplicate key update BCGREVL = LAST_INSERT_ID(BCGREVL);
COMMIT;

As you can see, I update the third table with two sets of values. However, the insertion of those values depends on the @last_inserted_id_var_1 and @last_inserted_id_var_2 variables.

I tried to do the same with this library in the following format.

(Here nestedData is of the following format):

[ [ 79, 56, 1, 15, 'NJPO1603761' ],
  [ 79, 56, 2, 15, 'NJPO1603761' ],
  [ 79, 56, 3, 15, 'NJPO1603761' ],
  [ 79, 56, 4, 15, 'NJPO1603761' ] ]

connection.query(`
 BEGIN;
 INSERT INTO FAB_BCGH (OCNO, CUSTNAME, STYLENO, STYLDESC) VALUES(${mysql.escape(OC)}, ${mysql.escape(CustName)}, ${mysql.escape(StyleNo)}, ${mysql.escape(StyleDesc)})
      on duplicate key update BCGUNIQ = LAST_INSERT_ID(BCGUNIQ);
      SELECT LAST_INSERT_ID() INTO @last_inserted_id_var_1;
      INSERT INTO FAB_BCGD(BCGUNIQ, ITEMCODE, ITEMDESC, TTLROLLGEN) VALUES (@last_inserted_id_var_1, ${mysql.escape(ItemNo)}, ${mysql.escape(ItemDesc)}, ${mysql.escape(tableData.length)})
      on duplicate key update BCGLINE = LAST_INSERT_ID(BCGLINE), TTLROLLGEN = TTLROLLGEN + VALUES(TTLROLLGEN);
      SELECT LAST_INSERT_ID() INTO @last_inserted_id_var_2;
      INSERT INTO FAB_BCGR (BCGUNIQ, BCGLINE, ROLLNO, RLENGTH, RWIDTH, PONO) VALUES (@last_inserted_id_var_1, @last_inserted_id_var_2, (?), (?), (?))
      on duplicate key update BCGREVL = LAST_INSERT_ID(BCGREVL);
      COMMIT;`, [nestedData], function(err, results, fields) {
      console.log(results)
      connection.release()
      if (err) {
        throw err
      }
    })

However, this doesn’t work. I get the following error when I run the above transaction.

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?), (?)) on duplicate key update BCGREVL = LAST_INSERT_ID(BCGREVL);

Now, presumably, the library doesn’t allow partially parameterized queries like the above for bulk inserts? I’m not sure.

I had to hack my way around it by doing the following

connection.query(`
     BEGIN;
     INSERT INTO FAB_BCGH (OCNO, CUSTNAME, STYLENO, STYLDESC) VALUES (${mysql.escape(OC)}, ${mysql.escape(CustName)}, ${mysql.escape(StyleNo)}, ${mysql.escape(StyleDesc)})
     on duplicate key update BCGUNIQ = LAST_INSERT_ID(BCGUNIQ);
     SELECT LAST_INSERT_ID() INTO @last_inserted_id_var_1;
     INSERT INTO FAB_BCGD(BCGUNIQ, ITEMCODE, ITEMDESC, TTLROLLGEN) VALUES (@last_inserted_id_var_1, ${mysql.escape(ItemNo)}, ${mysql.escape(ItemDesc)}, ${mysql.escape(tableData.length)})
     on duplicate key update BCGLINE = LAST_INSERT_ID(BCGLINE), TTLROLLGEN = TTLROLLGEN + VALUES(TTLROLLGEN);
     SELECT LAST_INSERT_ID() INTO @last_inserted_id_var_2;
     SELECT @last_inserted_id_var_1 as index1, @last_inserted_id_var_2 as index2;
     COMMIT;
   `, function(err, results, fields) {
     if (err) throw err
     const { index1, index2 } = results.map(r => {
       if (Array.isArray(r)) {
         return r
       }
     }).filter(r => r)
       .flat()[0]
     const nestedData = tableData.map(rollObj => {
       return [index1, index2, rollObj.roll, rollObj.length, PONumber]
     })
     connection.query(`
       INSERT INTO FAB_BCGR (BCGUNIQ, BCGLINE, ROLLNO, RLENGTH, PONO) VALUES ?
       on duplicate key update BCGREVL = LAST_INSERT_ID(BCGREVL);
     `, [nestedData], function(err, results, field) {
       if (err) throw err
     })
     connection.release()
   })

Now, I don’t believe that this is the best way to achieve the results I want. Is there a better way to do what I want?

[kai-koch]

A nesteddata array does not work for your usecase. If you have an nested array like:

nesteData = [[123456, 100, 50, '21t8h12h'], [123678, 100, 50, '21t8h12h']];

This is transformed with the values escaped into:

(123456, 100, 50, '21t8h12h'), (123678, 100, 50, '21t8h12h')

Which is convenient when you got a simple Bulk-Insert like:

INSERT INTO myTable (A,B,C,D) VALUES ?

and have the end result of:

INSERT INTO myTable (A,B,C,D)
 VALUES (123456, 100, 50, '21t8h12h'), (123678, 100, 50, '21t8h12h')

Your example SQL-String:

INSERT INTO FAB_BCGR (BCGUNIQ, BCGLINE, ROLLNO, RLENGTH, PONO)
VALUES (
    @last_inserted_id_var_1,
    @last_inserted_id_var_2,
    ?
)
/* would result in */
INSERT INTO FAB_BCGR (BCGUNIQ, BCGLINE, ROLLNO, RLENGTH, PONO)
VALUES (
    @last_inserted_id_var_1,
    @last_inserted_id_var_2,
    (123456, 100, 50, '21t8h12h'),
    (123678, 100, 50, '21t8h12h')
)

What you try to do is: Mixing serverside mysql variables @last_inserted_id_var_1, @last_inserted_id_var_2 with clientside data provided by JavaScript.

So building the Query with mysql.escape() is the way to go for your usecase, even if it looks a bit ugly.

The ‘nestedData’ functionality is convenient for simple Bulk-Statements. Under the Hood it use mysql.escape() just the same as you have in one of your examples.

[redixhumayun]

@kai-koch I just realised I never thanked you for your answer! So, thank you!