question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Issue loading bulk data - column definition is repeating

See original GitHub issue

This is an odd one, but after spending a couple of hours dealing with it I need some help.

It appears that there is some column caching going on in table.js. I am using table.columns.add(name, type, options) where name is a string, type is a sql defined type, and options is an object with nullable and primary as keys.

I have the columns generated from a different schema and the aim is to create the table in the target schema. I have defined the column data and so I have an array of column objects, defined as described above.

After about 5 columns of a 357 column table, the library (I’m guessing here) is either removing the previous column and adding a new one, or overwriting the previous column.

When I hardcode the column names (as in the example), it works great. But if I cycle through any loops (i.e. using for…of or map) this issue pops up.

Expected behaviour:

The table object has the correct columns as part of it’s columns attribute.

Actual behaviour:

The table object has repeated columns. (see screenshots)

Configuration:

I am using the msnodesqlv8 driver because I need to use windows auth and I don’t see how to do that with tedious.

Relevant code:

//attempted to make it a promise to see if that would isolate the 
//issue - but I also had this in the main function where I define the table 
//first.  Same problem.
  
/********************************
 * the process works great up to LastName, but once it hits that column, it replaces the Name column.  
 * (see screenshots for examples).
*/
async function addColumns(table, tableColumns) {
  await tableColumns.map(columnData => {
    const { name, type, options } = columnData;
    table.columns.add(name, type, options);
  });
}

...

 const table = new sql.Table(tableName);
  table.create = false;
  addColumns(table, tableColumns);
...

The tableColumns object is created like so:

...
      columnDefsForInsert.push({
        name: `${theField.name}`,    //string
        type: fieldTypeForBulkInsert, // lookup table to translate to a sql Type.
        options: fieldOptions,  //Object defined as {nullable: boolean, primary: boolean}
      });
...

Software versions

  • NodeJS: 10.15.2
  • node-mssql: 5.1.0
  • msnodesqlv8: 0.6.12
  • SQL Server: 2014

This is a segment of the column data. image

This is what it looks like prior to the issue - the columns look good so far. image

On the next iteration, it adds a new column BUT replaces the previous column. image

At the end - I get this IMNOD error from the driver - invalid parameter type because the columns aren’t defined correctly for the data coming in.

image

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:6

github_iconTop GitHub Comments

1reaction
willmorgancommented, Apr 25, 2019

@Suraiya-Hameed @arthurschreiber I notice you’ve both collaborated on issues relating to TediousJS’s Windows auth. Can you confirm whether it is available or not?

0reactions
cromieentcommented, May 1, 2019

I actually moved on to a different solution, so we can go ahead and close this out. It’s pretty strange though. Perhaps I’ll revisit once I get through my main development, but if this isn’t happening for others it’s likely happening just for me (I’m so lucky!). Thanks for the responses!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Eliminating Repeating Groups In Your Database Tables
A repeating group is a series of fields/attributes that are repeated throughout a database table. It is a common problem faced by ...
Read more >
Add bulk loading · Issue #104 · prisma/tiberius - GitHub
Since the bulk-load stream does not repeat the type info before each value, I've split the methods encoding the data and the type...
Read more >
PostgreSQL: Bulk loading huge amounts of data - CYBERTEC
This article is about bulk loading, which is the quickest way to import large amounts of data into a PostgreSQL database.
Read more >
how can I Ignore truncation error on bulk insert - Stack Overflow
So add a staging table that has that column (as a max), and then load the real table from the staging table without...
Read more >
Using the Bulk-Load Tools in SQL Server - Erland Sommarskog
Column numbers start on 1, and 0 means that the field is not to be imported. The seventh property is the column name,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found