Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

table.dropColumns doesn't work with sqlite3

See original GitHub issue

When using SQLite, table.dropColumns doesn’t accept multiple column names the way it does with other drivers, per the documentation (I compared with how table.dropTimestamps is implemented just to make sure that I was using it correctly). Here’s how to reproduce this, in a fresh directory:

$ npm init -y
$ npm i --save knex sqlite3
$ node_modules/.bin/knex init
$ node_modules/.bin/knex migrate:make start
$ cat >migrations/*_start.js
exports.up = function(knex, Promise) {
  return Promise.all([
    knex.schema.createTable('user', function(table) {
      table.increments();
    })
  ]);
};

exports.down = function(knex, Promise) {
  return Promise.all([
    knex.schema.dropTable('user')
  ]);
};
$ node_modules/.bin/knex migrate:make add-columns
$ cat >migrations/*_add-columns.js
exports.up = function(knex, Promise) {
  return Promise.all([
    knex.schema.table('user', function(table) {
      table.text('name');
      table.text('login');
    })
  ]);
};

exports.down = function(knex, Promise) {
  return Promise.all([
    knex.schema.table('user', function(table) {
      table.dropColumns(['name', 'login']);
    })
  ]);
};
$ node_modules/.bin/knex migrate:latest
$ node_modules/.bin/knex migrate:rollback

Instead of successfully dropping the columns, the output is:

Using environment: development
Knex:warning - sqlite does not support inserting default values. Set the `useNullAsDefault` flag to hide this warning. (see docs http://knexjs.org/#Builder-insert).
Knex:warning - migrations failed with error: The column name,login is not in the user table
Error: The column name,login is not in the user table
    at SQLite3_DDL.<anonymous> (/home/cjwatson/k/node_modules/knex/lib/dialects/sqlite3/schema/ddl.js:56:28)
    at SQLite3_DDL.tryCatcher (/home/cjwatson/k/node_modules/bluebird/js/release/util.js:16:23)
    at SQLite3_DDL.getColumn (/home/cjwatson/k/node_modules/bluebird/js/release/method.js:15:34)
    at client.transaction.connection (/home/cjwatson/k/node_modules/knex/lib/dialects/sqlite3/schema/ddl.js:223:21)
    at /home/cjwatson/k/node_modules/knex/lib/transaction.js:81:20
    at tryCatcher (/home/cjwatson/k/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/home/cjwatson/k/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/home/cjwatson/k/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/home/cjwatson/k/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

I have to drop the columns one at a time instead (which is presumably particularly inefficient given how table alteration is implemented for SQLite).

I encountered this with knex 0.12.8.

Issue Analytics

State:
Created 7 years ago
Reactions:1
Comments:5 (1 by maintainers)

Top GitHub Comments

1reaction

cjwatsoncommented, Mar 19, 2017

I only mentioned performance in passing. Sure, I know that performance isn’t especially important here, but my point is that it’s a clear knex bug: the behaviour of the SQLite driver for this function is not the same as the documented behaviour. I reported it because it caused me to spend time figuring out why code that appeared to match your documentation didn’t work, not because I think anyone is going to be too worried about having to drop columns one at a time as such.

The query never makes it as far as the database:

$ DEBUG=knex:\* node_modules/.bin/knex migrate:rollback
Using environment: development
Knex:warning - sqlite does not support inserting default values. Set the `useNullAsDefault` flag to hide this warning. (see docs http://knexjs.org/#Builder-insert).
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=0 +0ms
  knex:client acquired connection from pool: __knexUid1 +8ms
  knex:query select * from sqlite_master where type = 'table' and name = ? +3ms
  knex:bindings [ 'knex_migrations' ] +1ms
  knex:client releasing connection to pool: __knexUid1 +4ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +2ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query select * from sqlite_master where type = 'table' and name = ? +1ms
  knex:bindings [ 'knex_migrations_lock' ] +0ms
  knex:client releasing connection to pool: __knexUid1 +1ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +2ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query select * from "knex_migrations_lock" +9ms
  knex:bindings [] +1ms
  knex:client releasing connection to pool: __knexUid1 +3ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +1ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query select "name" from "knex_migrations" order by "id" asc +1ms
  knex:bindings [] +0ms
  knex:client releasing connection to pool: __knexUid1 +1ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +1ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query select * from "knex_migrations" where "batch" = (select max("batch") from "knex_migrations") order by "id" desc +2ms
  knex:bindings [] +0ms
  knex:client releasing connection to pool: __knexUid1 +1ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +1ms
  knex:tx trx2: Starting top level transaction +2ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +1ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query BEGIN; +2ms
  knex:bindings undefined +0ms
  knex:query select * from "knex_migrations_lock" +2ms
  knex:bindings [] +0ms
  knex:query update "knex_migrations_lock" set "is_locked" = ? +3ms
  knex:bindings [ 1 ] +0ms
  knex:query COMMIT; +1ms
  knex:bindings undefined +1ms
  knex:tx trx2: releasing connection +16ms
  knex:client releasing connection to pool: __knexUid1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +4ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query select max("batch") as "max_batch" from "knex_migrations" +1ms
  knex:bindings [] +0ms
  knex:client releasing connection to pool: __knexUid1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +1ms
  knex:tx trx3: Starting top level transaction +6ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +0ms
  knex:client acquired connection from pool: __knexUid1 +0ms
  knex:query BEGIN; +1ms
  knex:bindings undefined +0ms
  knex:query PRAGMA table_info("user") +5ms
  knex:bindings [] +0ms
  knex:tx trx5: Starting top level transaction +1ms
  knex:query SAVEPOINT trx5; +0ms
  knex:bindings undefined +1ms
  knex:query ROLLBACK TO SAVEPOINT trx5 +3ms
  knex:bindings undefined +1ms
  knex:tx trx5: not releasing external connection +3ms
  knex:query ROLLBACK; +1ms
  knex:bindings undefined +0ms
  knex:tx trx3: releasing connection +1ms
  knex:client releasing connection to pool: __knexUid1 +0ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
Knex:warning - migrations failed with error: The column name,login is not in the user table
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=1 available=1 +1ms
  knex:client acquired connection from pool: __knexUid1 +1ms
  knex:query update "knex_migrations_lock" set "is_locked" = ? +0ms
  knex:bindings [ 0 ] +1ms
  knex:client releasing connection to pool: __knexUid1 +12ms
  knex:pool INFO pool sqlite3:sqlite3:client0 - dispense() clients=0 available=1 +0ms
Error: The column name,login is not in the user table
    at SQLite3_DDL.<anonymous> (/home/cjwatson/k/node_modules/knex/lib/dialects/sqlite3/schema/ddl.js:56:28)
    at SQLite3_DDL.tryCatcher (/home/cjwatson/k/node_modules/bluebird/js/release/util.js:16:23)
    at SQLite3_DDL.getColumn (/home/cjwatson/k/node_modules/bluebird/js/release/method.js:15:34)
    at client.transaction.connection (/home/cjwatson/k/node_modules/knex/lib/dialects/sqlite3/schema/ddl.js:223:21)
    at /home/cjwatson/k/node_modules/knex/lib/transaction.js:81:20
    at tryCatcher (/home/cjwatson/k/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/cjwatson/k/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/home/cjwatson/k/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/home/cjwatson/k/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/home/cjwatson/k/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

src/dialects/sqlite3/schema/ddl.js has a dropColumn function that always calls getColumn on its first parameter. getColumn is defined in the same file and checks its own records of what columns are in the table, rejecting “name,login” before it gets as far as SQLite itself. Now look in src/schema/tablebuilder.js, where table.dropColumns is defined: notice that it just converts its argument to an array and passes that to table.dropColumn via the table compiler machinery.

The generic TableCompiler.prototype.dropColumn implementation in src/schema/tablecompiler.js handles the case where its argument is an array of columns. The specific TableCompiler_SQLite3.prototype.dropColumn implementation in src/dialects/sqlite3/schema/tablecompiler.js does not. I haven’t figured out exactly where the comma-join is coming from here, but that seems like a minor point.

Hopefully this is enough to be going on with.

0reactions

razvanzcommented, Mar 22, 2017

Same error when attempting to rename columns:

exports.up = function (knex) {
  console.log('second migration')
  return knex.schema
    .table('organizations', function (table) {
      console.log(table)
      table.renameColumn('createdTime', 'created_time')
      table.renameColumn('updatedTime', 'updated_time')
    })
}

It seems this hasn’t worked at all in 0.12.*. It works for mysql.

PS: It does work if the sqlite file already has the tables created, thus only the second migration is run.