Connection terminated unexpectedly (Postgresql / AWS Lambda)

Environment

Knex version: 0.20.8 Database + version: Postgres 9.5.14 OS: AWS Lambda - Node 12.x

Recently, I’ve been seeing upticks in a connection terminated unexpectedly error. I can’t find a discernible pattern at this point but unlike #3523, it doesn’t seem to occur after long periods of activity. I see instances that are minutes apart in some cases.

I added some logging to the Knex afterCreate method and am seeing two errors:

1. The connection terminated unexpectedly error
2. And Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed - this seems to happen on the 1st or second invocation after seeing the connection terminated unexpectedly error.

I’ve tried a number of configurations with Knex pooling:

1. { pool: { min: 1, max: 1 } }
2. { pool: { min: 0 }
3. { pool: { min: 1 } }

I’ve also tried setting { pool: { idleTimeoutMillis: 300000 } } as per #3523.

Additional information:

• I’m defining my Knex instance outside of my AWS handlers, so the pool should be reused per container.
• I’m using the following for my afterCreate function – I’m not sure if there’s a better way to get at the underlying error?

function afterCreate(connection, done) {
  connection.on('error', (err) => {
    console.log('POSTGRES CONNECTION ERROR');
    console.error(err)
  });
  done()
}

Appreciate any help! This is driving me nuts 😃

UPDATE: the only change that I’ve made is migrating the lambda function to Node 12 from Node 8 as AWS is removing support for 8. Unfortunately, it doesn’t look like I can go back. My other functions in production are on Node 8 and not seeing any issues 🤔🤔🤔

ISSUE FOUND: for whatever reason, reverting back to Node 8.x fixes the issue. Still not certain on the underlying cause.

[lastmjs]

After digging into this more, it seems not possible to use prepared statements with any kind of transaction pooling on postgres. The two main types of pooling are session pooling and transaction pooling. Transaction pooling seems the most generally performant. Transaction pooling is what RDS Proxy uses. node-postgres or pg does not seem compatible with any kind of transaction pooling if using prepared statements, and Knex I believe always uses prepared statements. This discussion that I’ve started is probably better moved over to the pg repo. I’ll leave some links here:

• https://github.com/brianc/node-postgres/issues/2327
• https://github.com/brianc/node-postgres/issues/2266
• https://github.com/brianc/node-postgres/issues/2326

[briandamaged]

@elhigu : when I said “query”, I meant “query a predicate method that returns synchronously”. I’m actually in the process of implementing an experimental branch in knex for this. If the experiment works, then the pg team can integrate the concept / expose a public API for it.

(Unfortunately, I’m encountering a minor roadblock at the moment. It appears that a handful of Postgres unit tests are attempting to connect to the wrong port. I’m not sure why this fix would suddenly uncover that issue)