client in pool timed out, causing pool to fail for all future queries

Hello! I’m using a pg-pool in a lambda environment. Last night, the pool hit an error:

2020-06-19T17:28:50.154-04:00 | START
2020-06-19T17:28:50.257-04:00 | ERROR	Invoke Error
"Error: Connection terminated due to connection timeout",
        "    at Timeout.<anonymous> (/var/task/node_modules/pg/lib/client.js:103:26)",

My pool is a static global to allow lambda to reuse across invocations. I guess pg wasn’t smart enough to check idle timeout before assigning a client, and in this case it was already dead – is there a way to flush the pool on each lambda invocation startup?

Anyway, somehow this first timeout put the pool in a bad state, and every call to the pool (which are made using the .query helper) instantly failed with a timeout.

  | 2020-06-19T17:29:17.492-04:00 | START
  | 2020-06-19T17:29:17.496-04:00 | ERROR	Invoke Error
Error: Connection terminated due to connection timeout

My config (pg 8.0.3, xray 3.0.1)

import pg from 'pg';
import xrayPostgres from 'aws-xray-sdk-postgres';
const { Pool } = xrayPostgres(pg)
const pool = new Pool({
      connectionTimeoutMillis: 5000,
      idleTimeoutMillis: 60000,
      ssl: ...
});

[sehrope]

The issue is that when the lambda freezes and then thaws, any TCP connections are broken but not necessarily notified that they’re broken. This can lead to all sorts of issues so the simplest advice is not to use any long lived pooling in lambda and keep things stateless.

See this thread for more on this topic: https://github.com/brianc/node-postgres/issues/2112

There’s likely some improvements that could be done to the pool / client code to better handle these types of errors but it’s not going to be anything that will make those connections usable. At best it will end up attempting to use them, hanging or erroring out, and then getting you a new connection after some timeout.

If you still want to stick with the pool interface to handle the connection management (as it’s convenient for cleaning up resources), check out the “ZeroPool” mentioned here: https://github.com/brianc/node-postgres/issues/1938

[sehrope]

That’s probably a bad idea anyway (each IDLE connections take up some resources so you can’t have infinite of them). Even if you could, there’s no way to guarantee it will survive a freeze / thaw. Better to avoid it entirely.

If you’re concerned about slow connection times then consider adding a dedicated connection pooler like pgbouncer and have your Lambda target the pooler.