The pg instrumentation includes the waiting time in the client's internal queue when recording query runtime

Got a bit of a jumpscare the other day when we refactored one of our apps in a way that caused us to issue more pg client calls in parallel. The change didn’t affect the number of queries being run or their actual execution times in the database, but newrelic’s graphs reported a severe performance regression:

Since you cannot multiplex parallel queries into a single postgres connection, the pg client has an internal waiting queue so that each query ends up getting exclusive access: https://github.com/brianc/node-postgres/blob/bf469399b88bcdf86eff096fd0dd05684adc1117/packages/pg/lib/client.js#L557

However, node-newrelic’s instrumentation of the pg module only shims the query method: https://github.com/newrelic/node-newrelic/blob/18de96b2bb37691d6820ad806644753f7cc237fe/lib/instrumentation/pg.js#L54-L60

As far as I can tell, the consequence is that the time the query spends waiting in queryQueue will be counted as part of its runtime. That also explains why the actual request response time reported in the chart above is way lower than the height of the solid colors.

It looks like the instrumentation has to dive one step deeper to account for this.

[carlo-808]

We were able to reproduce the behavior and a reproduction is here: https://github.com/carlo-808/pg-query-timing-repro

I suspect that we will need to instrument the Query object itself possibly around the submit function. We will get this issue further prioritized.

[bizob2828]

Thanks for reaching out @papandreou. I moved this close to the top of our backlog. We had other priorities come in and this may have got bumped further down the stack than expected.