SequelizeConnectionAcquireTimeoutError randomly DB connections pool error

I’m running a very simple app that using sequelize basics, it was working for the recent year without any issues on sequelize v4. Recently, the app is crashing every day once a day, with the following error:

TimeoutError: ResourceRequest timed out
    at ResourceRequest._fireTimeout (/usr/src/app/node_modules/generic-pool/lib/ResourceRequest.js:62:17)
    at Timeout.bound (/usr/src/app/node_modules/generic-pool/lib/ResourceRequest.js:8:15)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)

It was happening with sequelize 4.42.0, mysql2 v1.6.5, node carbon alpine. After reading a lot about this issue, I figured out that sequelize v5 might handle the connections pool better, so I’ve tried sequelize 5.7.6, mariadb v2.0.3, but nothing, now the error that I’m getting is:

SequelizeConnectionAcquireTimeoutError: Operation timeout
    at pool.acquire.catch.err (/usr/src/app/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:281:53)
    at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
    at /usr/src/app/node_modules/bluebird/js/release/catch_filter.js:17:41
    at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
    at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
    at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
    at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
    at Immediate.Async.drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)

It’s very important to tell that I have different apps that run on the same K8s cluster with the same DB, same auth, and it’s not crashing.

Any suggestions?

[jzavisek]

@demian85 I was seeing this error as well and it seems that in my case it happened when some sequelize calls within a transaction weren’t using the parent transaction. So something like:

sequelize.transaction(async transaction => {
  await Model.create({ name: 'Test' }, { transaction })
  await DifferentModel.findAll() // missing transaction
})

It seems that these calls blocked the transaction for larger amount of time and other transactions failed with SequelizeConnectionAcquireTimeoutError. I am not sure if this could be related though.

[yicheng-irun]

我这里同样是在k8s集群上出现这个问题，我把同一pods下面的另一个协同容器的cpu内存配置提升成和主容器一样的配置之后正常了。

提升配置只是延缓了报错的时机。我这个报错的真正原因是因为k8s会发送非常大流量的嗅探 head请求，因为我之前不恰当的使用方式使每一次head请求都会触发一次db的查询，一段时间后，就批量会报这个错误。我后面的处理方式是在express的app注册回调的最前面 app.head(‘/’, (req, res) => { res.send(“ok”) }); 直接拦截掉head请求，使之不会往下一步的回调走，解决了这个问题。