Connection pool Google Cloud SQL connection loss issue using Cloud Functions for Firebase

I am using connection pools with Promise wrapper in Cloud Functions for Firebase connecting to a Google Cloud SQL (for MySQL) instance.

I follow all the Connecting to Cloud SQL Connection Reuse recommendations:

• using a globally scoped connection pool

• not close connections at the end of the function call

I also follow all the Connecting to Cloud SQL Connection Pools & Connection Loss recommendations:

• use a client library that supports connection pools that automatically reconnect broken client connections

• maximum connections is set to 1
• By now I am only using 1 connection pool per cloud function.

Sometimes the following error occurs in production:

Unhandled rejection

Error: Connection lost: The server closed the connection.
    at PromisePool.query (/srv/node_modules/mysql2/promise.js:330:22)
.
.
.
    at functions.region.https.onRequest (/srv/index/handleRequest/index.js:10:3)
    at cloudFunction (/srv/node_modules/firebase-functions/lib/providers/https.js:49:9)
    at /worker/worker.js:783:7
    at /worker/worker.js:766:11
    at _combinedTickCallback (internal/process/next_tick.js:132:7)

As I read this library should reconnect by it self: https://github.com/sidorares/node-mysql2/issues/572#issuecomment-304774468 https://github.com/sidorares/node-mysql2/issues/836#issuecomment-414281593 Am I correct?

(Somewhat similar issue: https://github.com/mysqljs/mysql/issues/2151)

Relevant source code: mysqlPool.js:

const functions = require('firebase-functions')
, mysql = require('mysql2')

, {mysql: {user, password, database}} = functions.config()

, mysqlConfig = {
	user, password, database
	, connectionLimit: 1
	, decimalNumbers: true
}

if (process.env.NODE_ENV == 'production')
	mysqlConfig.socketPath = '/cloudsql/'
		+ '<CLOUD SQL INSTANCE CONNECTION NAME>'

module.exports = mysql.createPool(mysqlConfig).promise()

Usage:

const mysqlPool = require('./mysqlPool')

I am using the mysqlPool.query and mysqlPool.execute functions multiple times as I need them.

Versions:

• Node.js: 8.15.0 runtime
• mysql2: 1.7.0
• firebase-functions: 3.2.0

[mattostanik]

I am having the same issue as noted by several others above. I am receiving a ‘PROTOCOL_CONNECTION_LOST’ error while connecting to a Google Cloud SQL instance from a Cloud Function.

There are a couple Stack Overflow discussions where someone has experienced the error with an individual connection, and the recommended solution is to switch to a connection pool. However I am using a connection pool already, similar to the others who have reported this.

const pool = mysql.createPool({
    socketPath  : '/cloudsql/project:us-central1:db1',
    user        : 'user1',
    password    : 'abc',
    database    : 'db',
    waitForConnections: true,
    connectionLimit: 10,
    queueLimit: 0
});

On the positive side, finding this discussion and open issue at least made me feel like I wasn’t crazy or doing something completely wrong. This is a real unresolved issue even with connection pools.

My cloud function is reading a batch of 20 records from my SQL database, looping thru them and retrieving additional data for each record from an external API, then saving each record back to the db with an individual query. In a typical batch of 20 write queries, I am seeing anywhere from zero to 6 of them experience the timeout error. I’d say an average is 1 or 2 errors per batch, so a 5% or 10% incident rate.

This is not a great solution, but my fix is to put my query in a promise and watch for the error. If a timeout error occurs, it then retries the query. 90% of the time it succeeds on the second attempt. My code allows up to 5 attempts if necessary. It’s not super elegant or efficient, but it keeps my function from losing the data.

     // function to save data to SQL database
     const saveData = async (param1, param2, param3, param4, param5, param6, param7) => {
       var success = false;
       var attempts = 1;

       while ((!success) && (attempts <= 5)) {
         success = await executeQuery(param1, param2, param3, param4, param5, param6, param7, attempts).catch(err => { console.log(err) });
         attempts += 1;
         
       }
     }

     // function to execute the SQL query inside a promise
     const executeQuery = (param1, param2, param3, param4, param5, param6, param7, attempts) => {

       return new Promise((resolve, reject) => {
         pool.query("update table1 set field1 = ?, field2 = ?, field3 = ?, field4= ?, field5 = ?, field6 = ? where field7 = ?", [param1, param2, param3, param4, param5, param6, param7], function(err, results) {

           if ((err) && ((err.code === 'PROTOCOL_CONNECTION_LOST') || (err.code === 'ETIMEDOUT'))) {
             return reject(false);
           } else {
             return resolve(true);
           }
         });
       });
     }

It would be great if there is a better solution to this issue in the future.

[iturn]

This also happens on cloud run