[BUG] Firebase auth triggers timeout

[READ] Step 1: Are you in the right place?

[REQUIRED] Step 2: Describe your environment

• Operating System version: GCF
• Firebase SDK version: 9.4.2 [already tried 9.6.0 and it has exact same proble]
• Firebase Product: auth (auth, database, storage, etc)
• Node.js version: 10
• NPM version: I’ve no idea what version google uses

[REQUIRED] Step 3: Describe the problem

I’ve been working on an Android project for over 3 years now. Since beginning my project uses Firebase for many purposes including authentication. Nowadays has grown a lot and I got a few thousand users per day and many hundreds users per hour. With this expansion I started noticing some errors that weren’t before [or weren’t that noticiable] I use a firebase function to authenticate my users. The function is simple it validates a request cryptographic signature (to guarantee that it came from my app and not from someone bruteforcing it), then extract the parameters and query the firebase database for that credentials (only 1 result/or no result, will always been returned) if everything is okay a firebase authentication token is created and returned to user. I’m using GCF with node 10 runtime, 40 seconds timeout and 128MB memory limit

const admin = require('firebase-admin');
admin.initializeApp({
});

const auth = admin.auth();
const igDatabase = admin.database();
const ServerValue = admin.database.ServerValue;

const crypto = require("crypto");
 exports.auth = (req, res) => {
    if (validateReqParams(req.body)) {

        if (validateCredentials(req.body)) {
	//here I extract the params from the json body
            authenticate(uid, newUser, regDate, res);
        } else
            invalidRequest('invalid body.d', req.body.d, res);

    } else
        invalidRequest('body.s doesn\'t match', req.body, res);

};


function validateReqParams(body) {

    return body && body.sign && body.data && body.sign == crypto.createHmac('sha256', 'my key').update(body.data).digest('hex');

}

function invalidRequest(error, item, res) {
    console.error(error);
    console.log(JSON.stringify(item));
    res.end();
}

function authenticate(uid, newUser, regDate, res) {
    auth.createCustomToken(uid).then(function(customToken) {

        res.status(200).json({
            t: customToken,
            n: newUser,
            r: regDate
        });
    }).catch(function(error) {
        console.error("Error creating custom token: " + JSON.stringify(error));
        res.status(200).json({
            e: error
        });
    });

}

I wont copy the validate credentials function because is a long code, but there ARENT any loops or recursion, neither it allocates memory, it just query few data from database each query having possible result 1 boolean or no data…

This is a VERY SIMPLE set up, I have few hundreds requests per hour with an average of 250 ms processing time and 99% of requests being faster than 300ms. But once and while one request takes 40,000 ms seconds and then is finished due timeout

I’ve extensively tried to debug and find the problem without the success, I even collected the exact params from the requests that caused timeout and tried to bruteforce the function with these params to see if at any point it would timeout, it DIDN’T

From the debug data I could collect, I noticed the timeout happens at the auth.createCustomToken(uid)call (I’ve place many debug code and monitored few hundreds executions and the only debug lines those where missing where after this function call)

Sent many emails to firebase support [which were ignored] and I couldn’t find the exact root cause of the problem… but with a look over the firebase function metrics charts and all the data I collected I’ve few hypotesys

Both these charts correspond of the same 24 hours period. I placed some dash lines to show how synchronized the peak execution times are compared to the garbage collection time… This tendency is even more noticiable on charts from shorter periods

These charts now show a six hours interval… and It is clear that the execution times peak COMES IN A VERY REGULAR 30 MINUTES INTERVAL… the thing is so regular that this image almost looks like a heart peacemaker chart. Also noticiabe, these peaks are aligned with a memory peak followed by a discharge

With all that being said these are my conclusions. 1- The memory grows very slowly and if you ignore the peak the second highest level is only 10% higher than the average level… so there is low chance of this being a memory leak from my code 2- The memory peak/discharge are very fast, similar what we would expect from a garbage collection routine.

With all the exposed data I come to ask: 1- Is it possible that garbage collection is affecting the node execution? 2- Is the natural garbage collection execution memory supposed to account for the whole memory allocated by the cloud function? 3- Is possible that the firebaseauth has a memory leak that causes all this? 4- Does anyone faces the same issue? Is there anything I can do to solve?

[hiranya911]

It’s already released. 9.7.0 available for download now.

[ganey]

Do the timeouts on GCF happen on cold starts? Or just randomly?

There is still an issue with firestore loaded though the admin SDK that causes it to sometimes take say 9 seconds to load, call firestore and do some auth.

There’s a thread on Google groups about it, but it doesn’t seem like Google are interested in fixing the random slowness on cold starts. The issue is over a year old and they have plenty of replication examples.