Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(Potential) Memory Leak In insert() Call

See original GitHub issue

Environment details

OS: Linux Alpine, macOS
Node.js version: v14.3.0
npm version: 6.14.5
@google-cloud/bigquery version: ^4.7.0

Steps to Reproduce

const {BigQuery} = require('@google-cloud/bigquery');

(async () => {
    const client = new BigQuery({
        keyFilename: './google-creds.json',
        projectId: 'myproductid'
    });
    const run = async () => {
        await client
            .dataset('events')
            .table('analytics')
            .insert([{
                insertId: '1',
                json: {
                    test: 1
                }
            }, {
                insertId: '2',
                json: {
                    test: 2
                }
            }], {raw: true});
    };

    setInterval(async () => {
        await run();
    }, 5000);
})();

Here’s a production screenshot that shows external memory (top), total heap (middle) and used heap (bottom).

As you can see external memory slowly goes up. If I remove my BigQuery insert statements in my endpoint external memory releases and doesn’t gradually increase.

Here’s an isolated external memory chart for one of the pods:

I’ve found out it’s (maybe) related to teeny-request being called in node_modules/@google-cloud/common/build/src/util.js. Specifically this function:

    /**
     * Make a request through the `retryRequest` module with built-in error
     * handling and exponential back off.
     *
     * @param {object} reqOpts - Request options in the format `request` expects.
     * @param {object=} config - Configuration object.
     * @param {boolean=} config.autoRetry - Automatically retry requests if the
     *     response is related to rate limits or certain intermittent server
     * errors. We will exponentially backoff subsequent requests by default.
     * (default: true)
     * @param {number=} config.maxRetries - Maximum number of automatic retries
     *     attempted before returning the error. (default: 3)
     * @param {object=} config.request - HTTP module for request calls.
     * @param {function} callback - The callback function.
     */
    makeRequest(reqOpts, config, callback) {
        const options = {
            request: teeny_request_1.teenyRequest.defaults(requestDefaults),
            retries: config.autoRetry !== false ? config.maxRetries || 3 : 0,
            shouldRetryFn(httpRespMessage) {
                const err = util.parseHttpRespMessage(httpRespMessage).err;
                return err && util.shouldRetryRequest(err);
            },
        };
        if (typeof reqOpts.maxRetries === 'number') {
            options.retries = reqOpts.maxRetries;
        }
        if (!config.stream) {
            return retryRequest(reqOpts, options, (err, response, body) => {
                util.handleResp(err, response, body, callback);
            });
        }

If I return before return retryRequest external memory doesn’t go up. Maybe I’m wrong but this is as far as I got before concluding I need some help from the authors.

I hope you can help! Thanks!