question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Possible memory leak with connection.execute or I lack understanding of connection.execute

See original GitHub issue

Hi

I wrote a nodejs script to generate 30M records and insert periodically using the mysql2 driver with promise.

Here is the cut down code. This runs out of heap and crashes when I use the connection.execute. However it works fine when I use the connection.query.

'use strict'
const mysql2 = require("mysql2/promise");
const util = require("util");
async function generateLargeRecordset(limit){
    const insertPrefix = "INSERT INTO SOME_LARGE_TABLE (col1,col2,col3, col4) VALUES ";
    const connection = await mysql2.createConnection({
        host: 'localhost',
        user: 'root',
        password: 'password',
        database: 'database'
    });
    const insertArray = [];
    for(let i=0;i<limit;i++){
        insertArray.push(util.format(`(col1${i},col2${i},col3${i},col4${i})`));
        if(insertArray.length >= 1000){
            let query = insertPrefix + insertArray.join(',') + ';';
        // let result = await connection.execute(query); // <== heap memory issue. crashes when it runs out of heap
        // let result = await connection.query(query); //<== No heap memory issue
            insertArray.length = 0;
        }
    }

    if(insertArray.length >= 0){
        let query = insertPrefix + insertArray.join(',') + ';';
        // let result = await connection.execute(query); // <== heap memory issue. crashes when it runs out of heap
        // let result = await connection.query(query); //<== No heap memory issue
        insertArray.length = 0;
        const used = process.memoryUsage().heapUsed / 1024 / 1024;
        // The memory usage keeps increasing with the execute function
        console.log(`The script uses approximately ${Math.round(used * 100) / 100} MB`);
    }

}

generateLargeRecordset(10000000).then(()=>{
    process.exit(0);
}); 


Issue Analytics

  • State:open
  • Created 5 years ago
  • Comments:12 (5 by maintainers)

github_iconTop GitHub Comments

2reactions
jspatelcommented, Nov 22, 2018

It makes sense that execute could be expensive. But it doesn’t explain why application is not releasing any memory from the application side when there are no binding variables (?) for the parameters, hence there is no need to prepare a statement. I will test it out and post my results.

1reaction
sidorarescommented, Nov 22, 2018

e when there are no binding variables (?) for the parameters, hence there is no need to prepare a statement

It’s still prepared. It’s like compiling a C++ command line utility, even when intent is to start it without command line arguments it worth compiling upfront so it starts faster

Read more comments on GitHub >

github_iconTop Results From Across the Web

Potential memory leak with SQLDataReader / SQLDataAdapter
Hi. I have a simple form with a DataGridView which displays results returned by an SQL query containing c. 50,000 rows.
Read more >
Troubleshooting Memory Leaks - Oracle Help Center
If your application's execution time becomes longer and longer, or if the operating system seems to be performing slower and slower, this could...
Read more >
How to Fix Memory Leaks in Python? - Section.io
The occurrence of a memory leak fills up the program's storage, thus reducing storage space. With a lack of space, the program may...
Read more >
Hunting Java Memory Leaks - Toptal
Why are these leaks so bad? Among other things, leaking blocks of memory during program execution often degrades system performance over time, as...
Read more >
Memory leak - Wikipedia
In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found