Possible memory leak with connection.execute or I lack understanding of connection.execute
See original GitHub issueHi
I wrote a nodejs script to generate 30M records and insert periodically using the mysql2 driver with promise.
Here is the cut down code. This runs out of heap and crashes when I use the connection.execute. However it works fine when I use the connection.query.
'use strict'
const mysql2 = require("mysql2/promise");
const util = require("util");
async function generateLargeRecordset(limit){
const insertPrefix = "INSERT INTO SOME_LARGE_TABLE (col1,col2,col3, col4) VALUES ";
const connection = await mysql2.createConnection({
host: 'localhost',
user: 'root',
password: 'password',
database: 'database'
});
const insertArray = [];
for(let i=0;i<limit;i++){
insertArray.push(util.format(`(col1${i},col2${i},col3${i},col4${i})`));
if(insertArray.length >= 1000){
let query = insertPrefix + insertArray.join(',') + ';';
// let result = await connection.execute(query); // <== heap memory issue. crashes when it runs out of heap
// let result = await connection.query(query); //<== No heap memory issue
insertArray.length = 0;
}
}
if(insertArray.length >= 0){
let query = insertPrefix + insertArray.join(',') + ';';
// let result = await connection.execute(query); // <== heap memory issue. crashes when it runs out of heap
// let result = await connection.query(query); //<== No heap memory issue
insertArray.length = 0;
const used = process.memoryUsage().heapUsed / 1024 / 1024;
// The memory usage keeps increasing with the execute function
console.log(`The script uses approximately ${Math.round(used * 100) / 100} MB`);
}
}
generateLargeRecordset(10000000).then(()=>{
process.exit(0);
});
Issue Analytics
- State:
- Created 5 years ago
- Comments:12 (5 by maintainers)
Top Results From Across the Web
Potential memory leak with SQLDataReader / SQLDataAdapter
Hi. I have a simple form with a DataGridView which displays results returned by an SQL query containing c. 50,000 rows.
Read more >Troubleshooting Memory Leaks - Oracle Help Center
If your application's execution time becomes longer and longer, or if the operating system seems to be performing slower and slower, this could...
Read more >How to Fix Memory Leaks in Python? - Section.io
The occurrence of a memory leak fills up the program's storage, thus reducing storage space. With a lack of space, the program may...
Read more >Hunting Java Memory Leaks - Toptal
Why are these leaks so bad? Among other things, leaking blocks of memory during program execution often degrades system performance over time, as...
Read more >Memory leak - Wikipedia
In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
It makes sense that execute could be expensive. But it doesn’t explain why application is not releasing any memory from the application side when there are no binding variables (?) for the parameters, hence there is no need to prepare a statement. I will test it out and post my results.
It’s still prepared. It’s like compiling a C++ command line utility, even when intent is to start it without command line arguments it worth compiling upfront so it starts faster