question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Not working with large SQL : Error: Connection terminated unexpectedly

See original GitHub issue

I’m trying to query a large SQL file (~65000 lines) and constantly getting Error: Connection terminated unexpectedly after 2 min of execution.

Here is code:

'use strict';

var fs = require('fs'),
    pg = require('pg'),
    Pool = pg.Pool;

(async () => {
    var pool = new Pool({
        database: 'some database',
        user: 'app',
        password: 'some password',
        host: 'some host',
        port: 9999,
        ssl: true,
        max: 20, // set pool max size to 20
        min: 4, // set min pool size to 4
        idleTimeoutMillis: 10000, // close idle clients after 1 second
        connectionTimeoutMillis: 10000, // return an error after 1 second if connection could not be established
    });

    var client = await pool.connect();

    pool.on('error', error => {
        console.error('error event on pool (%s)', error);
    });

    try {
        await client.query('BEGIN');


        const queryText = fs.readFileSync('./v2-game_data.sql') + '';


        const res = await client.query(queryText)


        await client.query('COMMIT')

    } catch (e) {
        await client.query('ROLLBACK')
        throw e
    } finally {
        client.release();
    }
})();

And here is error:

events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: Connection terminated unexpectedly
    at Connection.con.once (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg\lib\client.js:252:9)
    at Object.onceWrapper (events.js:286:20)
    at Connection.emit (events.js:198:13)
    at TLSSocket.<anonymous> (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg\lib\connection.js:131:10)
    at TLSSocket.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Emitted 'error' event at:
    at connectedErrorHandler (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg\lib\client.js:199:10)
    at Connection.con.once (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg\lib\client.js:268:9)
    at Object.onceWrapper (events.js:286:20)
    [... lines matching original stack trace ...]
    at process._tickCallback (internal/process/next_tick.js:63:19)

UPD

With pg-native the error is:

events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: SSL SYSCALL error: EOF detected

    at module.exports.Client._readError (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg-native\index.js:154:13)
    at module.exports.Client._read (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg-native\index.js:203:17)
    at PQ.emit (events.js:198:13)
Emitted 'error' event at:
    at module.exports.<anonymous> (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg\lib\native\client.js:101:14)
    at module.exports.emit (events.js:198:13)
    at module.exports.Client._readError (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg-native\index.js:155:8)
    at module.exports.Client._read (C:\Users\ihatemicrosoft1\Desktop\backend\node_modules\pg-native\index.js:203:17)
    at PQ.emit (events.js:198:13)

How to make it work?

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
andrew-medvedevcommented, Dec 3, 2019

I really believe that there is a vast amount of other methods of doing this job. And I can understand if it’s pointless for you, but it doesn’t mean that it pointless for me. I have tasks and they have to be accomplished in exact way. So my answer will be like because of reasons

1reaction
sehropecommented, Dec 3, 2019

If the SQL file contains basic DML commands (ex: INSERT, UPDATE, etc) then unless you hit some kind of memory issue in node, it should work fine.

If the SQL file contains COPY commands with inline data this will not work. The COPY protocol requires special handling and changes the client protocol handlers to expect a copy data stream.

Read more comments on GitHub >

github_iconTop Results From Across the Web

node-postgres, Connection terminated unexpectedly
I have tried this, problem is asynchronous property of JavaScript. Connection is being ended before execution of query.
Read more >
7.18.2: "Connection terminated unexpectedly" when using ...
Any insight about this? I am facing the same problem, my app throws an error when a connection sits iddle for a long...
Read more >
Connection terminated unexpectedly for long queries-postgresql
Turns out it was just a Railway issue (the PostgreSQL host I was using). The same issue appears when using Render and other...
Read more >
PostgreSQL "connection terminated" - quite frustrated
Error : Connection terminated unexpectedly at Connection.con.once ... works by running this from the server psql <MY CONNECTION STRING> ).
Read more >
Troubleshoot common connection issues to Azure SQL ...
You receive this error when detach workflow fail to clean-up the in-memory cache on time and subsequent lookups to the database point to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found