Error on streaming huge recordset
See original GitHub issueHi @patriksimek please consider following issue.
"mssql": "^3.3.0",
mssql: Object.assign({}, {
stream: true,
options: {encrypt: false},
debug: true,
requestTimeout: 50000,
pool: {
max: 10,
min: 0,
idleTimeoutMillis: 100000,
},
}),
Hi, I have error on while streaming 1000.000. It begin streaming normally but after interruption can’t proceed.
Code looks like following:
const requestGenerator = MssqlAdaptor.Factory(mssql);
requestGenerator.then((adaptor) => {
let request = adaptor.request;
request.on('error', function (err){
console.log('Request error');
console.log(err);
});
request.on('row', function (data){
console.log('Next data: ' + data);
elasticAdaptor.addRawData([data]);
});
request.on('done', function (){
elasticAdaptor.replaceOldIndex().then(function (err, result) {
done(null, "Finished");
});
});
adaptor.executeRequest(query);
});
Request error
{ RequestError: Timeout: Request failed to complete in 50000ms
at Request.userCallback (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/mssql/lib/tedious.js:802:25)
at Request.callback (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/request.js:33:27)
at Connection.message (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/connection.js:1209:31)
at Connection.dispatchEvent (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/connection.js:519:45)
at MessageIO.<anonymous> (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/connection.js:439:23)
at emitNone (events.js:86:13)
at MessageIO.emit (events.js:185:7)
at ReadablePacketStream.<anonymous> (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/message-io.js:92:15)
at emitOne (events.js:96:13)
at ReadablePacketStream.emit (events.js:188:7)
at readableAddChunk (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_readable.js:198:18)
at ReadablePacketStream.Readable.push (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_readable.js:157:10)
at ReadablePacketStream.Transform.push (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-
[mssql_debug_1486634130964.txt](https://github.com/patriksimek/node-mssql/files/763287/mssql_debug_1486634130964.txt)
[mssql_debug_1486634516673.txt](https://github.com/patriksimek/node-mssql/files/763293/mssql_debug_1486634516673.txt)
[mssql_debug_1486634844538.txt](https://github.com/patriksimek/node-mssql/files/763294/mssql_debug_1486634844538.txt)
[mssql_debug_1486634900132.txt](https://github.com/patriksimek/node-mssql/files/763291/mssql_debug_1486634900132.txt)
[mssql_debug_1486634978027.txt](https://github.com/patriksimek/node-mssql/files/763292/mssql_debug_1486634978027.txt)
stream/lib/_stream_transform.js:123:32)
at ReadablePacketStream._transform (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/message-io.js:57:16)
at ReadablePacketStream.Transform._read (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_transform.js:159:10)
at ReadablePacketStream.Transform._write (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_transform.js:147:83)
at doWrite (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_writable.js:313:64)
at writeOrBuffer (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_writable.js:302:5)
at ReadablePacketStream.Writable.write (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_writable.js:241:11)
at CleartextStream.ondata (_stream_readable.js:561:20)
at emitOne (events.js:96:13)
at CleartextStream.emit (events.js:188:7)
name: 'RequestError',
message: 'Timeout: Request failed to complete in 50000ms',
code: 'ETIMEOUT',
number: 'ETIMEOUT',
lineNumber: undefined,
state: undefined,
class: undefined,
serverName: undefined,
procName: undefined }
Issue Analytics
- State:
- Created 7 years ago
- Comments:6
Top Results From Across the Web
Firebase Rest API Streaming error: Connection gets closed ...
Firebase Rest API Streaming error: Connection gets closed when child has huge data in it - Stack Overflow. Collectives™ on Stack Overflow – ......
Read more >Troubleshooting Streaming - Oracle Help Center
This error occurs because the PutMessages call sent to the service is too large. The size must be less than or equal to...
Read more >Troubleshooting Kinesis Data Streams Consumers
Some Kinesis Data Streams Records are Skipped When Using the Kinesis Client Library. The most common cause of skipped records is an unhandled...
Read more >80020009 error when you retrieve data - SQL Server
SQL Server is sending back the data on the wire and the client is essentially receiving a stream of bits read sequentially on...
Read more >Failure on writing large amount of data to excel even ... - GitHub
I am using exceljs with Streams but it still fails to export large amount of data due to Heap out of memory issue....
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I think streaming should be free from whatever set in
requestTimeout
This is most likely because requests didn’t implement any kind of pausing or back pressure. This is now implemented in v5