Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

sftp.put hangs after creating the file

See original GitHub issue

Hi,

I’m trying to do a simple upload of a small CSV file I’m generating on the fly. Here is the relevant portion of the code

const data = Buffer.from(csv)
const sftp = new Client()
try {
  await sftp.connect({
    host: HOST,
    username: USERNAME,
    password: PASSWORD,
    port: PORT,
    debug: console.info
  })
  await sftp.put(data, '/path/to/file.csv')
} catch (e) {
  console.error(e)
} finally {
  await sftp.end()
}

When doing so, most of the time the upload will hang, with logs showing:

CLIENT[sftp]: exists -> /path/to/file.csv
CLIENT[sftp]: stat -> /path/to/file.csv
DEBUG[SFTP]: Outgoing: Writing STAT
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:60,padLen:9,remainLen:48
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: ATTRS
CLIENT[sftp]: stats <-  {
 "mode": 33206,
 "permissions": 33206,
 "uid": 1026,
 "gid": 100,
 "size": 0,
 "atime": 1595344292,
 "mtime": 1595347428
}
CLIENT[sftp]: Removing listener from error
CLIENT[sftp]: Removing listener from close
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
CLIENT[sftp]: exists <-  {
 "mode": 33206,
 "uid": 1026,
 "gid": 100,
 "size": 0,
 "accessTime": 1595344292000,
 "modifyTime": 1595347428000,
 "isDirectory": false,
 "isFile": true,
 "isBlockDevice": false,
 "isCharacterDevice": false,
 "isSymbolicLink": false,
 "isFIFO": false,
 "isSocket": false
}
CLIENT[sftp]: put remote path info  {
 "path": "/path/to/file.csv",
 "type": "-",
 "valid": true
}
CLIENT[sftp]: put -> /path/to/file.csv
DEBUG[SFTP]: Outgoing: Writing OPEN
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
CLIENT[sftp]: put source is a buffer
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:44,padLen:17,remainLen:32
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: HANDLE
DEBUG[SFTP]: Outgoing: Writing FSETSTAT
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:44,padLen:6,remainLen:32
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: STATUS
DEBUG[SFTP]: Outgoing: Writing WRITE
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:28,padLen:18,remainLen:16
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_WINDOW_ADJUST (0, 98482)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)

and it will stay like so forever

Sometime, completely at random, it will go through and in thise case the logs will be like that:

CLIENT[sftp]: exists -> /path/to/file.csv
CLIENT[sftp]: stat -> /path/to/file.csv
DEBUG[SFTP]: Outgoing: Writing STAT
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:60,padLen:9,remainLen:48
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: ATTRS
CLIENT[sftp]: stats <-  {
 "mode": 33206,
 "permissions": 33206,
 "uid": 1026,
 "gid": 100,
 "size": 0,
 "atime": 1595344292,
 "mtime": 1595347522
}
CLIENT[sftp]: Removing listener from error
CLIENT[sftp]: Removing listener from close
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
CLIENT[sftp]: exists <-  {
 "mode": 33206,
 "uid": 1026,
 "gid": 100,
 "size": 0,
 "accessTime": 1595344292000,
 "modifyTime": 1595347522000,
 "isDirectory": false,
 "isFile": true,
 "isBlockDevice": false,
 "isCharacterDevice": false,
 "isSymbolicLink": false,
 "isFIFO": false,
 "isSocket": false
}
CLIENT[sftp]: put remote path info  {
 "path": "/path/to/file.csv",
 "type": "-",
 "valid": true
}
CLIENT[sftp]: put -> /path/to/file.csv
DEBUG[SFTP]: Outgoing: Writing OPEN
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
CLIENT[sftp]: put source is a buffer
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:44,padLen:17,remainLen:32
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: HANDLE
DEBUG[SFTP]: Outgoing: Writing FSETSTAT
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:44,padLen:6,remainLen:32
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: STATUS
DEBUG[SFTP]: Outgoing: Writing WRITE
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:28,padLen:18,remainLen:16
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_WINDOW_ADJUST (0, 98482)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:28,padLen:18,remainLen:16
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_WINDOW_ADJUST (0, 102171)
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: Decrypting
DEBUG: Parser: pktLen:44,padLen:6,remainLen:32
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: Decrypting
DEBUG: Parser: HMAC size:32
DEBUG: Parser: IN_PACKETDATAVERIFY
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
DEBUG[SFTP]: Parser: Response: STATUS
DEBUG: Parser: IN_PACKETBEFORE (expecting 16)
DEBUG[SFTP]: Outgoing: Writing CLOSE
DEBUG: Outgoing: Writing CHANNEL_DATA (0)
CLIENT[sftp]: Removing listener from error
CLIENT[sftp]: Removing listener from close
DEBUG: Outgoing: Writing DISCONNECT (BY_APPLICATION)
CLIENT[sftp]: Removing listener from error

I’m currently running Node v14.4.0, so I’m wondering if this could be linked to https://github.com/mscdex/ssh2-streams/issues/156 ?

You mention in this issue that using a concurrency of 1 it’s working all the time, and my understanding was that client.put is running in the same configuration which is why I’m opening this issue here as I’m not sure it’s the same.

Do you have any idea what I could do to ensure a more consistent result while running Node >= v14.1.0 ?

Thanks

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:17 (10 by maintainers)

github_iconTop GitHub Comments

3reactions
theophilusxcommented, Jan 2, 2021

Update: I have recently tested against the new ssh2 version. While the issues specific to fastPut and fastGet appear to be resolved in the new ssh2 version, there are still some issues with put/get which appear to be related to streams. The seems to still be a bit of work required on the new version of ssh2 before it will be ready for a release.

I have tested the existing 0.8.9 version of ssh2 against latest node v15.4.0. The changes introduced in node v14.x which caused problems with ssh2 have been rolled back. This means that ssh2 and ssh2-sftp-client seems to work OK with node version 15.4.0. However, the changes which were rolled back did provide significant performance improvements in other areas, so it is likely that at some point they will be re-applied in one fashion or another. Hopefully, the new version of ssh2 will be released before this occurs.

3reactions
theophilusxcommented, Oct 24, 2020

The ssh2 module is being re-written to address this and some other limitations of that module. The ssh2-streams module is also being deprecated. This means a re-write of ssh2-sftp-client will be necessary to work with the updated ssh2 module.

I have created the branch version-6, which will be used to do the re-write.

Read more comments on GitHub >

github_iconTop Results From Across the Web

sftp session occasionally hang during file transfer - any advice
If the "hang" is in during the file transfer it is most likely an issue with the sftp portion file manipulation and not...
Read more >
SFTP Connection hangs during put operation - Stack Overflow
I'm attempting to simply push a local file to an FTP destination. However my script always seems to hang during the "put" operation....
Read more >
SFTP hangs after correct authentication, SSH works
It appears that the connection is successful but something gets stuck along the way. Running SFTP command from inside raspberry pi seems to...
Read more >
SFTP upload hanging – IDEs Support (IntelliJ Platform)
Hi,I'm currently using PhpStorm version 6.0.3.Just today, my sftp uploads to my deployment server is not working. It just hangs saying...
Read more >
SFTP hangs in SFTP Begin Session service and all threads ...
This problem has been know to occur when the remote SFTP server has put the SFTP client IP Address on a banned list....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Why can't this create a new file on the remote server? "no such file or directory"

Next page

Add ignore pattern in upload/downloadDir method.