Possible to 'pipe' file from one client to another?

Hi, I’m looking for a way to ‘pipe’ from one sftp client to another. I’m thinking to leverage either this.sftpClient.fastGet(remotePath, localPath) where remotePath is from one client and localPath is from another client or this.sftpClient.fastPut(localPath, remotePath) in a similar manner. Is this possible? The reason for this is, in a serverless environment, I don’t want to hold the file in memory while I download from one server and upload to another.

[theophilusx]

Theoretically, I think you could get this to work using streams. You could try using get() and put() as both of these will accept a stream object as the destination/source. The fastPut() and fastGet() methods only accept string paths as destination/source.

I’ve never tried this and suspect there will be some complexity associated with the creation of the streams and managing errors effectively. It isn’t something which fits cleanly with a promise based API, but you might be able to get it to work. On the other hand, it might be easier to implement this using just ssh2, which you have more direct access to the underlying ssh2 API. However, this would mean you need to use an event model rather than a promise based approach.

[alanhg]

If you are wanting to pipe from one ssh2-sftp-client to another, then you will need to create an intermediate stream object which will work as a bridge between the two ssh2-sftp-client instances. The details on how to do this are outlined in the streams section of the node documents. You can probably also find a NPM package which implements what you need - at a very basic level, what you are after is almost identical to a transform stream except no data is transformed - it is just passed through. One you have that stream, it would be the destination stream for the get() method in the first ssh2-sftp-client and the source in the put() call for the second. You will need to implement some reasonably sophisticated error handling as your architecture and underlying design is inherently fragile - at the very least, you now have doubled the possible errors (each client has potential to fail) and no easy way to recover.

Note that current development version of ssh2-sftp-client has added the ability to get a readStream and a writeStream directly from the client. This would appear to make what you are trying to do easier, but that appearance is deceptive. However, you may find it conceptually easier to work with. The important point to note is that once you use these low level calls, your no longer using a Promise or async/await API, you are using an event API and you will be totally responsible for managing the events, cleaning up and releasing resources like file handles etc. It will be very similar to using the lower level ssh2 module directly where you will be responsible for dealing with a lot more of the interface. Note that one of the reasons I did not add the low level 'createReadStream and createWriteStream methods previously is because these are inherently event based processing models and event models don’t work well with promise/async based models. In reality, I still think that if you want to work at this low level, ultimately, your far better off just using the ssh2 module and not use ssh2-sftp-client at all as mixing event and promise based APIs does not facilitate robust code. At the same time, adding these calls to the API is not complicated and if people want to shoot themselves in the foot with them, I’m happy to provide the ‘foot gun’.

I suspect a lot of the challenges your facing here could be resolved with a bit of time doing a bit of a deep dive into understanding Node streams. Everything you actually need is part of the basic node API. Some time spent experimenting with the streams API would likely help refine your design and help with implementation of effective error handling.

Thanks a lot. As you said, the package ssh2-sftp-client support stream, I just need to understand node’s handling of the stream.

I have tried and succeed. Thank you.