Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CopyFilesOverSSHV0 unbearably slow

See original GitHub issue

Type: Bug

Enter Task Name: CopyFilesOverSSHV0

Environment

Server - Azure Pipelines
- If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
  - Account Name: numberjak
  - Team Project: Magpie
  - Release Definition Name: DigitalOcean
  - Release Name: Release-4
Agent - Hosted or Private:
- If using Hosted agent, provide agent queue name: WIN-DF00C5S5J33 (this is the machine name. hope that’s what you mean)

Issue Description

My release pipeline copies build artifacts to a remote server on DigitalOcean using the CopyFilesOverSSHV0 task. The files are around 500MB total in size. However the process of copying these files to the remote server consistently takes around 43 minutes, which is obviously ridiculously slow.

Running some benchmarks on the remote server the files are being sent to, I can see the problem is not on DigitalOcean server:

I/O Benchmarks:

root@blahblah:~# dd if=/dev/zero of=benchmark bs=1M count=2048 conv=fdatasync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.83064 s, 759 MB/s
root@blahblah:~# dd if=/dev/zero of=benchmark bs=64K count=32K conv=fdatasync
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.3467 s, 642 MB/s

Download speed benchmark:

root@blahblah:~# wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip
--2020-06-19 16:15:09--  http://speedtest.wdc01.softlayer.com/downloads/test500.zip
Resolving speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)... 158.85.230.20, 2607:f0d0:3006:6c::4
Connecting to speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)|158.85.230.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 524288000 (500M) [application/zip]
Saving to: ‘/dev/null’

/dev/null                                         100%[=============================================================================================================>] 500.00M  46.6MB/s    in 30s     

2020-06-19 16:15:48 (16.7 MB/s) - ‘/dev/null’ saved [524288000/524288000]

Task logs

ReleaseLogs_5.zip

Issue Analytics

State:
Created 3 years ago
Comments:15 (6 by maintainers)

Top GitHub Comments

2reactions

alexander-smolyakovcommented, Jul 7, 2020

Reopen this, since the fix is waiting to be deployed. We will let you know once the fix will be published.

0reactions

github-actions[bot]commented, Nov 17, 2021

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

Top Results From Across the Web

Slow data transfer using SSH - linux - Super User

SSH works fine for console programs. But data transfer is slow, which is very annoying when transferring large files. I tried two different ......

Rsync transfer via SSH is very slow - backup - Server Fault

rsync can become very slow (not the transfer itself) with millions of files, because rsync intially checks the filelist src<->dst to decide ...

CopyFilesOverSSH very slow · Issue #3485 - GitHub

copying files from build machine to deployment server using pscp works fine. copying files using VSTS CopyFilesOverSSH is VERY slow (12 minutes for...

Copying large amount of files extremely slow in Linux vs ...

I realised that in Windows I always get speeds of 60-80 MB/s while on Linux it starts fast but soon slows down to...

Why is scp so slow and how to make it faster?

You could use rsync (over ssh ), which uses a single connection to transfer all the source files. rsync -avP cap_* user@host:dir.