Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

zstd compression hanging on Windows with large files

See original GitHub issue

The caching action always hangs when using actions/cache@master on Windows in my test runs using a random 150 MB file.

To dig in more, I created a separate repo that runs the compression commands. On a 1 MB file, it works fine:

2020-05-08T19:01:50.9215932Z bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.f-ipp
2020-05-08T19:01:51.7976258Z Create random file
2020-05-08T19:01:51.8560894Z 1+0 records in
2020-05-08T19:01:51.8563149Z 1+0 records out
2020-05-08T19:01:51.8568671Z 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011073 s, 94.7 MB/s
2020-05-08T19:01:51.8578782Z Running zstd standalone
2020-05-08T19:01:52.3317293Z 
2020-05-08T19:01:52.3325676Z 
2020-05-08T19:01:52.3331145Z     Directory: D:\a\test-zstd\test-zstd
2020-05-08T19:01:52.3376822Z 
2020-05-08T19:01:52.3392726Z Mode                 LastWriteTime         Length Name
2020-05-08T19:01:52.3395053Z ----                 -------------         ------ ----
2020-05-08T19:01:52.3412014Z -a---            5/8/2020  7:01 PM        1048613 file.zstd
2020-05-08T19:01:52.3412805Z Running tar standalone
2020-05-08T19:01:52.3608654Z -a---            5/8/2020  7:01 PM        1050112 file.tar
2020-05-08T19:01:52.3622919Z Running tar + zstd in two steps
2020-05-08T19:01:52.4062645Z -a---            5/8/2020  7:01 PM        1050624 file_combined.tzst
2020-05-08T19:01:52.4064236Z Running tar + zstd using --use-compress-program
2020-05-08T19:01:52.4668289Z -a---            5/8/2020  7:01 PM        1049135 file.tzst
2020-05-08T19:01:52.4670493Z Done

But on a 150 MB file, it hangs until it I cancel the action:

2020-05-08T19:03:52.6081814Z bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.f-ipp
2020-05-08T19:03:53.4979676Z Create random file
2020-05-08T19:03:53.6897822Z 150+0 records in
2020-05-08T19:03:53.6909594Z 150+0 records out
2020-05-08T19:03:53.6912241Z 157286400 bytes (157 MB, 150 MiB) copied, 0.147127 s, 1.1 GB/s
2020-05-08T19:03:53.6925733Z Running zstd standalone
2020-05-08T19:03:54.3540113Z 
2020-05-08T19:03:54.3584708Z 
2020-05-08T19:03:54.3616876Z     Directory: D:\a\test-zstd\test-zstd
2020-05-08T19:03:54.3634999Z 
2020-05-08T19:03:54.3648596Z Mode                 LastWriteTime         Length Name
2020-05-08T19:03:54.3650325Z ----                 -------------         ------ ----
2020-05-08T19:03:54.3650520Z -a---            5/8/2020  7:03 PM      157290014 file.zstd
2020-05-08T19:03:54.3650643Z Running tar standalone
2020-05-08T19:03:54.5655674Z -a---            5/8/2020  7:03 PM      157287936 file.tar
2020-05-08T19:03:54.5658027Z Running tar + zstd in two steps
2020-05-08T19:03:55.0461307Z -a---            5/8/2020  7:03 PM      157292032 file_combined.tzst
2020-05-08T19:03:55.0498037Z Running tar + zstd using --use-compress-program
2020-05-08T19:06:33.2001585Z ##[error]The operation was canceled.

Test repo - https://github.com/dhadka/test-zstd Example run that’s hanging - https://github.com/dhadka/test-zstd/actions/runs/99414776

CC @imbsky

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:27 (14 by maintainers)

Top GitHub Comments

7reactions

lazkacommented, Jul 24, 2020

I did a quick test with my workflow to add (a cygwin based, so a bit hacky) gnu tar to PATH so zstd is used on Windows.

gzip saveCache: 84 seconds
zstd saveCache: 27 seconds

So I think it’s worth trying to fix this 😃

2reactions

lazkacommented, Jul 25, 2020

I’ve filed a bug with libarchive: https://github.com/libarchive/libarchive/issues/1419

Top Results From Across the Web

ZStd hangs on files on random · Issue #909 · facebook/zstd

Usually this works fine, but randomly zstd does hang and never finishes compression. The state of the process then always remains as 'T'....

zstd 1.5.1 Manual

The zstd compression library provides in-memory compression and decompression ... ensure that the window size is large enough to contain the entire source....

Better Compression with Zstandard

Compressing each file discretely at zstd level 12 yields 9,177,410 bytes of data. But if we produce a 131,072 byte dictionary by training...

Zstd: Everything to Know About Zstandard Compression

Zstd : Everything to Know About Zstandard Compression. Internet users often encounter large files and documents which require compression to ease download and ......

zstd command man page

If no files are given or file is -, zstd reads from standard input and writes the processed data to standard output. zstd...