question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Streaming unzipping crashes

See original GitHub issue

Issue Description

How to reproduce This zip seems to reproduce the issue: https://sethealth-customers.b-cdn.net/repo.zip

Try to parse the ZIP file with a single huge chunk (just trying to reproducee the issue):

import {Unzip, UnzipInflate} from 'fflate';
import {readFileSync} from 'fs';


const readFile = (file) => {
  return new Promise((resolve, reject) => {
    const chunks = [];
    file.ondata = (err, chunk, final) => {
      if (err) {
        reject(reject);
      }
      if (chunk) {
        chunks.push(chunk);
      }
      if (final) {
        resolve(concatBuffers(chunks));
      }
    };
    file.start();
  });
}

const concatBuffers = (arrays) => {
  const totalLength = arrays.reduce(
    (acc, value) => acc + value.byteLength,
    0
  );
  const output = new Uint8Array(totalLength);
  let offset = 0;
  for (let array of arrays) {
    output.set(array, offset);
    offset += array.byteLength;
  }
  return output;
};


const chunk = new Uint8Array(readFileSync("repo.zip"));
const promises = [];
const unzipper = new Unzip((handler) => {
  promises.push((async () => {
    return {
      name: handler.name,
      stream: await readFile(handler),
    }
  })());
});
unzipper.register(UnzipInflate);
unzipper.push(chunk, false);
unzipper.push(new Uint8Array(), true);
const output = await Promise.all(promises);
console.log(output);

^ This code will crash with “Invalid zip data”.

The problem

fflate seems to unzip correctly this file when using unzipSync(), but fails in stream mode:

import {readFileSync} from 'fs';
import {unzipSync} from 'fflate';

const chunk = new Uint8Array(readFileSync("repo.zip"));
const output = unzipSync(chunk);
console.log(output);

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
manucorporatcommented, Jun 3, 2021

Hey! i guess i change how we generate to not bee in streaming, but it would still be problematic in case someone uploads a streaming generated .zip that has this issue…

Thanks a lot of your help! feel free to close issue 😃

0reactions
101arrowzcommented, Jun 3, 2021

i guess the chances of it happening are more likely for big files, or a zip within a zip

Pretty much, but the magic number was specifically designed to be quite rare, so it’s unfortunate this is a problem.

Out of curiosity, do you know if .tar has a similar problem?

TAR is a far superior format to ZIP IMO. No, it does not suffer from this issue during decompression; however, it can’t be streamed in compression.

Since it seems that you’re able to choose .zip or .tar, it looks like you have control over the file creation process. In that case, note that the zip command will not yield a streamed ZIP. It’s important to note that ZIPs created without streaming can always be decompressed streaming, but ZIPs created with streaming can usually be decompressed streaming. So if you can force the ZIP to be created in a non-streaming manner, you will never face this problem. You can use InfoZIP to do this: zip -r myzip.zip folder/.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Windows 11 zipped folders / files immediately crashes File
When I right click any zipped folder, it immediately crashes Windows Explorer, but then everything comes back. I can't unzip any zipped file ......
Read more >
Unzip buffer with large data length is crashing - Stack Overflow
This is the function I am using to unzip buffer. string unzipBuffer(size_t decryptedLength, unsigned char * decryptedData) { z_stream stream ...
Read more >
Zip files crashes Windows Explorer | Windows 11 Forum
I have a very duplicatable issue where Windows Explorer constantly crashes (restarts) when clicking on or attempting to unzip a file.
Read more >
[SOLVED] - Right Clicking a Zip file causes explorer Crash
The problem is this, the NAS works flawlessly UNLESS I right click on a zip file on the drive and then it causes...
Read more >
Archive Utility freezes when trying to unzip a zip file
Fortunately I also have Stuffit Expander on my Mac and I used that to unzip the archives. It had no problems and the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found