Streaming unzipping crashes

How to reproduce This zip seems to reproduce the issue: https://sethealth-customers.b-cdn.net/repo.zip

Try to parse the ZIP file with a single huge chunk (just trying to reproducee the issue):

import {Unzip, UnzipInflate} from 'fflate';
import {readFileSync} from 'fs';


const readFile = (file) => {
  return new Promise((resolve, reject) => {
    const chunks = [];
    file.ondata = (err, chunk, final) => {
      if (err) {
        reject(reject);
      }
      if (chunk) {
        chunks.push(chunk);
      }
      if (final) {
        resolve(concatBuffers(chunks));
      }
    };
    file.start();
  });
}

const concatBuffers = (arrays) => {
  const totalLength = arrays.reduce(
    (acc, value) => acc + value.byteLength,
    0
  );
  const output = new Uint8Array(totalLength);
  let offset = 0;
  for (let array of arrays) {
    output.set(array, offset);
    offset += array.byteLength;
  }
  return output;
};


const chunk = new Uint8Array(readFileSync("repo.zip"));
const promises = [];
const unzipper = new Unzip((handler) => {
  promises.push((async () => {
    return {
      name: handler.name,
      stream: await readFile(handler),
    }
  })());
});
unzipper.register(UnzipInflate);
unzipper.push(chunk, false);
unzipper.push(new Uint8Array(), true);
const output = await Promise.all(promises);
console.log(output);

^ This code will crash with “Invalid zip data”.

The problem

fflate seems to unzip correctly this file when using unzipSync(), but fails in stream mode:

import {readFileSync} from 'fs';
import {unzipSync} from 'fflate';

const chunk = new Uint8Array(readFileSync("repo.zip"));
const output = unzipSync(chunk);
console.log(output);

[manucorporat]

Hey! i guess i change how we generate to not bee in streaming, but it would still be problematic in case someone uploads a streaming generated .zip that has this issue…

Thanks a lot of your help! feel free to close issue 😃

[101arrowz]

i guess the chances of it happening are more likely for big files, or a zip within a zip

Pretty much, but the magic number was specifically designed to be quite rare, so it’s unfortunate this is a problem.

Out of curiosity, do you know if .tar has a similar problem?

TAR is a far superior format to ZIP IMO. No, it does not suffer from this issue during decompression; however, it can’t be streamed in compression.

Since it seems that you’re able to choose .zip or .tar, it looks like you have control over the file creation process. In that case, note that the zip command will not yield a streamed ZIP. It’s important to note that ZIPs created without streaming can always be decompressed streaming, but ZIPs created with streaming can usually be decompressed streaming. So if you can force the ZIP to be created in a non-streaming manner, you will never face this problem. You can use InfoZIP to do this: zip -r myzip.zip folder/.