unsupported compression method: 9

We recently had a report against Etcher that a user was getting a unsupported compression method: 9 error when trying to extract a particular archive.

I downloaded the archive myself, and managed to trace the problem to yauzl. When I clone this repo and run node examples/dump.js --no-contents /home/andrew/Downloads/RPI-CNC-V400.zip I get this output:

Entry {
  versionMadeBy: 45,
  versionNeededToExtract: 45,
  generalPurposeBitFlag: 0,
  compressionMethod: 9,
  lastModFileTime: 25321,
  lastModFileDate: 18998,
  crc32: 2459691682,
  compressedSize: 2359888958,
  uncompressedSize: 8006926336,
  fileNameLength: 14,
  extraFieldLength: 12,
  fileCommentLength: 0,
  internalFileAttributes: 0,
  externalFileAttributes: 32,
  relativeOffsetOfLocalHeader: 0,
  fileName: 'RPI_CNC_V4.img',
  extraFields: [ { id: 1, data: <Buffer 00 00 40 dd 01 00 00 00> } ],
  fileComment: '',
  comment: '' }
2017-01-22T12:23:18.000Z

Without the --no-contents flag I get:

Entry {
  versionMadeBy: 45,
  versionNeededToExtract: 45,
  generalPurposeBitFlag: 0,
  compressionMethod: 9,
  lastModFileTime: 25321,
  lastModFileDate: 18998,
  crc32: 2459691682,
  compressedSize: 2359888958,
  uncompressedSize: 8006926336,
  fileNameLength: 14,
  extraFieldLength: 12,
  fileCommentLength: 0,
  internalFileAttributes: 0,
  externalFileAttributes: 32,
  relativeOffsetOfLocalHeader: 0,
  fileName: 'RPI_CNC_V4.img',
  extraFields: [ { id: 1, data: <Buffer 00 00 40 dd 01 00 00 00> } ],
  fileComment: '',
  comment: '' }
2017-01-22T12:23:18.000Z
/tmp/yauzl/examples/dump.js:25
        if (err) throw err;
                 ^

Error: unsupported compression method: 9
    at /tmp/yauzl/index.js:511:25
    at /tmp/yauzl/index.js:618:5
    at /tmp/node_modules/fd-slicer/index.js:32:7
    at FSReqWrap.wrapper [as oncomplete] (fs.js:682:17)

Are there any plans for yauzl to support this compression method 9? Wikipedia says https://en.wikipedia.org/wiki/Zip_(file_format)#Compression_methods and unzip -v /home/andrew/Downloads/RPI-CNC-V400.zip (on Linux) reports:

Archive:  /home/andrew/Downloads/RPI-CNC-V400.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
8006926336  Def64N 2359888958  71% 2017-01-22 12:23 929beaa2  RPI_CNC_V4.img
--------          -------  ---                            -------
8006926336         2359888958  71%                            1 file

(and unzip is able to extract the file correctly).

[thejoshwolfe]

Here’s what I know:

• compression method 9 is called DEFLATE64, which is invented by PKWARE, the inventor in the zip file format​.
• DEFLATE64 is a proprietary algorithm, which doesn’t necessarily mean open source software can’t implement it, but it does make open source software sad to implement it (the .mp3 format is/was this kind of situation fwiw.).
• zlib does not support DELFATE64 despite how easy it would be for them to support it. https://zlib.net/zlib_faq.html#faq40 their cited reasons are its proprietary specification and the algorithm being generally ineffective.
• yauzl relies on zlib built into the core of node.js for decompression, and yauzl doesn’t have any native code dependencies (other than the node core). so no gyp or waf or anything when npm install yauzl.
• implementing a decompression algorithm in javascript is the wrong tool for the job due to performance and lack of “basic” primitive features like 64bit integers.
• In the ~2 years of yauzl’s “mainstream” adoption, this is the first report I’ve gotten of a compression method other than the two common ones 0 and 8.

All these points are arguments against taking action on this issue and simply resolving it as wontfix. However, I’m willing to consider ways to improve yauzl’s feature set and adoption, and if that means either adding a native code dependencies or implementing a decompression algorithm in javascript, I’ll at least consider it a little.

What i’d like to do next is do an even more thorough investigation of which zipfile reading libraries support compression ​methods other than 0 and 8.

It may take a week or so before I have time to do this, so in the immediate future, the direct answer to your question is: No, there are no plans to support compression methods other than​ 0 and 8.

[alexwlchan]

Thanks for the detailed info, this was super useful and saved me a bunch of research!

7-Zip: probably supports lots of compression methods probably including 9.

FWIW, I’ve been having this issue with an archive created with 7-Zip (I’m not sure which version, but probably new-ish). It does seem to support compression method 9. 😭