question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add "extra fields" support

See original GitHub issue

Would be awesome to add support for reading&writing the “extra fields” from the specifications: https://datatracker.ietf.org/doc/html/rfc1952#page-8

 2.2. File format

      A gzip file consists of a series of "members" (compressed data
      sets).  The format of each member is specified in the following
      section.  The members simply appear one after another in the file,
      with no additional information before, between, or after them.

   2.3. Member format

      Each member has the following structure:

         +---+---+---+---+---+---+---+---+---+---+
         |ID1|ID2|CM |FLG|     MTIME     |XFL|OS | (more-->)
         +---+---+---+---+---+---+---+---+---+---+

      (if FLG.FEXTRA set)

         +---+---+=================================+
         | XLEN  |...XLEN bytes of "extra field"...| (more-->)
         +---+---+=================================+

      (if FLG.FNAME set)

         +=========================================+
         |...original file name, zero-terminated...| (more-->)
         +=========================================+

      (if FLG.FCOMMENT set)

         +===================================+
         |...file comment, zero-terminated...| (more-->)
         +===================================+

      (if FLG.FHCRC set)

         +---+---+
         | CRC16 |
         +---+---+

         +=======================+
         |...compressed blocks...| (more-->)
         +=======================+

           0   1   2   3   4   5   6   7
         +---+---+---+---+---+---+---+---+
         |     CRC32     |     ISIZE     |
         +---+---+---+---+---+---+---+---+

2.3.1.1. Extra field

         If the FLG.FEXTRA bit is set, an "extra field" is present in
         the header, with total length XLEN bytes.  It consists of a
         series of subfields, each of the form:

            +---+---+---+---+==================================+
            |SI1|SI2|  LEN  |... LEN bytes of subfield data ...|
            +---+---+---+---+==================================+

         SI1 and SI2 provide a subfield ID, typically two ASCII letters
         with some mnemonic value.  Jean-Loup Gailly
         <gzip@prep.ai.mit.edu> is maintaining a registry of subfield
         IDs; please send him any subfield ID you wish to use.  Subfield
         IDs with SI2 = 0 are reserved for future use.  The following
         IDs are currently defined:

Thanks in advance!

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:17 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
101arrowzcommented, Sep 18, 2021

The reason it looks like garbage is because the IDs are not ASCII values. You actually can use ASCII letters with my code but the specification does not limit the IDs to ASCII so I made it more generic by letting it be a number. Also the second “character” (SI2, the most significant byte in the ID) cannot be zero because that is reserved. So it’s just easier from a coding point of view to take a number, but obviously you can use ASCII as well:

const asciiToID = str => {
  if (str.length != 2) throw new TypeError('extra ID must be two characters long');
  return str.charCodeAt(0) | (str.charCodeAt(1) << 8);
}

const gzipWithExtraFields = insertExtraFields(gzipSync(yourData), {
    [asciiToID('ab')]: new Uint8Array([1, 2, 3, 4]),
    [asciiToID('cd')]: new TextEncoder().encode("Hello world")
});
1reaction
101arrowzcommented, Sep 3, 2021

I’ve gotten a basic implementation working but it’s added a pretty big chunk to bundle size for GZIP support (about 1kB, which is a lot when the total size for GZIP support was previously 5kB). If this were a compiled language, I could just add a feature flag and avoid adding the bloat unless it’s needed, but GZIP is one of the most popular parts of the library and minimizing its bundle size is very important to me, so unfortunately I think it’s not going to be released.

However, I just took another look at your initial message to me and it seems you want a CRC32 checksum of the uncompressed contents of the file, which is included by default in the spec. If so, you can just read the values between 8 and 4 bytes from the end of the file. For example:

import { gzipSync } from 'fflate';

// Data doesn't have to be from fflate, it just needs to be gzipped
const gzipData = gzipSync(someData);

// When you want to calculate the checksum:
const len = gzipData.length;
const checksum = (
  gzipData[len - 8]
  | (gzipData[len - 7] << 8)
  | (gzipData[len - 6] << 16)
  | (gzipData[len - 5] << 24)
) >>> 0;
console.log(checksum); // CRC32 of the uncompressed contents

By the way, most untar utilities that support .tar.gz files will call the zlib library to decompress the GZIP data, and thereby automatically verify that the CRC32 of the uncompressed data matches the CRC32 in the GZIP footer. So if you’re doing this to avoid corruption, you don’t need to worry; it’s already handled for you.

So sorry for all the confusion and for taking so long on this. Let me know if you have any questions.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Adding custom fields to your tickets and support request form
In Admin Center, click Objects and rules in the sidebar, then select Tickets > Fields. Click Add field. Select a field type, then...
Read more >
Custom Fields for Awesome Support
The Custom Fields add-on for Awesome Support enable regular users to easily add new fields to the ticket form. The regular ticket form...
Read more >
Extra Fields for Jira Service Management
Empower your customers. Customize request view and make extra (read only) fields visible to customers in Service Management portal.
Read more >
How to add extra fields? - WordPress.org
Hi all, Can I add the registration form to extra fields other than username ... If on regular registration forms, then no, but...
Read more >
Extra Fields
To create or delete extra fields, a user must have the Extra Field Staging Access user right. No special user rights are required...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found