Nock record outputs array of hex values instead of plain JSON when response is encoded

Related: https://github.com/nock/nock/issues/457#issuecomment-419480974

What is the expected behavior? When recording responses which are compressed and chunked, e.g. Content-Encoding: 'gzip' and Transfer-Encoding: 'chunked', I was expecting the generated fixture to decompress and combine the chunks into the human readable response; this would then allow us to remove sensitive information from the response and modify it to satisfy scenarios which are hard to reproduce against the real APIs.

What is the actual behavior? The fixtures response is an array of hex values.

Possible solution I’m working around this by modifying the nockDefs like this:

onst { ungzip } = require('node-gzip');

try {
  if (Array.isArray(def.response)) { // NOTE: This is a v. naive check
    def.response = JSON.parse(
      (await ungzip(Buffer.from(def.response.join(''), 'hex'))).toString(
        'utf-8'
      )
    );
  }
} catch (ex) {
  console.warn('Failed to decode response');
}

If this is in fact a bug then it’d be good to fix it in nock itself – if it’s as intended, i.e. the nock is technically returning the exact same response as the real server, then perhaps an option could be passed to nock record / nock.back to have it output the decompressed response?

How to reproduce the issue The issue can be reproduced by recording an API which returns compressed, chunked JSON.

Does the bug have a test case? https://github.com/richardscarrott/nock-record-chunked-encoding

Versions

Software	Version(s)
Nock	9.6.1
Node	10.5.0

[protoEvangelion]

@richardscarrott thanks for pointing me in the right direction with this. The additional problem I had to figure out was that I needed to re-gzip after normalizing. This was because https://github.com/octokit/rest.js expects the response to be gzipped.

To work with nockBack, without changing nock internally, here is what I did in the afterRecord callback:

function decodeBuffer(fixture) {
  // Decode the hex buffer that nock made
  const response = isArray(fixture.response) ? fixture.response.join('') : fixture.response

  try {
      const decoded = Buffer.from(response, 'hex')
      var unzipped = zlib.gunzipSync(decoded).toString('utf-8')
  } catch (err) {
      throw new Error(`Error decoding nock hex:\n${err}`)
  }

  return JSON.parse(unzipped)
}

function afterRecord(fixtures) {
  const normalizedFixtures = fixtures.map(fixture => {
      fixture.response = decodeBuffer(fixture)

      // do normalization stuff
      // Re-gzip to keep the @octokit/rest happy
      const stringified = JSON.stringify(fixture.response)
      const zipped = zlib.gzipSync(stringified)

      fixture.response = zipped

      return fixture
  })

  return normalizedFixtures
}

[richardscarrott]

FYI, this is the complete afterNock function I’m using

const parseNockDefs = (
  nockDefs: (nock.NockDefinition & { rawHeaders: string[] })[]
) => {
  return nockDefs.map(def => {
    try {
      const headers = def.rawHeaders.reduce<Dictionary<string>>(
        (acc, curr, i, arr) => {
          if (i % 2 === 0) {
            acc[arr[i].toLowerCase()] = arr[i + 1].toLowerCase();
          }
          return acc;
        },
        {}
      );
      if (
        headers['transfer-encoding'] === 'chunked' &&
        headers['content-encoding'] === 'gzip' &&
        Array.isArray(def.response)
      ) {
        def.response = JSON.parse(
          gunzipSync(Buffer.from(def.response.join(''), 'hex')).toString(
            'utf-8'
          )
        );
        def.rawHeaders = Object.entries(headers).flatMap(([key, value]) => {
          if (key === 'transfer-encoding' || key === 'content-encoding') {
            return [];
          }
          return [key, value];
        });
      }
    } catch (ex) {
      console.warn('Failed to decode response');
    }
    return def;
  });
};