question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

utf8 encoding on responses corrupts data

See original GitHub issue

Currently I don’t see any ways to disable this behaviour, but binary data that is sent gets corrupted, i.e., downloading a public spreadsheet file is impossible, because it’s corrupted as soon as you get it.

drive.files.export({
    auth: API_KEY,
    fileId: "<mySpreadSheet>",
    mimeType: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
}, (err: any, result: any) => {

at this point result is utf8.

Workaround: make a https request and don’t set the encoding to utf8. Bonus points for being lazy and finding the url via debugging this library.

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:11 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
rsxdalvcommented, Oct 19, 2017

Ok, so I wrote this to satisfy my use case:

import * as google from "googleapis";

const drive = google.drive("v3");

import { writeFileSync } from "fs";

const buffers = new Array<Buffer>();

drive.files.export({
    auth: key,
    fileId: "<file id>",
    mimeType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
})
    .on('data', (buffer: Buffer) => {
        buffers.push(buffer);
    })
    .on('end', function () {
        const result = Buffer.concat(buffers);
        writeFileSync("./test.xlsx", result);
        console.log('Done');
    })
    .on('error', (err: any) => {
        console.log('Error during download', err);
    });

Hopefully my woes will act as a documentation for poor souls who didn’t guess that the output has an encoding set to utf8 that corrupts binary data, and that they need to use the (at least back then, my guess is still now) implicit pipe API with omitting the callback.

So yes, this can be closed

0reactions
bantinicommented, Oct 18, 2017

Didn’t the pipe solve the issue for file downloads? If not, can you be more specific on what didn’t work on the file download? Accordingly, I can reopen this issue, or create a new one.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Why does UTF8 encoding change/corrupt bytes as oppose to ...
In any case, I repeat - do not encode/decode anything unless it is actually string data/required format. Share. Share a link to this...
Read more >
High Ascii UTF-8 characters appear corrupted and incorrect in ...
Setting the encoding to UTF-8 in the Request Properties for the SOAPUI tool , resolves the issue. The response now shows the correct...
Read more >
UTF-8 Characters get corrupted in HTTP request — oracle-tech
I'm using the following code to read HTTP reponse from Mongo DB to Oracle table. Some parts of the source (json-like) is in...
Read more >
How to ensure that data with special characters is not ...
While doing import using data loader you must ensure to enable setting "Read all CSVs with UTF-8 encoding" in Data Loader to avoid...
Read more >
How to use character encoding classes in .NET - Microsoft Learn
For example, the UTF8Encoding class describes the rules for encoding to, ... unexpected behavior and produce inaccurate or corrupted data.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found