Ever increasing json file when creating resumable upload urls

Environment details

• OS: Linux
• Node.js version: 10
• npm version:
• @google-cloud/storage version: 5.8.5

Steps to reproduce

1. create a resumable upload url by using createResumableUpload()
2. look at file .config/configstore/gcs-resumable-upload.json

When creating a resumable upload urls the package seems to write that to a json file that is ever increasing and never cleaned up. This makes calls to createResumableUpload slower over time because that json is parsed on each call. Another side effect are high volume of disk writes.

Not sure if this is an issue with this package or with gcs-resumable-upload because this lib is using a method from there. The file seems to be the one sent as configPath.

[ruidfigueiredo]

Hi @shaffeeullah

I’d argue that feature request (described as a nice to have improvement) is not appropriate at all for this issue.

After generating a few thousand resumable upload ulrs the generation process will start taking seconds to complete, the amount of IO on disk will be very high (in one month we saw 28TB of writes because of this issue) and generating one url was taking more than 3 seconds.

[ruidfigueiredo]

Hi @shaffeeullah

To reproduce first create a storage object, bucket and file. On the file call the createResumableUpload method a couple of times then check the ~./config/configstore/gcs-resumable-upload.json file and you’ll find there all the generated resumable upload urls.

This file will grow unbound and when it reaches a certain size it will make the resumable url generation process take a non reasonable amount of time and resources (as it needs to be read, parsed and written back to disk every time createResumableUrl is called).

Here’s a full sample:

const { Storage } = require('@google-cloud/storage')

const storage = new Storage({
    projectId: 'the gcloud project id',
    credentials: {
        client_email: "servcice account email or some other method of providing credentials",
        private_key: "the key",
    }
})

const bucket = storage.bucket('bucket name');

async function getResumableUploadUrl() {
    const res = await bucket
        .file(`the file name ${Math.random()}.wav`)
        .createResumableUpload({
            metadata: {
                contentType: "audio/wav",
            },
            origin: 'https://somedomain.com/'
        });

    return res[0];
}

async function main() {
    console.log(await getResumableUploadUrl())
    console.log(await getResumableUploadUrl())
    console.log(await getResumableUploadUrl())
}

main().then(() => console.log('done'), console.error);

After running this check the current user’s .config folder at ~/.config/configstore/gcs-resumable-upload.json. You’ll find the generated urls there (along with some extra information).

Extra details:

The implementation of createResumableUpload makes use of a dependency (another npm package from google) called gcs-resumable- upload. Here you can confirm that resumableUpload is an import * as resumableUpload from 'gcs-resumable-upload'

Important thing to note here is that createResumableUpload simply calls gcs-resumable-upload’s createURI.

When this function is called an instance of an Upload object gets created: https://github.com/googleapis/gcs-resumable-upload/blob/master/src/index.ts#L689

On that object’s constructor a ConfigStore is created using the packageName gcs-resumable-upload: https://github.com/googleapis/gcs-resumable-upload/blob/master/src/index.ts#L295

On actually creating the resumable upload url it is saved to this config store: https://github.com/googleapis/gcs-resumable-upload/blob/master/src/index.ts#L377

And in this scenario it is never removed (this Upload class from gcs-resumable-upload seems to have other intended scenarios other than just generating the resumable uri, and in those it seems that the uri gets clean up [I didn’t test this]).

If you look at ConfigStore’s implementation for set you’ll see it will become extremely inefficient when the config file gets large:

https://github.com/yeoman/configstore/blob/main/index.js#L32

https://github.com/yeoman/configstore/blob/main/index.js#L79