[BUG] Multiple thumbnails folder creation (critical)

[READ] Step 1: Are you in the right place?

yes

[REQUIRED] Step 2: Describe your configuration

• Extension name: storage-resize-images

[REQUIRED] Step 3: Describe the problem

When uploading an image to the storage bucket, seemingly infinite thumbnails folders are created, which exceeded my quota for SocketConnectNonbillable within seconds.

Steps to reproduce:

Upload an image to storage bucket.

Expected result

Image should be resized inside one lowRes folder.

Actual result

The folder was created multiple times, each containing the resized image, which probably lead to my quota exceeding.

Logs i managed to get before my quota was exceeded: (shortened paths with … in order to make it more readable)

Function execution started Billing account not configured. External network is not accessible and quotas are severely limited. Configure billing account to remove these restrictions Started execution of extension with configuration { bucket: '[...]', cacheControlHeader: undefined, imageSizes: [ '200x200' ], resizedImagesPath: 'lowRes' } Creating temporary directory: '/tmp/[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes' Created temporary directory: '/tmp/[...]/lowRes/.../lowRes' Downloading image file: '[...]/lowRes.../lowRes/_200x200' Downloaded image file: '[...]/lowRes/.../lowRes/_200x200' to '/tmp/[...]/lowRes/.../lowRes/_200x200' Resizing image at path '/tmp/_200x200' to size: 200x200 Resized image created at '/tmp/_200x200' Uploading resized image to '[...]/lowRes/.../lowRes/_200x200' Uploaded resized image to '[...]/lowRes/.../lowRes/_200x200' Deleting temporary resized file: '[...]/lowRes/.../lowRes/_200x200' Deleted temporary resized file: '[...]/lowRes/.../lowRes/_200x200' Completed execution of extension Deleting temporary original file: '[...]/lowRes/.../lowRes/_200x200' Deleted temporary original file: '[...]/lowRes/.../lowRes/_200x200' Function execution took 465 ms, finished with status: 'ok' Error: quota exceeded (Quota exceeded for quota group 'SocketConnectNonbillable' and limit 'CLIENT_PROJECT-100s' of service 'cloudfunctions.googleapis.com' for consumer 'project_number:[...]'.);

In conclusion, the cloud function was called ~500 times which can be seen in my statistics. (I never used cloud functions before and only tried to upload an image two times) Later i also got this error, probably as a result:

    at Util.parseHttpRespBody (/srv/node_modules/@google-cloud/common/build/src/util.js:191:38)
    at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:135:117)
    at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:423:22)
    at onResponse (/srv/node_modules/retry-request/index.js:206:7)
    at /srv/node_modules/teeny-request/build/src/index.js:158:17
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
  code: 400,
  errors: 
   [ { domain: 'global',
       reason: 'invalid',
       message: 'The maximum object length is 1024 characters, but got a name with 1025 characters: \'\'[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRe...\'\'' } ],
  response: undefined,
  message: 'The maximum object length is 1024 characters, but got a name with 1025 characters: \'\'[...]/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRes/lowRe...\'\'' }```

[codediodeio]

It looks like this issue is caused when an image file is uploaded without an extension. The isResizedImage should be be expanded to check for the extension, or the function should auto-assign an extension if missing, or as @Salakar mentioned a custom metadata field may be the best option.

Here is a test case for the current implementation. In prod, the result is an infinite loop that uploads resized images with an empty filename. This will cause runaway billing if not caught early.

const path = require('path')

const suffixRegex = /_\d+x\d+$/;
const isResizedImageValidator = (fileName) => suffixRegex.test(fileName);

const filePath = 'thumbs/someImageWithoutExt_200x200' 
const fileName = path.basename(filePath);
const fileExtension = path.extname(filePath); // ""
const fileNameWithoutExtension = fileName.slice(0, -fileExtension.length);  // ""
const isResizedImage = isResizedImageValidator(fileNameWithoutExtension);

console.log(isResizedImage) // false :(

[Salakar]

Hey @marvinsxtr, looks like this is the regex failing here to pickup that it’s already a resized image.

I think we can look at removing this regex and instead use a custom metadata field to infer that an image has already been resized.