Textile Buckets: Issue with Stream Upload for larger file sizes, even when HighWaterMarked.

Introduction

I’d like to use Readable Streams (or any stream) to send data to the Textile pushPath method.

• I don’t want to have a temporary local file I add and remove.
• Helps me avoid bad failure states.

There are 3 strategies in this issue, the first two are designed to work with Amazon S3, so I would expect they work with Textile’s Buckets too.

Current implementations

Broken for large files (over 5mb) stream implementation: https://github.com/filecoin-project/slate/blob/main/node_common/upload.js

FS.createReadStream implementation (In use now / works): https://github.com/filecoin-project/slate/blob/main/node_common/upload-fs.js

First stream implementation attempt.

This is the first implementation attempt at getting streams to work with the PassThrough stream constructor.

import * as LibraryManager from "~/node_common/managers/library";
import * as Utilities from "~/node_common/utilities";

import FORM from "formidable";

import { PassThrough } from "stream";

export const formMultipart = (req, res, { user }) =>
  new Promise(async (resolve, reject) => {
    const f = new FORM.IncomingForm();
    const p = new PassThrough({ highWaterMark: 1024 * 1024 * 3 });
    const file = {};

    const { buckets, bucketKey } = await Utilities.getBucketAPIFromUserToken(user.data.tokens.api);

    f.keepExtensions = true;

    f.onPart = (part) => {
      if (!part.filename) {
        form.handlePart(part);
        return;
      }

      file.name = part.filename;
      file.type = part.mime;

      part.on("data", function (buffer) {
        p.write(buffer);
      });

      part.on("end", function (data) {
        p.end();
      });
    };

    f.on("progress", (bytesReceived, bytesExpected) => {
      // console.log({ bytesReceived, bytesExpected });
    });

    f.parse(req, async (e) => {
      if (e) {
        return reject({});
      }

      if (!file && !file.name) {
        return reject({});
      }

      // NOTE(jim): Creates a Slate compatable Data object.
      const data = LibraryManager.createLocalDataIncomplete(file);

      let push;
      try {
        push = await buckets.pushPath(bucketKey, data.name, p);
      } catch (e) {
        return reject({});
      }

      return resolve({});
    });
  });

Results

• When I upload 5mb files, everything works as expected.
• When I upload 25mb files, everything works as expected.
• When I upload 70mb+ files, the stream is constructed (PassThrough) as expected, but I get the following error from Textile

{
  decorator: 'SERVER_BUCKETS_VERIFY_ISSUE',
  error: true,
  message: Error: Auth expired. Consider calling withKeyInfo or withAPISig to refresh.
      at Object.<anonymous> (/Users/whiteharbor/Development/slate/node_modules/@textile/security/src/index.ts:146:32)
      at Module._compile (internal/modules/cjs/loader.js:1200:30)
      at Module._compile (/Users/whiteharbor/Development/slate/node_modules/pirates/lib/index.js:99:24)
      at Module._extensions..js (internal/modules/cjs/loader.js:1220:10)
      at Object.newLoader [as .js] (/Users/whiteharbor/Development/slate/node_modules/pirates/lib/index.js:104:7)
      at Module.load (internal/modules/cjs/loader.js:1049:32)
      at Function.Module._load (internal/modules/cjs/loader.js:937:14)
      at Module.require (internal/modules/cjs/loader.js:1089:19)
      at require (internal/modules/cjs/helpers.js:73:18)
      at Object.<anonymous> (/Users/whiteharbor/Development/slate/node_modules/@textile/context/src/index.ts:2:1)
}

If you’re curious about the Stream, whether or not the node constructor is getting the correct _writeableState or _readableState have the correct HighWaterMark value, I have checked and it seems correct:

_writableState: WritableState {
    objectMode: false,
    highWaterMark: 3145728,
    finalCalled: false,
    needDrain: false,
    ending: true,
    ended: true,
    finished: false,
    destroyed: false,
    decodeStrings: true,
    defaultEncoding: 'utf8',
    length: 791699,
    writing: true,
    corked: 0,
    sync: false,
    bufferProcessing: false,
}

Thank you Textile!

[jimmylee]

Solved it here: https://github.com/filecoin-project/slate/blob/main/node_common/upload.js

[andrewxhill]

sweet. just for closing the loop. implementation can be seen here,

https://github.com/textileio/js-hub/blob/master/packages/buckets/src/api/index.ts#L308

and

https://github.com/textileio/js-hub/blob/master/packages/buckets/src/api/normalize.ts