[gatsby-plugin-sharp] fs read/write streams and sharp?

Summary

In trying to look into how gatsby handles processing and transforming images, I started to look into process-file.js and noticed that for the most part, reading and writing from disc seems to happen without Node streams. Would using streams help performance wise in this file? Was this already tried out and determined to not be such a great idea?

An example would be changing processFile’s transforms loop to create a sharp transformation pipeline that takes a single read stream, transforms the data, then writes the transformed data to disk. Something like (but not exactly):

const pipeline = sharp();
const readStream = fs.createReadStream(filename);
return transforms.map(async transform => {
  const { outputPath, args } = transform
  await pipeline.clone().toFile(outputPath);
});
const writeStream = fs.createWriteStream(outputPath);
readStream.pipe(pipeline).pipe(writeStream);

For the purposes of example, I’ve skipped the handling of extra sharp processing but can provide how that’d work too.

Relevant information

Examples of sharp streams with I/O

[smurrayatwork]

Update:

I think I have some benchmarks now. I’m using gnu-time for these. Node: 12.10.0

The test looks through a directory that contains a bunch of .png images, ranging in resolution from 8 x 8 to 3997 x 3997, runs them through sharp, then writes them to another directory as 100 x 100 png images.

Original gatsby-plugin-sharp:

Command being timed: "node src/__tests__/stream.js"
        User time (seconds): 86.09
        System time (seconds): 64.50
        Percent of CPU this job got: 597%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.19
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1275764
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 397
        Minor (reclaiming a frame) page faults: 8781833
        Voluntary context switches: 4423
        Involuntary context switches: 4818575
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 1692
        Socket messages received: 5980
        Signals delivered: 846
        Page size (bytes): 4096
        Exit status: 0

Modified gatsby-plugin-sharp:

Command being timed: "node src/__tests__/stream.js"
        User time (seconds): 68.60
        System time (seconds): 43.11
        Percent of CPU this job got: 555%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:20.11
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1215360
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 411
        Minor (reclaiming a frame) page faults: 3013657
        Voluntary context switches: 1006
        Involuntary context switches: 4982318
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Test Code:

const sharp = require(`../safe-sharp`)

const fs = require(`fs-extra`)

const { processFile } = require(`../process-file`)
const path = require('path');

const transforms = {
  args: {
    width: '100',
    height: '100',
    pngCompressionLevel: 9,
    toFormat: 'png',
    quality: 100,
    background: 'rgba(0,0,0,1)',
    trim: 0,
    rotate: 0,
    cropFocus: sharp.strategy.attention,
    fit: sharp.fit.cover,
    duotone: null,
    jpegProgressive: false,
  },
};
const runTest = async () => {
  await fs.readdir(path.resolve(__dirname, './stream_images'), (error, files) => {
    files.forEach( async (file) => {
      if (error) {
        throw error;
      }
      const fileTransform = Object.assign(transforms, { outputPath: path.resolve(__dirname, `./stream_test/${file}`)});
      await processFile(path.resolve(__dirname, `./stream_images/${file}`), [fileTransform], {
        stripMetadata: false,
      });
    });
  });
}
runTest();

[smurrayatwork]

@KyleAMathews @sidharthachatterjee ,

Sure, I’ll try my best! I have a WIP modification branch that’s mostly completed using streams now. I need to clean up the code and get some benchmarks tied into it, but from what I was seeing, there wasn’t a noticeable difference in speed at all.