gridfs-stream compatability

I am trying to use your library to stream multipart file uploads directly into gridfs using gridfs-stream. I’m having an issue with the encoding of the original file changing. If I pipe the busboy file stream into an fs writestream, then read the file back out with an fs readstream and pipe it to gridfs, all is well. but if I skip the fs streams and go directly to gridfs, the encoding gets messed up.

I have an issue open with gridfs-stream, but I wanted to check if there’s anything I could do from this end.

https://github.com/aheckmann/gridfs-stream/issues/76

Edit:

Here’s how I want to do it, followed by what is actually working for me.

Results in bad/incorrect encoding when read back out:

exports.create = function(req, res) {
    var busboy = new Busboy({ headers: req.headers });
    var fileId = new mongoose.Types.ObjectId();

    busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
        var writeStream = gfs.createWriteStream({
            _id: fileId,
            filename: filename,
            mode: 'w',
            content_type: mimetype,
            metadata: {
                uploadedBy: req.user._id,
                encoding: encoding,
            }
        });

        file.pipe(writeStream);
    });

    busboy.on('finish', function() {
        return res.status(200).send({
            message: fileId.toString()
        });
    });

    req.pipe(busboy);
};

works:

exports.create = function(req, res) {
    var busboy = new Busboy({ headers: req.headers });
    var fileId = new mongoose.Types.ObjectId();

    busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
        var tmpPath = '/tmp/' + filename;
        var fsWriteStream = fs.createWriteStream(tmpPath);

        busboy.on('finish', function() {

            var fsReadStream = fs.createReadStream(tmpPath);

            var writeStream = gfs.createWriteStream({
                _id: fileId,
                filename: filename,
                mode: 'w',
                content_type: mimetype,
                metadata: {
                    uploadedBy: req.user._id,
                    encoding: encoding,
                }
            });

            fsReadStream.on('end', function() {
                fs.unlink(tmpPath, function (err) {
                    if (err) throw err;
                    res.status(200).send({
                        message: fileId.toString()
                    }); 
                });
            });

            fsReadStream.pipe(writeStream);

        });

        file.pipe(fsWriteStream);
    });

    req.pipe(busboy);
};

[wdawson4]

@pos1tron I am experiencing the same issue. Were you able to figure out what was causing this?

I managed to debug the issue. I narrowed it down to where i was confident that the problem was a piece of express middleware modified the request. I disabled my middleware one by one until i found the unlikely culprit: connect-livereload

I commented out app.use(require('connect-livereload')()); and the problem went away. I believe it was injecting the livereload script into the response (a binary image file).

[SOSANA]

1+ thanks everyone! helped me as well