Request used as a streaming proxy breaks when query string parameters contain URLs

We use request to proxy using the stream API, so we can’t really control what query strings people will pass. We noticed this error in our logs:

Error: options.uri must be a path when using options.baseUrl

When a client requests the path:

/item/detail/395691820/?$deeplink_path=item/detail/395691820/&$fallback_url=https://offerup.com/item/detail/395691820/&button=item_view&user_id=36953831&event_id=2018-01-26_LosAngeles_CA_5&_branch_match_id=459093596557993047

After some tinkering it seems to be the URL that’s the problem. Here’s the stack trace:

Error: options.uri must be a path when using options.baseUrl
    at Request.init (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/request/request.js:212:33)
    at new Request (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/request/request.js:130:8)
    at request (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/request/index.js:54:10)
    at /Users/john.simons/repos/webapp/monorepo/services/web/src/request_test.js:18:25
    at Layer.handle [as handle_request] (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/express/lib/router/index.js:317:13)
    at /Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/express/lib/router/index.js:275:10)
    at expressInit (/Users/john.simons/repos/webapp/monorepo/services/web/src/node_modules/express/lib/middleware/init.js:40:5)

Also I boiled it down to as simple of a repro case as I could, but it still depends on Express:

That code is below. If you request any path on localhost:8080 you’ll repro the error.

const request = require('request');
const express = require('express');

let baseUrl = 'https://www.google.com';

// Breaks with "Error: options.uri must be a path when using options.baseUrl"
// if you use this URL. 
let uri = '/movies/about/?x=https://foo.com';

// It seems to be something about an unescaped URL as a query string
// parameter. This works fine:
// let url = '/movies/about/?x=https%3A//foo.com';
// 
// But colons themselves aren't the problem because this works also:
// let url = '/movies/about/?x=y:z';


const app = express();

app.use(function(req, res, next) {

    var remoteRequest = request({
        baseUrl: baseUrl,
        uri: uri,
        headers: req.headers,
        followRedirect: false,
    });

    req.pipe(remoteRequest)
        .on('error', next)
        .pipe(res);
});

app.listen(8080);

Workaround we’re currently using is to escape colons with s/:/%3A/

[johnnysprinkles]

Wait, I believe I see the issue reading the source:

if (self.uri.indexOf('//') === 0 || self.uri.indexOf('://') !== -1) {
  return self.emit('error', new Error('options.uri must be a path when using options.baseUrl'))
}

That seems like a mistake, something more like:

if (self.uri.indexOf('//') === 0 || self.uri.match(/^\w+:\/\//)) {

Might be better? Checks for “[scheme]😕/” anchored left, while allowing 😕/ to still occur later on. Should I put together a pull request?

[johnnysprinkles]

Ping