Probot's setup mode is easily overlooked and leaves users confused why the app is not working

Bug Report

Current Behavior I am trying to add a simple express route for a health check on my probot app. I was getting a 404 on my custom configuration, so I revisited the docs to see how to add the express route for this. Even when I overwrite my exported function with the example code from the documentation below, I still get a 404. How is this supposed to work? Are express routes supported?

module.exports = app => {
  // Get an express router to expose new HTTP endpoints
  const router = app.route('/my-app')

  // Use any middleware
  router.use(require('express').static('public'))

  // Add a new route
  router.get('/hello-world', (req, res) => {
    res.send('Hello World')
  })
}

  Welcome to Probot! Go to http://localhost:3000 to get started.
  
22:44:08.663Z  INFO probot: Forwarding https://smee.io/2xurzllqp8L84xa to http://localhost:3000/
22:44:08.664Z  INFO probot: Listening on http://localhost:3000
22:44:08.742Z  INFO probot: Connected https://smee.io/2xurzllqp8L84xa
22:44:09.424Z  INFO http: GET /probot/my-app/hello-world 404 - 4.36 ms (id=8b2c2cbd-bad5-4fce-bc49-7e604944a6ce)
22:44:11.912Z  INFO http: GET /my-app/hello-world 404 - 0.98 ms (id=fb8c57dc-a01f-49a0-85db-ce48369696e8)

Expected behavior/code I would expect to get a 200 back in this case.

Environment

• Probot version(s): tested 9.9.4 & 9.11.3
• Node/npm version: tested v12.14.1 (npm v6.13.4) & v14.0.0 (npm v6.14.4)
• OS: OSX 10.14.6

Possible Solution N/A

Additional context/Screenshots None

[hardboiled]

https://github.com/probot/probot/issues/1069 ^ Seems like someone else was previously having the same problem. When I do an app.log within the block, it seems like it’s ignored

$ cat ./lib/src/index.js
"use strict";
module.exports = (app) => {
    // Get an express router to expose new HTTP endpoints
    const router = app.route('/my-app');
    // Use any middleware
    router.use(require('express').static('public'));
    app.log('is this code running?');
    // Add a new route
    router.get('/hello-world', async (_req, res) => {
        res.send('Hello World');
    });
};
$ npx probot run ./lib/src/index.js &
[1] 11428

23:34:48.516Z  INFO probot:
  
  Welcome to Probot! Go to http://localhost:3000 to get started.
  
23:34:48.560Z  WARN probot: Run `npm install --save-dev smee-client` to proxy webhooks to localhost.
23:34:48.562Z  INFO probot: Listening on http://localhost:3000

$ curl -I http://localhost:3000/my-app/hello-world
HTTP/1.1 404 Not Found
X-Powered-By: Express
x-request-id: b3ec76c4-bf60-4aaa-b553-fb31ff7e933b
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 158
Date: Wed, 29 Apr 2020 23:35:24 GMT
Connection: keep-alive

$ cat .env
APP_ID=
PRIVATE_KEY=
WEBHOOK_SECRET=development
LOG_LEVEL=debug
NODE_ENV=development
WEBHOOK_PROXY_URL=#https://smee.io/<some-url>

[hardboiled]

I’m sure this creates confusion when you download the boilerplate, copy and paste the code and expect things to work without registering a Github App first, but that’s not how it is intended to be used.

Is this documented somewhere? I already have QA and production versions of my github app that I created while being a superuser in my org, and I was trying to circle back and create a sane development environment for our other devs, so they can build and test any additions to our app. They won’t be able to update the app configuration (e.g. the event endpoint) or install any github apps to our org. Thus I didn’t want to share any configuration with the existing deployed environments. Is there an easy workaround or recommended setup for this?

It was frustrating to try add simple code like a health check endpoint and logging only to see them not run locally. I wasn’t able to figure out how to test these without deploying, and there were no error messages telling me why my code wasn’t running.