401 on startup and 'iat' error on outbound traffic to GHE

Bug Report

Current Behavior I am running Probots in Docker containers on internal systems connected to GitHub Enterprise (2.20.3). Probots were operational until this weekend when our GHE instance was moved to new servers. They started having errors communicating with GHE.

Error message on startup:

19:21:34.101Z DEBUG github: GitHub request: GET /app - 401
  params: {
    "baseUrl": "https://github.*****.com/api/v3 ",
    "mediaType": {
      "format": "",
      "previews": []
    },
    "request": {
      "validate": {},
      "retryCount": 1
    }
  }

Error message on outbound messages to GHE:

19:48:09.247Z ERROR event: 'Issued at' claim ('iat') must be an Integer representing the time that the assertion was issued (id=1abfc200-93c0-11ea-8c03-2b692b209f3b)
  HttpError: 'Issued at' claim ('iat') must be an Integer representing the time that the assertion was issued
      at /app/node_modules/@octokit/request/dist-node/index.js:66:23
      at processTicksAndRejections (internal/process/task_queues.js:94:5)

In an attempt to fix the issue, I updated a bot to the latest version as well as spun up a fresh bot via npx create-probot-app to see if it would resolve the issues. I have included the details of both pre-update and post-update environments below.

Throughout the attempts I have validated the following:

• PRIVATE_KEY is set to value of the .pem files provided via the GH Apps interface
• WEBHOOK_SECRET is correct
• APP_ID is being set to the correct id
• GHE_HOST is set to the correct domain (github.*****.com)
• No firewall issues preventing communication
  • validated incoming with webhooks being received in logs
  • validated outgoing via curl to GHE APIs
• System clocks match on Decker and GHE servers (both in UTC)
• IP resolution of github.*****.com resolves to correct IP address within cluster

Expected behavior/code Probot to be able to connect with authentication to GHE

Environment (Pre-update)

• Probot version(s): v6.2.1
• Node/npm version: Node v8.9.4 / npm v5.6.0
• OS: Docker

Environment (Post-update & via npx create-probot-app)

• Probot version(s): v9.11.3
• Node/npm version: Node v12.14.1 / npm v6.13.4
• OS: Docker

Possible Solution N/A

Additional context/Screenshots Read through #967 and #1075 which do not seem to provide a valid solution to this.

[PseudoCoding]

It was found that there was a bad server that was part of our GHE infrastructure which was causing the validation failure. Thank you for your help.

[gr2m]

Hmm very odd. Do you get the error on different machines? I ask because I’ve seen that error when the date/time was not set correctly on the machine that the code is running on. In case of GHES I guess it could also be the server where GHES is running. Could you please double check that?