Random build timeouts using `firebase deploy`

I have a large project with many cloud functions, and lately an error has been popping up on random functions in my deploy: Deployment error. Build failed: Build has timed out. Sometimes it says Build error details not available instead of the timeout.

This affects a small percentage of functions but is enough to be annoying (often requiring many redeploys).

Triage

If I check Stackdriver logs for the build failure, I see an error message INVALID_ARGUMENT in all cases.

But I never used to get this before and the timeouts are not at all deterministic (when they occur OR for which functions they occur). If this is a GCP limit I’m running into, I’m not really sure which one it is.

Moreover, some days seem to have far fewer build timeouts than other days.

Environment

The only things that have changed outside the code are package version bumps, including an upgrade from the Node 6 runtime to the Node 10 runtime, as well as a version bump to the latest firebase CLI (installed globally).

Also worth noting is that I do a large number of parallel deploys, but the errors occur even when only a handful of functions are being deployed concurrently.

Any ideas? Much appreciated!

[op]

We’ve seen an increasing amount of timeouts too. Might be our CI provider that’s having issues though. (We’re also on Node 10.)

Wouldn’t it make sense to make an effort to retry on timeout failures though?

[whaatt]

Just for clarity, I think two separate issues are being discussed in this thread: (A) the original deploy issue for Cloud Functions and (B) the Firebase hosting issue this morning, which is being worked on (as @mitchelloptima noted).