Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[training failure] Backend response timeout (30s)

See original GitHub issue

First, training with web app:

ibm_ca_failed

Then training with cacli:

$ cacli train

Bucket panels-01

⠏ Starting training run... error Backend response timeout (30s)

$ cacli list
──────────────────────────────────────────────────────────────────────────────────────────
 name                                          model id         status          submitted 
 panels-01                                  model-1qn1nzc4      error         an hour ago 
 panels-01                                  model-mno1p33h      error         an hour ago 
 panels-01                                  model-y7lg8gi8      error         an hour ago 
 panels-01                                  model-z9od9ow4      error      36 minutes ago 
 panels-01                                  model-2iwj18c1      error      30 minutes ago

$ cacli logs model-1qn1nzc4
error NoSuchKey: The specified key does not exist.
	status code: 404, request id: 70e8fc34-d48d-4055-9eaf-a2ea61703910, host id:

Issue Analytics

State:
Created 4 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

playrobotics-alexcommented, Mar 1, 2020

I think I found the problem. The machine learning service is set to London location by default, while the storage is in the US. I deleted the ML service and created a new one in Dallas location, I am no longer getting the error , the message now says “Waiting for an available GPU (this may take a while)…” which seems to be normal.

0reactions

bourdakos1commented, Mar 5, 2020

The issue with London has been resolved, training should work now 😃