[BUG] flytectl sandbox failed to start

Describe the bug

🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
delete existing sandbox cluster [y/n]: y
🐋 Running Flyte v0.19.2 release
🐋 pulling docker image for release cr.flyte.org/flyteorg/flyte-sandbox:dind-cb382d0f85cc9e2dc52e6e96dd7a6ae79801eb39
{"status":"Pulling from flyteorg/flyte-sandbox","id":"dind-cb382d0f85cc9e2dc52e6e96dd7a6ae79801eb39"}
{"status":"Digest: sha256:9d61423ca947b53669410bc003debc814feac72d48a5a0e70efdc4c7d6cdb80f"}
{"status":"Status: Image is up to date for cr.flyte.org/flyteorg/flyte-sandbox:dind-cb382d0f85cc9e2dc52e6e96dd7a6ae79801eb39"}
🧑‍🏭 booting Flyte-sandbox container
92022-02-22T10:55:08.360985868Z Starting Docker daemon...
%2022-02-22T10:55:09.426768152Z Done.
72022-02-22T10:55:09.426802741Z Starting k3s cluster...

....

+---------------------------------------------+-----------+-----------+
|                   SERVICE                   |  STATUS   | NAMESPACE |
+---------------------------------------------+-----------+-----------+
| flyte-contour-contour-7cfc9f6fb5-qjg59      | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| flyte-contour-envoy-tn7jb                   | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| minio-7749dfc49d-q4sl7                      | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| flyteconsole-6dd485d769-jxk4z               | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| flytepropeller-6f5c985cd4-fj2ts             | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| postgres-8664477894-jb2ww                   | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| datacatalog-6c5b75c485-bss8r                | Pending   | flyte     |
+---------------------------------------------+-----------+-----------+
| flyte-pod-webhook-5f5d86c46d-5k8nd          | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| syncresources-27425413-rv6f6                | Pending   | flyte     |
+---------------------------------------------+-----------+-----------+
| flyte-kubernetes-dashboard-7fd989b99d-mf6ht | Running   | flyte     |
+---------------------------------------------+-----------+-----------+
| syncresources-27425414-c54kr                | Pending   | flyte     |
+---------------------------------------------+-----------+-----------+
| syncresources-27425412-t2kxb                | Succeeded | flyte     |
+---------------------------------------------+-----------+-----------+
| flyteadmin-84bb997f9b-m52vj                 | Pending   | flyte     |
+---------------------------------------------+-----------+-----------+
| syncresources-27425411-nhf46                | Succeeded | flyte     |
+---------------------------------------------+-----------+-----------+
| syncresources-27425415-cfpnb                | Succeeded | flyte     |
+---------------------------------------------+-----------+-----------+
| flytescheduler-9d5f844cc-ld9s9              | Pending   | flyte     |
+---------------------------------------------+-----------+-----------+
Error: Get "https://127.0.0.1:30086/api/v1/nodes": dial tcp 127.0.0.1:30086: connect: connection refused

Expected behavior

sandbox should just works.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn’t been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[Abdullahi-Ahmed]

I had the same issue; since I was using vm thus my machine was slow that kills the request with short timeout kicks. Also sometimes i forget to turn my docker daemon on, this is important because flyte sandbox runs locally in a docker container. restarting my vm solved for me.

[evalsocket]

Flyte will downloading all docker images of it’s component… this can be slow depending on your network bandwidth and geographical location

Please make sure the docker daemon has enough resources (say 8Gb mem), We are working on reducing the package size to speed up this process for first time users… stay tuned!

To know what’s happening with these, you can do this:

flytectl sandbox exec -- kubectl describe pod -n flyte 

Finally note that if it remains Pending for long, the operation will timeout and you will get a failure. You can increase the timeout by running:

flytectl sandbox start --env FLYTE_TIMEOUT=1800

To allow it to wait up to 30mins…