Che-machine-exec fails to execute commands after deployment is restarted

Describe the bug

If a workspace deployment restarts for any reason, che-machine-exec will be unable to execute any tasks until the workspace is refreshed, failing with error

 Error launching task '1. Build': Request runTask failed with message: connection is closed 

refreshing the page resolves the issue. Note also that opening terminals in unaffected.

Che version

• other: Tested on 7.26; reproduces on workspaces.openshift.com

Steps to reproduce

1. Start a workspace from a devfile that includes commands
2. Launch a task from the sidebar, completes successfully
3. Delete the workspace’s pod from the commandline so that the deployment recreates it. This causes Theia to go temporarily offline as the pod is recreated
4. Once Theia is online again, attempt to launch a task from the sidebar.

Expected behavior

Once theia reconnects, launching tasks should work

Additional context

The only suspect logs I see are in Theia:

2021-03-02 01:30:45.569 root ERROR Request runTask failed with error: connection is closed Params: Error: connection is closed
    at Object.e.create (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:801416)
    at e.remoteCall (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:803191)
    at Proxy.n.<computed>.r.charCodeAt.n.<computed> (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:803030)
    at http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:666583
    at e.<anonymous> (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:103683)
    at http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:102796
    at Object.next (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:102901)
    at http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:101813
    at new Promise (<anonymous>)
    at o (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:101558)
2021-03-02 01:30:45.579 root ERROR Error launching task 'sync-template': Request runTask failed with message: connection is closed Params: Error: Request 'run' failed
    at Proxy.<anonymous> (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:1586464)
    at t.<anonymous> (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:922871)
    at http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:922053
    at Object.next (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:922158)
    at a (http://workspace99a9e5e1468b496b-theia-3100.192.168.49.2.nip.io/theia.838437d9e5ea8a7869a6.js:1:920872)
2021-03-02 01:30:46.589 root ERROR [hosted-plugin: 48] Promise rejection not handled in one second: Error: Run task config does not return after being started , reason: Error: Run task config does not return after being started 
2021-03-02 01:30:46.589 root ERROR Promise rejection not handled in one second: Error: Run task config does not return after being started , reason: Error: Run task config does not return after being started
 
2021-03-02 01:30:46.590 root ERROR [hosted-plugin: 48] With stack trace: Error: Run task config does not return after being started
    at TasksExtImpl.<anonymous> (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:199:31)
    at step (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:48:23)
    at Object.next (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:29:53)
    at fulfilled (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:20:58)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) 
2021-03-02 01:30:46.590 root ERROR With stack trace: Error: Run task config does not return after being started
    at TasksExtImpl.<anonymous> (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:199:31)
    at step (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:48:23)
    at Object.next (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:29:53)
    at fulfilled (/home/theia/node_modules/@theia/plugin-ext/lib/plugin/tasks/tasks.js:20:58)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

[RomanNikitenko]

@RomanNikitenko I’m able to open terminals without issue in my testing, it’s only devfile commands that have the issue.

OK, It’s helpful to localize the problem, looks like the problem on plugin side. thank you, Angel!

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.