[UI] Improve App background statuses feedback

Preflight Checklist

• I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Problem Description

Relying on progress bar to make sure that all background services are up/running could be a danger zone. If something went really bad and the process bar was detached (hidden) and you than you notice something isn’t working… you might need to dig into the logs, access the VM and do a bunch of manual process to double check if everything is working.

Proposed Solution

• Provide a health check endpoint/feature for each services loaded on background e.g: K8s cluster healthz check ( kubectl healthz curl -k https://localhost:6443/livez?verbose)
• Add a visual system status on the UI (could be a HTML element, but an element that receive a list with all statuses from the services)

Additional Information

Motivator: Currently our E2E tests rely on .progress class element. We’re assuming after the progress bar being detached from the page, it means that all services were properly loaded (k3s, k8s api, VM, and etc). So far, it is working, but it will be much better to have feature that we can ensure all the services were loaded properly or not, without rely on a HTML element (visible/hidden states).

[rak-phillip]

I think it would make sense to fit in a red/yellow/green status indicator. I imagine that it could include a dropdown with a status check for each of our services, so that we could easily see where a warning or failure occurred. Making use of a status file could provide us with all the information we need to get a quick heads up about the state of our system, while clicking on any given service could open the respective log for further troubleshooting.

[jandubois]

I was wondering if it would make sense to write a JSON file with status information to disk (in the log directory) every time the status changes?

To avoid race conditions between writing and reading the file, the data should always be written to a temp file, which then gets renamed to the canonical name. If the rename fails, the error is ignored because the canonical file is probably locked by somebody reading it.

This file could both be used to wait until RD reaches a specific state, and also for inspecting the last state when the app hangs or crashes.

Not sure if this would be helpful for CI and/or QA.

PS: I just noticed that this issue has a [UI] tag in the subject, so this comment is probably off-topic. 😄