Improve the FORCE_QUEUE process in Helix-machines

• This issue is blocking
• This issue is causing unreasonable pain

By default, PRs to helix-machines will only attempt to generate images for a single windows and a single linux queue. This is often not enough testing for some of the changes to machine configurations, which have different subtleties depending on the OS and version.

We have the FORCE_QUEUE feature that allows to generate specific images during the PR process. However, this feature is sometimes not used, and we end up having to revert changes that fail in specific OSes that weren’t tested during the PR.

We should look into ways of making the FORCE_QUEUE feature more visible, so that both PR authors and code reviewers are more easily aware of any gaps in their testing.

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

For internal use only…

We have removed default queues and tightened the FORCE_QUEUE logic. Specifically:

• build only FORCE_QUEUE images in PR builds
  • avoid complications like checking for other queues in the same OS group
  • those checks previously chose the first Linux and Windows definitions found as defaults
• require at least two FORCE_QUEUEs in PR and manual builds
  • named queues must exist and require distinct images
• support BUILD_EXISTING_IMAGES in commit descriptions to force rebuilds
• add words to the PR template (when targeting ‘main’) about how to use FORCE_QUEUE and BUILD_EXISTING_IMAGES
• refactor some of the CreateCustomImage code; change logging and comments

[chcosta]

@riarenas - just a note, so we don’t forget our hallway conversation : consider removing default queues (just make them suggestions) so people are forced to be intentional about which queues are running (require at least one queue [maybe more]) be defined.

[dougbu]

Builds of main are no longer showing ill effects of the fix. Closing this issue. If necessary, we’ll track the latest build break (which might just require a few more retries0 in a new issue.