Enhancement: import filtering

As of west 0.11.x, you cannot combine groups with import in a single project definition. The documentation says:

As a restriction, no project may use both import: and groups:. (This avoids some edge cases whose semantics are difficult to specify.)

If you try, you’ll hit the error handling block that begins here:

https://github.com/zephyrproject-rtos/west/blob/38e656b05ea8f4c8d80b953f6d88b1ed604d11f8/src/west/manifest.py#L2027

This issue describes:

• a motivating use case for relaxing this restriction
• a rationale for why I introduced the restriction in the first place
• a proposal for how to remove the restriction and deal with the resulting difficulties
• a chicken bit I would put in place to save us in case the whole idea is a great big mistake

Use case

Consider a CI maintainer for a downstream Zephyr-based software distribution that provides additional platform support on top of the base OS.

As that CI maintainer, I (the hypothetical CI maintainer, not necessarily @mbolivar-nordic) would like to associate a set of test repositories with the manifest repository for my Zephyr distribution. I would like to do this by including some repositories containing test cases in my west manifest.

This will allow me to conveniently version the test repository revisions alongside the revisions of the code being tested. That allows me to maintain different branches of the same test case repositories to exercise different branches of the accompanying downstream distribution. Furthermore, putting my test repositories into west allows my platform developers to more easily reproduce results such as build errors in my test applications by running west update to get the test code.

I would like to adapt the continuous integration overrides pattern from the west documentation for my use case. This will also allow me to exercise changes that have to go into my test and platform repositories at the same time, which may occur for example if a Zephyr API used by both types of repository changes. I would like to adapt the documented pattern as follows.

First, I do not want developers to have to clone these test repositories by default into the workspace for several potential reasons:

• they are not relevant to day-to-day development
• they contain proprietary information
• they contain large amounts of binary test data or other artifacts

I therefore would like the test repositories to be inactive projects by default.

Second, I do not want the names or locations of the test repositories to be made public as they are also considered proprietary. (This is not incompatible with an open source platform, e.g. sqlite is another OSS project that has a proprietary test suite.)

I therefore would like to organize my manifests like this:

# my-manifest-repo/west.yml
manifest:
  group-filter: [-proprietary-tests]
  projects:
    # ... import zephyr + add my special sauce platform/application repositories
  self:
    import: submanifests

# my-manifest-repo/submanifests/99-tests.yaml
manifest:
  projects:
    - name: my-main-test-repo
      url: https://git.mycompany.com/my-main-test-repo
      revision: SOME-SHA
      groups: [proprietary-tests]
      import: true

# my-main-test-repo/west.yml
manifest:
  # ...
  projects:
    - name: my-test-repo-1
      groups: [proprietary-tests]
    - name: my-test-repo-2
      groups: [proprietary-tests]
    - name: my-test-repo-3
      groups: [proprietary-tests]
    # ...
    - name: my-test-repo-N
      groups: [proprietary-tests]

This would accomplish my goals:

1. I can use different revision fields for my-main-test-repo in different release branches of my platform to keep parallel test branches alive at the same time
2. I can dynamically generate my-manifest-repo/submanifests/00-test-overrides.yml for individual pull requests to selectively override any test repository or platform repository, allowing me to test workspaces where changes to test and platform repositories must be exercised in concert
3. My developers and users can still run west update using my-manifest-repo without getting the tests
4. The names and locations of the test case repositories themselves are not public because my-main-test-repo has a generic name and is access controlled
5. My developers can enable the proprietary-tests group and run west update to get an environment that matches my CI, in order to reproduce errors that require test repositories to build and run

However, I can’t do this because west won’t let me do this snippet which combines groups and import:

      groups: [proprietary-tests]
      import: true

This is a showstopper for me.

Rationale for the restriction

For context, I (@mbolivar-nordic this time) was the main designer and the implementer for both the import: and groups: features; import: came first.

When working on groups:, I found that specifying the semantics for the combination of the two ran into some edge cases that I found difficult to specify without a good motivating use case that would help guide the design. I therefore decided to simply forbid it for convenience, and wait to see if anything would come along that would light the way forward.

Here’s an example I cooked up that shows the seemingly paradoxical results that can arise from naive interpretations of what to do:

# manifest-repo/west.yml
manifest:
  projects:
    - name: submanifest-repo
      groups: [foo]
      import: true

# submanifest-repo/west.yml
manifest:
  group-filter: [-foo]
  projects:
    - name: proj1
    - name: proj2

I might say “submanifest-repo is obviously active because foo is enabled in manifest-repo/west.yml.” But then if west imports submanifest-repo/west.yml, I end up with foo disabled in the resolved manifest, because group-filter values from imported manifests affect the resolved manifest. This makes submanifest-repo inactive, because its only group is disabled.

A paradox:

• if submanifest-repo is considered active and we import from it, then it’s inactive, for reasons described above
• but if it’s considered inactive, then we would never import the [-foo] group filter, so then… it’s active, actually?

What about proj1 and proj2 then? Should they be “deleted” from the resolved manifest somehow, or are they still OK to include as active projects in situations like this?

Proposal for removing the restriction

I think the above use case is a valid one and I would like to propose that west enables it. However, we have to figure out how to resolve the above difficulties.

I think we should go ahead and perform the import:, even if the project has groups:, as long as the available information indicates that the project is active. So in the example from the above rationale, we should import submanifest-repo unless the workspace’s manifest.group-filter configuration option has disabled group foo, because manifest-repo/west.yml on its own would treat submanifest-repo as an active project since not all its groups are disabled by default.

Then, after resolving the complete manifest, we should do a postprocessing check that all projects with an import: remain active projects after all the imports are done. (A project with no groups is active by definition, which is why forbidding the combination of the two sidestepped these difficulties.) If so, the resulting resolved manifest seems to be internally consistent to me. If not, we skip the paradox by throwing an exception from the west.manifest.Manifest constructor, preventing users from constructing this type of self-contradictory (even Rusellian) Manifest object.

This behavior would handle the main use case just fine, since the CI maintainer is not going to do something strange like disable the proprietary-tests group from my-main-test-repo/west.yml. So the workspace works as intended with and without that group enabled: if it’s enabled, the test projects are active. If it’s disabled (the default), they are inactive, and that includes my-main-test-repo itself. Users and developers get what they want by default (i.e. no test repositories), but it’s easy to make test repositories into active projects by enabling a single group.

The above error handling might force west update to clone some additional projects that it would ordinarily not have in order to compute the resolved manifest, especially in the case of multiple levels of imports. That in turn could lead to permissions errors if the repositories are access controlled. So be it.

Chicken bit

The above proposal is a bit risky since we only have one use case, so I think this should be an experimental behavior we can rip out later. Furthermore, I think it should require explicit “opt-in” support. So I am also proposing a new boolean manifest.experimental-group-import configuration option that must be enabled to turn on this new behavior.

Therefore the CI environment would have to do something like this in order to opt in:

west config --global manifest.experimental-group-import true
west init -m my-manifest-repo-url my-workspace
cd my-workspace
west config manifest.group-filter +proprietary-tests
west update

If manifest.experimental-group-import is disabled, the restriction remains in place and west update fails with the existing error message. Otherwise we proceed as above.

The maintainers of CI environments have many options available for controlling exactly the version of west used for testing and their setup scripts, so this allows us to get some real world experience without fully committing to the above behavior if we run into unanticipated consequences.

[DatGizmo]

Would a new filter flag for imports be less prone to the confusion you described in the “Rational restricrions”? So instead of reusing the group filter use a new “import-filter” flag.

[marc-hb]

I don’t see how this would get more complex when taking the local worktree status. I would argue it is a lot less surprising if results actually match the files I see and modified…

This is not about the least surprise for the user. It’s about the complexity of the internal implementation meaning bugs, corner cases and maybe… even more surprising behaviors in the end.

I think @mbolivar-nordic explained this once somewhere but I couldn’t find it sorry.