Unable to run custom west commands before all imported repositories are cloned

If a manifest imports from other projects, no custom west commands can be run from any project until all imported projects have been cloned. For example, if a manifest has the following form:

projects:
  - name: internal1
    revision: some-branch
  - name: internal2
    revision some-branch

  - name: zephyr
    remote: zephyr
    revision: v2.5.0
    import:
      name-whitelist:
      - proj1
      - proj2
  - name: external1
    remote: external-repo
    revision: v1.1.0
    import:
      name-whitelist:
      - proja
self:
  west-commands: scripts/west-commands.yml

Custom commands from the scripts/west-commands.yml will be unavailable until both zephyr and external1 are cloned. This can be a problem if some of the custom commands do important environment setup or supplement west update in some way; forcing user to west update before running the custom commands that are required for setting up/updating their environment properly.

Suggestion would be to ignore non-cloned imports from the repository instead of causing them to cancel the additional of all custom commands entirely.

(Note, in our project, we have replaced west update with another command that does what west update does as well as some other tasks, but we cannot run this custom command on new workspaces nor if the manifest is changed to include new imported projects, even though the custom command is defined in the same repository as the manifest file and thus always available)

[mrrosen]

Ive created a simple example for you to see. It does seem to be tied to import, from what Ive observed: https://github.com/mrrosen/manifest

west init -m https://github.com/mrrosen/manifest.git
=== Initializing in C:\Users\mrrosen\identity\test\simple
--- no --manifest-rev was given; using remote's default branch: refs/heads/main
--- Cloning manifest repository from https://github.com/mrrosen/manifest.git, rev. refs/heads/main
Initialized empty Git repository in C:/Users/mrrosen/identity/test/simple/.west/manifest-tmp/.git/
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 6 (delta 0), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), 1001 bytes | 27.00 KiB/s, done.
From https://github.com/mrrosen/manifest
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Note: switching to 'FETCH_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 96126e5 simple west test
--- setting manifest.path to manifest
=== Initialized. Now run "west update" inside C:\Users\mrrosen\identity\test\simple.

C:\Users\mrrosen\identity\test\simple>west hello
usage: west [-h] [-z ZEPHYR_BASE] [-v] [-V] <command> ...
west: error: argument <command>: invalid choice: 'hello' (choose from 'init', 'update', 'list', 'manifest', 'diff', 'status', 'forall', 'help', 'config', 'topdir', 'selfupdate')

C:\Users\mrrosen\identity\test\simple>west update
=== updating zephyr (zephyr):
--- zephyr: initializing
Initialized empty Git repository in C:/Users/mrrosen/identity/test/simple/zephyr/.git/
--- zephyr: fetching, need revision v2.5.0
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 532644 (delta 1), reused 1 (delta 1), pack-reused 532642 eceiving objects: 100% (532644/532644), 365.25 MiB | 7.26 MiB/s
Receiving objects: 100% (532644/532644), 368.58 MiB | 7.59 MiB/s, done.
Resolving deltas: 100% (403942/403942), done.
From https://github.com/zephyrproject-rtos/zephyr
 * tag               v2.5.0         -> FETCH_HEAD
 * [new tag]         v1.0.0         -> v1.0.0
 * [new tag]         v1.1.0         -> v1.1.0
 * [new tag]         v1.1.0-rc1     -> v1.1.0-rc1
 * [new tag]         v1.10.0        -> v1.10.0
 * [new tag]         v1.10.0-rc1    -> v1.10.0-rc1
                   [snip]
 * [new tag]         zephyr-v2.1.0  -> zephyr-v2.1.0
 * [new tag]         zephyr-v2.2.0  -> zephyr-v2.2.0
 * [new tag]         zephyr-v2.2.1  -> zephyr-v2.2.1
 * [new tag]         zephyr-v2.3.0  -> zephyr-v2.3.0
 * [new tag]         zephyr-v2.4.0  -> zephyr-v2.4.0
 * [new tag]         zephyr-v2.5.0  -> zephyr-v2.5.0
Updating files: 100% (14668/14668), done.
HEAD is now at fe7c2efca8 release: Bump version to 2.5.0
HEAD is now at fe7c2efca8 release: Bump version to 2.5.0
=== updating littlefs (modules/fs/littlefs):
--- littlefs: initializing
Initialized empty Git repository in C:/Users/mrrosen/identity/test/simple/modules/fs/littlefs/.git/
--- littlefs: fetching, need revision 9e4498d1c73009acd84bb36036ee5e2869112a6c
remote: Enumerating objects: 2204, done.
remote: Total 2204 (delta 0), reused 0 (delta 0), pack-reused 2204 eceivin
Receiving objects: 100% (2204/2204), 1.09 MiB | 7.39 MiB/s, done.
Resolving deltas: 100% (1559/1559), done.
From https://github.com/zephyrproject-rtos/littlefs
 * [new branch]      zephyr          -> refs/west/zephyr
 * [new tag]         littlefs-v2.1.0 -> littlefs-v2.1.0
 * [new tag]         littlefs-v2.1.1 -> littlefs-v2.1.1
 * [new tag]         littlefs-v2.1.2 -> littlefs-v2.1.2
 * [new tag]         littlefs-v2.1.3 -> littlefs-v2.1.3
 * [new tag]         littlefs-v2.1.4 -> littlefs-v2.1.4
 * [new tag]         littlefs-v2.2.0 -> littlefs-v2.2.0
HEAD is now at 9e4498d lfs_util: fix log declaration when logging is disabled
HEAD is now at 9e4498d lfs_util: fix log declaration when logging is disabled

C:\Users\mrrosen\identity\test\simple>west hello
=== Hello!

If you delete the import statement for Zephyr, the command works fine:

C:\Users\mrrosen\identity\test\simple>west init -m https://github.com/mrrosen/manifest.git
=== Initializing in C:\Users\mrrosen\identity\test\simple
--- no --manifest-rev was given; using remote's default branch: refs/heads/main
--- Cloning manifest repository from https://github.com/mrrosen/manifest.git, rev. refs/heads/main
Initialized empty Git repository in C:/Users/mrrosen/identity/test/simple/.west/manifest-tmp/.git/
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 6 (delta 0), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), 1001 bytes | 27.00 KiB/s, done.
From https://github.com/mrrosen/manifest
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Note: switching to 'FETCH_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 96126e5 simple west test
--- setting manifest.path to manifest
=== Initialized. Now run "west update" inside C:\Users\mrrosen\identity\test\simple.

<<<<<<<<< Here I manually edited manifest/west.yml to remove just lines 18-20, the import of Zephyr

C:\Users\mrrosen\identity\test\simple>west hello
=== Hello!

[marc-hb]

the big thing we are replacing west update for to apply some patches to some repos after updating them since we rather get some of the larger repositories (like Zephyr) from the public git instead of maintaining our own clone (similar to what crosstools does, applying patch files to gcc).

Thanks, this clarifies.

First, I think you should really maintain your own clones and rebase them regularly but I understand automating a rebase can be scary so let’s leave that aside.

Now sticking to your dynamic patching approach there is a tool much better than patch to implement the exact same approach: git merge! In other words store your patches into a git branch somewhere; you do NOT need to rebase this branch regularly. Then perform a one-line git merge --no-edit my-custom-patches at build time.

Unlike patch, git merge is smart enough to realize something has already been merged and do nothing if you run it multiple times (idempotence). So you can make it part of west build and run it as many time as you want. I agree making “dynamic patching” part of west update feels much more logical, on the other hand making git merge part of west build Just Works. This is exactly what github continuously does for every Pull Request BTW, find the result in every pull/12345/merge

Other huge benefits of git merge over patch:

• no massive pollution of git status and git describe ---dirty
• proper version control with ability to reproduce builds.
• no need to edit .patch files any more (the horror)
• ability to breakdown the custom patches into multiple commits, no “per-file” restriction.
• etc.

(similar to what crosstools does, applying patch files to gcc)

It’s not because many people are still stuck with obsolete .patch files that they are a good idea 😃