Support use of `git worktree` in west

Today, the manifest in Zephyr might contain multiple entries of the same repository, but in different versions (SHAs), such as:

• mcuboot for both Zephyr and TF-M https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml#L96-L98

    - name: mcuboot
      revision: 3fc59410b633a6d83bbb534e43aac43160f9bd32
      path: bootloader/mcuboot
  

  https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml#L147-L150

    - name: tfm-mcuboot # This is used by the trusted-firmware-m module.
      repo-path: mcuboot
      path: modules/tee/tfm-mcuboot
      revision: 1.7.0-rc1
  

• mbedtls in Zephyr and TF-M (mbedtls is cloned during build time (ninja), but there is work ongoing to see if could be improved): https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml#L93-L95

   - name: mbedtls
     revision: 24d84ecff195fb15c889d9046e44e4804d626c67
     path: modules/crypto/mbedtls
  

  https://github.com/zephyrproject-rtos/trusted-firmware-m/blob/96340fb6c0b7e31c2070e1f428ca24076d2700a6/trusted-firmware-m/lib/ext/mbedcrypto/CMakeLists.txt#L22-L23

            GIT_REPOSITORY https://github.com/ARMmbed/mbedtls.git
            GIT_TAG ${MBEDCRYPTO_VERSION}
  

This means that west will clone and update the same repository multiple times (mcuboot) instead of a single fetch.

If git worktree was used, then a single fetch would be enough, and the two paths would be sharing same repo but having different versions.

As example, we could have:

    - name: mcuboot
      revision: 3fc59410b633a6d83bbb534e43aac43160f9bd32
      path: bootloader/mcuboot
      worktrees:
        - name: tfm-mcuboot # This is used by the trusted-firmware-m module.
          path: modules/tee/tfm-mcuboot
          revision: 1.7.0-rc1

would allow to just fetch mcuboot once, but still allow for two different versions to be used for Zephyr and TF-M.

Similar improvement will be possible for mbedtls in future.

[vapier]

the default git-repo behavior of using symlinks for internal .git state to share objects has a number of downsides that can lead to data loss if you try to use it for more than one checkout. i would not recommend it at all. git gc and related operations run in the leaf repo and don’t see any of the others, so it can discard objects that the other repos are using. and the worst part is that these operations tend to kick in weeks/months after use 😦. i prob should add a warning note block to the internal-fs-layout.md file in case someone tries to use it as a model for their own thing.

worktrees avoid that because every leaf repo has visibility into refs of all other repos. i get that the behavior will be diff and a bit weird at first, but i would strongly recommend trying to utilize standard git features rather than do your own thing behind git’s back whenever possible.

along those lines, git itself is pushing a “superproject” approach for creating monorepos from multiple repositories. we have people in Google seeing if they can use this flow to replace git-repo usage entirely for Android & Chromium OS, and adding features to git when something is missing.

wrt relative paths, git-repo worksaround it by manually rewriting the .git files. it’s ugly, but seems to be working. i filed a report for git to hopefully support this itself someday. https://bugs.chromium.org/p/git/issues/detail?id=47

Can this be auto-detected by seeing that another entry has the same URL? That would make this work across separate manifests.

fwiw, in git-repo we specifically avoid using the base URIs as indexes to support mirror switching. we allude to that in the doc Marc referenced earlier:

This layout is designed to allow people to sync against different remotes (e.g. a local mirror & a public review server) while avoiding duplicating the content. However, this can run into problems if different remotes use the same path on their respective servers. Best to avoid that.

[tejlmand]

This in particular is not a concern to me, as west could be using git worktree move in this case.

How? If the manifest moves a project directory, west has no idea where the old one is in the workspace, because west update doesn’t pull.

Right, if we strictly mimic today’s behavior on how west simply abandon a project that is moved / removed in a manifest. west would leave the old project / repo in the workspace and never clean up, and just create a new project at the new location. That would be no different than to add a new worktree, and rely on the user to remove the abandoned one.

However, with worktrees, west would in principle be able to use the git worktree list to obtain knowledge of worktrees, and through that issue a move instead of abandoning. This would of course require a form of understanding, similar to manifest-rev, to ensure west wouldn’t move a user created worktree.

But I guess exploring such options makes little sense now.