New verb proposal: fetch

In our lab, we found that we need this functionality for managing our project dependencies.

Roughly, what we expect this to accomplish is to go through the repositories, analyze their dependencies, and perform a git clone from some default location. Additionally, the dependencies could be set explicitly in the package.xml under <export> tag, like:

<export>
    <git_url target="package_name" url="<some_url>/package_name.git"/>
</export>

I have implemented a python module for that and we currently call it from cmake running over CMakeLists.txt of a particular package, which does the job, but the downloaded package is of course excluded from current catkin build as it was not there before cmake started running.

There are no additional dependencies involved. All the downloads are managed using subprocesses from within python.

The current python module is easy to convert into a verb as described in the documentation to catkin_tools.

Would you be open to accept a pull request for this?

PS. I feel like clone doesn’t seem to be a really fitting name, so any input is welcome.

[wjwwood]

Let me start by saying, I’d prefer to not put this software in this repository. I believe it would be useful to have these features built into the catkin tool, but not at the expense of added code here or more dependencies to this project. I have worked hard to ensure you can extend this tool from a different Python package, see: https://catkin-tools.readthedocs.io/en/latest/development/extending_the_catkin_command.html It notes that these entry points do not need to be in the catkin_tools package. So you could make your own Python package, e.g. catkin_tools_vcs_tools or catkin_tools_fetch, and put this verb there. This is similar to how people extend Sphinx and this is how I intended the tool to be extended and so I don’t want to accept new verbs that add significant functionality and/or new dependencies.

---

Can you expand on why you don’t think the existing tools the ROS community uses to manage source checkouts are not sufficient? Tools like wstool (http://wiki.ros.org/wstool) and rosinstall_generator (http://wiki.ros.org/rosinstall_generator). We don’t include functionality in catkin_tools that you’ve described because we have that functionality in those other tools already.

wstool is generally useful, whether or not your software is released in the ROS ecosystem. It operates on .rosinstall files, which are just yaml files with repository tuples (vcs type, url, version/branch/commit). It can clone them, change their version, check for difference and so on. It supports git, hg, svn, bzr, and can download and extract .tar files from urls.

If your software depends on other packages that are released in ROS’s ecosystem, then you can use rosinstall_generator to create these .rosinstall files automatically. But if you’re code isn’t released in the ROS ecosystem or even does not depend on software in the ROS ecosystem, then this tool won’t be very useful for you.

rosdep is only useful if you want to install the binary packages for your dependencies. It doesn’t help you checkout the source code for them, in the case where you want to build them from source or modify them before building them for some reason. It’s still a very useful tool, especially when working with code from source. It does have some ROS specific things about it, but when used with catkin, this is not very noticeable. I think you should give a try before deciding not to use it.

---

You mentioned that wstool using a single file to express the different locations of software was unnecessary. I don’t understand what you mean by this. How do you know where to clone the software from in the first place without the package.xml? Are you putting the urls of your package’s dependencies in the package.xml or the url of the package itself?

For the former, that’s not going to scale well, as you have to explicitly list the url and version of each dependency in every package. Consider the case where you have a package A, with packages B, C, D, and E all depend on it. Now you have to put the address of A (url and branch) in each of the packages B, C, D, and E. Now consider having to change the address of A, or consider that you don’t maintain B, C, D, and E. Imagine that B and C got updated to A’s new location/version, but D and E did not. Also consider that changing the package.xml of B, C, D, and E might require that you make a new versioned release of those packages if the users are getting them from binaries. It is much more flexible to use the name of the dependency and separately have a centralized, but overridable mapping from dependency name to method for how to get it. That’s essentially what our ros/rosdistro files are, and that’s the purpose of rosinstall_generator. Entries in the ros/rosdistro like https://github.com/ros/rosdistro/blob/be5ad6d3b44ab1a9869c520a74c95f9b3771d39e/kinetic/distribution.yaml#L40-L44 get converted into .rosinstall files using the rosinstall_generator tool. You can create your own ros/rosdistro if you want too. If you feel that for your needs, the packages will not change that often, then just create a .rosinstall file once, for all of your packages. Or create a .rosinstall file in each of your package’s directory, next to the package.xml, rather than storing this information in the <export> section of the package.xml. Then the existing tools like wstool can just operate on those files.

For the latter, we have the <url> tag in the package.xml already, but I guess that’s not what you mean, since it would be very helpful unless you had collected the package.xml’s for everyone already. That tag is only mean for documentation purposes.

---

I think that if you want to convince people of your strategy for managing the software, you’ll need to give a better step by step demonstration on how you imagine it should be used and what issues it resolves. For example, the use of rosinstall_generator + wstool + rosdep + catkin build tools is demonstrated by our “from source” installation instructions:

http://wiki.ros.org/kinetic/Installation/Source

You’d need a similar “we’re trying to achieve X, and this is how we do that with this tool, step by step”. Then people can comment on each step of the process, making suggestions and what not.

[niosus]

So, I have pushed the verb to pypi. You can get it with pip install catkin_tools_fetch

I am closing this issue.