General setup of static outputs vs. shared ones

Static builds are already an exception in conda-forge (see CFEP-18), but some do exist for specific use-cases. The micromamba feedstock is a good example of a user of static libs (presumably to stay “micro” and have no runtime dependencies).

Several feedstocks of this kind follow a pattern as follows:

outputs:
  - name: libxyz
    build:
      run_exports:
        - {{ pin_subpackage('libxyz') }}
    [...]
  - name: libxyz-static
    requirements:
      build:
        - [...]
      host:
        - {{ pin_subpackage("libxyz", exact=True) }}
      run:
        - {{ pin_subpackage("libxyz", exact=True) }}

This has some advantages & disadvantages:

• 👍
  • deduplication of files between static & shared builds
  • shared & static builds are co-installable
    • I’d argue this is actually an anti-pattern, but it can become relevant for dependencies with run-exports, see below
• 👎
  • libxyz-static pulls in both dynamic and static builds
    • this makes it possible for static builds to “silently” depend on the shared builds
    • in practice this is not a huge issue because run-exports from libxzy don’t get picked up though for a -static host dep, so reliance on the dynamic lib would be detected by going boom at runtime
  • CMake integration is impossible for both packages simultaneously
    • either the targets are wrong for the non-static version (“libzstd.a” not found)
    • or the CMake-specific files clobber each other

The CMake issue in particular is quite painful, because an ever-increasing number of packages in C/C+±land come with built-in CMake integration. For example, recent LLVM-builds failed on https://github.com/conda-forge/zstd-feedstock/issues/58. There, I untangled the dependence in https://github.com/conda-forge/zstd-feedstock/pull/62, but this has the following trade-off:

[…] we can have only 3 out of the following 4 (AFAICT):

1. working CMake integration
2. no clobbering of CMake files
3. no manual hacking (of CMake files resp. upstream CMake integration)
4. zstd & zstd-static co-installable

Currently I’ve chosen to give up 4. - perhaps an argument can be made that giving up 2. is less harmful despite being against best practice.

The lack of being able to co-install libxyz and libxyz-static would become problematic in the following scenario (quoting @hmaarrfk):

1. User builds package a, which depends on zstd dynamic. zstd exports a requirement of zstd.
2. User tries to build package b that depends on a and zstd-static. This user, can no longer build their package.

Package b cannot depend on just zstd-static because the dependency on zstd is controlled by package a.

In this case, that PR was merged (and the existing consumers are not affected by the run-export-induced conflict between dynamic & static libs), but now I’ve encountered the same in libprotobuf, and fixing https://github.com/conda-forge/libprotobuf-feedstock/issues/68 is not possible without running into the same issue, hence I thought I’d open a wider discussion.

@hmaarrfk sketched out a different approach:

The other alternative, which I don’t have time to flesh out, is to return to the previous system, but, to ensure that:

1. cmake works with the dynamic libraries.
2. cmake fails to automatically report the static libraries.

This would favor the dynamic library usage, instead of forcing users of cmake+zstd to have the static libraries installed.

to which I said:

Wouldn’t that defeat the point of zstd-static? If I use an output named like that, I’d certainly expect the static lib to be picked up, not the dynamic one (i.e. compiling against zstd-static would seem to work, but would actually do the same as compiling against zstd, i.e. create a runtime dependence on zstd)

Summary

• The setup currently used by several static outputs is incompatible with CMake, which is IMO a bigger downside than getting some file duplication and even co-installability
• In general, I think it’s really iffy to have static and shared builds of the same libs in the same host environment.
• There are other feedstocks where this setup is not possible/sensible, e.g. https://github.com/conda-forge/abseil-cpp-feedstock/pull/35

Proposal: Forbid co-installation of libxyz-static builds with libxyz

This would solve the CMake issues (each output could have its own copy of the respective CMake files without risks of inconsistency or clobbering). The only downside would be that users of -static packages cannot depend on another package (say a) which has been built against a shared version of the same lib. Since most of the static libs live pretty close to the bottom of the stack, I don’t think this would be much of a problem. And even so, there would be a solution: build a (perhaps also static?) version of a against libxyz-static.

I think (users of) static libs are special enough that we can inflict this (hypothetical!) pain on them

Thoughts @conda-forge/core?

[carterbox]

Is the following an accurate summary of the discussion?

No top-level package. Build shared first output. Build static second output. Static output depends on shared output in both run and host. Use tests to check that shared output excludes static libraries. Static CMake files probably clobber shared CMake files (depends on build config)?

[h-vetinari]

You have the solution right there. Add libxyz as a host dep in libxyz-static. For eg: Add - libzstd at https://github.com/conda-forge/zstd-feedstock/blob/main/recipe/meta.yaml#L90

Needed to be a run-dep as well (to get the headers), but yes, this works (though it’s unclear to me which lib gets found by CMake; but I don’t care that much anymore, as this approach seems to solve the most issues at once).