Packaging of Java components

At the suggestion of @jakirkham, I am opening this issue here to start a conversation about how conda might manage the packaging for Java components in a uniform way.

Quick introduction: I am maintainer of the ImageJ and Fiji projects for N-dimensional scientific image processing and analysis. A major interest of mine is making ImageJ accessible from other ecosystems such as PyData and Node.js. We now have an imagej module on PyPI allowing to access an ImageJ installation as Python code in-process or interprocess. But it was suggested that we also create a conda package that ships all of ImageJ. @jakirkham indicated the most proper way to do this would be publish each Java component (i.e. JAR file) as its own conda package, with a dependency hierarchy matching that declared in the Maven POMs.

So my first question is: do others concur that the ideal thing would be a one-for-one correspondence between Java components and Conda packages?

If so, my second question then is: what would be the best way forward to achieve that?

@jakirkham suggested we could invent some machinery to auto-update conda packages of Java components based on their Maven releases. I countered with the idea that it would be more elegant to augment conda itself to understand Maven repositories directly, so that conda install java:org.scijava:scijava-common or some such would install the latest version of org.scijava:scijava directly from Maven Central.

A third related question is: would it be OK to simply ship binaries, rather than rebuilding from source? I am ignorant of the conda-forge (and conda in general) policy on building from source… but one nice thing about Java artifacts is that once a release is built, it is completely portable and there is no need to rebuild it—and in fact, rebuilding it can be problematic, since JAR files are typically signed by the author to ensure fidelity. (Of course, artifacts can be rebuilt from source by third parties, but the resulting JAR itself will not be identical, even if the bytecode of all the .class files inside is the same.) So it would be ideal if conda could support distributing the already-built artifacts rather than mandating rebuilds from source.

There are many issues here, and I expect the conversation will be quite complex—but I am willing to invest some energy if people in the community are earnestly interested in making Java components available via conda. See also imagej/imagej#162 which is about packaging Java components for Gentoo but has probably a high degree of overlap with many of the concerns here.

Thoughts? Questions? Ideas?

[kalefranz]

Conda’s original language ecosystem target was the python world. And I’m calling that out because we’re just now in the last several months starting to work out if/how we want to give conda the ability to directly install packages from PyPI. It’s not an easy problem to solve, even though we eventually want to get there someday. Ultimately, if we succeed, it’s not going to be something that’ll be complete in the next month or two. It could be a year or more before we’re able to directly install even a subset of packages (e.g. pure-python wheels) directly from a PyPI-like repository. This, after a half-decade of this conda community working most intensely around python.

I love this topic, and the discussion about more richly supporting the java community. I’m personally 👍 👍 . However, we need to start with conda packages. We need to build out a substantial base of java as conda packages, just as we have for python and r, so that we can understand the design, architecture, and usage patterns involved. So that we can gain enough expertise as a community for us to know what something much more advanced like having the conda solver interact with a maven repository, and having conda’s unlink-link transactions directly install artifacts from such repositories, might ultimately look like.

[ctrueden]

An update on the jrun front: it was renamed to jgo, to avoid a clash with an existing PyPI project of that name (which executes Jupyter notebooks from the CLI). @hanslovsky and I have released version 0.1.0 of jgo to PyPI. We also released a package called scyjava which is a union of jgo + pyjnius allowing easy import and usage of Java code from remote Maven repositories in Python. The plan is to submit jgo and scyjava also to conda-forge in future; in the meantime, from our perspective, packaging of Java components with Python code is now “solved”, without requiring Java components to be repackaged as conda packages. Here is an example from the scyjava documentation which invokes ImageJ from Python:

>>> import scyjava_config
>>> scyjava_config.add_repositories({'imagej.public': 'https://maven.imagej.net/content/groups/public'})
>>> scyjava_config.add_endpoints('net.imagej:imagej:2.0.0-rc-65')
>>> import scyjava, jnius
>>> System = jnius.autoclass('java.lang.System')
>>> System.setProperty('java.awt.headless', 'true')
>>> ImageJ = jnius.autoclass('net.imagej.ImageJ')
>>> ij = ImageJ()
>>> formula = "10 * (Math.cos(0.3*p[0]) + Math.sin(0.3*p[1]))"
>>> blank = ij.op().create().img([64, 16])
>>> sinusoid = ij.op().image().equation(blank, formula)
>>> print(ij.op().image().ascii(sinusoid))
,,,--+oo******oo+--,,,,,--+oo******o++--,,,,,--+oo******o++--,,,
...,--+ooo**oo++--,....,,--+ooo**oo++-,,....,,--+ooo**oo++-,,...
 ...,--++oooo++--,... ...,--++oooo++--,... ...,--++oooo++-,,...
   ..,--++++++--,..     ..,--++o+++--,..     .,,--++o+++--,..
   ..,,-++++++-,,.      ..,,-++++++-,,.      ..,--++++++-,,.
    .,,--++++--,,.       .,,--++++--,,.       .,,--++++--,..
    .,,--++++--,,.       .,,-+++++--,,.       .,,-+++++--,,.
   ..,--++++++--,..     ..,--++++++--,..     ..,--++++++-,,..
  ..,,-++oooo++-,,..   ..,,-++oooo++-,,..   ..,,-++ooo+++-,,..
...,,-++oooooo++-,,.....,,-++oooooo++-,,.....,,-++oooooo+--,,...
.,,,-++oo****oo++-,,,.,,,-++oo****oo+--,,,.,,,-++oo****oo+--,,,.
,,--++o***OO**oo++-,,,,--++o***OO**oo+--,,,,--++o***OO**oo+--,,,
---++o**OOOOOO**o++-----++o**OOOOOO*oo++-----++o**OOOOOO*oo++---
--++oo*OO####OO*oo++---++oo*OO####OO*oo++---++o**OO####OO*oo++--
+++oo*OO######O**oo+++++oo*OO######O**oo+++++oo*OO######O**oo+++
+++oo*OO######OO*oo+++++oo*OO######OO*oo+++++oo*OO######OO*oo+++

Of course, this does not solve the non-Python-centric “install Java packages via conda” question. But it bridges the two worlds in a fairly convenient and extensible way.