Proposal: use entrypoints to make array-api submodules discoverable

Over in https://github.com/HypothesisWorks/hypothesis/pull/3065, we’re working on test tooling that takes an array-api implementation (e.g. numpy.array_api, etc) and returns test helpers which use that implementation. For ergonomics we’d also like to automatically find the array-api implementation, if there is one, for a given module - but as I understand it there’s no way to discover that mapping. Yet!

I therefore propose that as part of the Array API standard, packages implementing the API should register an entry point declaring which (sub)module contains the implementation. For example, Numpy would register:

        entry_points={
            'console_scripts': f2py_cmds,
            'array_api': ['numpy = numpy.array_api'],  # new: maps top-level name to array api implementation
        },

Implementations can then be found using importlib.metadata (see here for < py3.7) as follows:

import numpy.array_api
from importlib.metadata import entry_points

try:
    eps = entry_points(group="array_api")
except TypeError:
    # Load-time selection requires Python >= 3.10 or importlib_metadata >= 3.6, so fallback:
    eps = importlib_metadata.entry_points().get("array_api", [])
implementations = {entry.name: entry for entry in eps}

npa = implementations["numpy"].load()
assert npa is numpy.array_api

This allows us to avoid hard-coded lists of known libraries and locations, which will inevitably be both incomplete and out of date as the ecosystem evolves. It also makes it easy to find all installed array-api implementations, supporting a much wider range of purposes than testing 🎉

[honno]

Early last month in the consortium meeting, this was discussed and there seemed to of been consensus that entrypoints would be a good optional feature for this years spec release.

The two immediate use-cases:

• Hypothesis with entrypoints could enable a user-friendly pattern to access module-bounded Array API strategies, e.g. from hypothesis.extra.array_api import torch as tst would be much nicer than asking the user to “bound” the array module into a strategies namespace
• The Array API test suite could default to the Array API implementation it can find with entrypoints, so array module developers don’t need the cognitive load of “setting” the module to be tested

And as @Zac-HD originally stated, other tools that end up supporting the Array API might find entrypoints useful.

As @rgommers stated, entrypoints only uses features in the 3.8+ standard library. They are already used for NumPy and PyTorch. I made a PR to NumPy which demonstrates how simple this is to support in numpy/numpy#19800, as well as how to test it.

So this should be fine to add to the spec, although I’m not sure where. Being optional saves us from any unforeseen consequences, and down the line folk (i.e. me) can ask array module authors which don’t initially adopt it to see if they’re interested/even knew about it.

[kgryte]

@honno I think a reasonable inclusion place would be in the “How to adopt this API” section of the specification. Feel free to submit a PR adding this guidance. 😃