[RFC][Ecosystem] Hooks for integrating with the Qobj constructor : `__qobj__` interface

Hello, I’m the Lead developer of netket, a jax-based framework for using Machine-Learning techiques to study quantum systems (open and closed).

We are finalising the next release and as I am a big fan of extensibility, I would like our two frameworks to coexist and provide a sensible, easy api to work together. Mainly, what I would like to do is provide an easy-to-use no-documentation-needed to convert NetKet’s types to Qutip, so that it becomes easy to use it to check our variational calculations.

TLDR: I propose creating and documenting a __qobj__ interface so that arbitrary objects having this method will support conversion to a QuTiP Qobj.

–

NetKet has mainly three types that are concerned: hilbert spaces, operators and variational states. Hilbert spaces describe the space every object is defined upon, and can be easily converted to your dims format.

Operators are used to represent operators and super-operators acting on hilbert spaces. We use a very custom format roughly corresponding to a lazy kronecker product. Those object satisfy the numpy __array__ interface and can be converted by calling np.array(netket_operator) or np.asarray(netket_operator). We can also obtain sparse representations (but there is no nice api to do that, so we simply provide a method netket_operator.to_sparse().

Variational states can also be converted to vectors or matrices (kets and density operators) as they also support the __array__ interface.

I would like our users to be able to call Qobj(netket_operator) or Qobj(netket_state) and obtain the corresponding qutip object. We could, in principle support a netket_operator.to_qobj() conversion method, but I am strongly opposed to that as this is an ugly design pattern: a way to construct a Qobj already exists, it’s Qobj(...), and the simplest-to-use api is, in my opinion, to overload this constructor to perform the conversion. This is a standard design pattern in Julia, which I believe has the best ecosystem inter-compatibility thanks to a standardisation of this pattern through (multiple)dispatch.

Numpy too, does the same, through the __array__ interface: any object that defines a __array__(self, dtype=None)->np.ndarray method will be supported by np.asarray and np.array.

Proposal: I would like QuTiP to support a __qobj__ interface, meaning that your Qobj constructor should check if an object has this method, and if it has, then use it to convert it to a Qobj. It should be relatively simple and involve adding another else-if in your constructor

if isinstance(inpt, Qobj):
...
elif inpt is None:
...
...
elif hasattr(inpt, "__qobj__"):
    data = input.__qobj(options)
    self.shape = data.shape
   ...

The only thing to do would be discussing the interface itself and document what it should return.

This will allow other software in the future to integrate with qutip, too, and I believe will help shape the ecosystem.

[hodgestar]

Apologies for another long delay. I’d be happy to hop on a video call sometime to discuss the design.

I think the __qutip_qobj__ interface is likely to remain fairly simple and be just:

def __qutip_qobj__(self, copy=True):
    """ Return a Qobj representation of this object.

         Parameters
         ---------------
         copy : {True, False}, default True
             If True, a new independent Qobj should be returned. If False, a reference to an existing Qobj
             may be returned.

         Returns
         ----------
         Qobj
             The Qobj representation of this object.
    """   
    return qutip.Qobj(self._stuff, copy=copy)

and we can figure out how to make that work on our side.

How that looks under the hood depends quite a bit on whether the Data object will grow dims support or not. We have another use case for dims on the data layer (a tensor network data backend that we’re starting work on) but it is a bit of a philosophical shift in what the Data layer is, so we’ll need to think carefully (e.g. all Data operations would have to keep track of dims which is a big change; and it’s also a bit strange because the operations themselves are just 2D matrix operations, where dims are meaningless).

@PhilipVinc I’m not sure if we discussed this before, but do you have some short code snippets showing how you see this Qobj support being used in user code? Apologies if I missed such examples elsewhere in the discussion already.

[PhilipVinc]

— Let me begin by saying that I completely understand your concerns.
I am not trying to enforce a view on the QuTiP project, but rather as I see I would like the various frameworks in the vast ‘quantum’ ecosystem to coexist in such a way that makes easy for users to jump from one tool to the other easily.

I personally envisage an ecosystem where one can write down an hamiltonian in the tool of choice, obtain the exact time evolution with qutip, and maybe have a look at it’s semiclassical trajectory. Jump into netket and check if a variational representation can properly capture the correlations. Trotterize the hamiltonian and get a circuit, without worrying about how to convert from one framework to the other.

I want to start the discussion on this topic eagerly because I am aware that it will take a fairly long time. If there is anything I can do to make your future internal discussions easier, do let me know.

I just want to caution you in case you’re trying to work towards a PR

Indeed. I was just trying to play around with this.

If we’re going to do this (and we haven’t decided if we will yet), we’re going to need to get a lot of wide-ranging input from many different libraries

I completely agree with your approach. I’m simply trying to push for this to be something that will come out in a reasonable timeframe and not be something forgotten.

We’ve got to be concerned with backwards and forwards compatibility; what if a user wants to install and use QuTiP and a different library in the same environment without using them together, but can’t even have them coexist because of version incompatibilities in optional conversion features?

I do understand your point, and I see how you want to be backward and forward compatible, however I don’t think that my proposal would break different tools in the same environment. What I propose would simply prevent qutip.Qobj(otherlibraryobject) from working (with an explicative error message) if otherlibrary does not support the same qobj interface version. Everything else would work the same. You can still import and use the two libraries independently without issues.

It’s indeed possible to design around this and supporting at the same time different versions of the interface, however, and there’s value in doing so.

To me, this is absolutely one of the mistakes that Numpy made in their implementation, and we shouldn’t be copying it.

I don’t know enough of the history behind numpy current api, but I see your point. however I’d like to reiterate that this is not what I am proposing. I think there is considerable difference between supporting arbitrary operations new_obj = Qobj + netket_operator*5 (though if you are going to export dispatch hooks in 5.0, this would be possible in a consistent way, I think) and qutip.mesolve(netket_operator, ...). In the first, I do agree with your point that the nature python’s __add__/__radd__ is not commutative, leading to surprising behavior. From my point of view, no package really owns + so any conversion would be implicit. But qutip.groundstate(netket_operator) is in itself an explicit cast. I am explicitly asking to qutip please give me the ground state of this object. I do expect to get a Qobj out, and qutip to make an effort converting this object to whatever format he internally wants to work with. Same goes with time evolution.

For greater discussion, here’s an alternate approach: instead of objects defining qutip_qobj, instead we expose an entry-point qutip.Qobj.register_conversion_function(converter, type, priority, version=None), and downstream libraries register functions rather than defining methods on their classes

I do prefer such an approach. I just did not think that was an option in QuTiP. In fact we have redesigned NetKet to allow for something similar using multiple dispatch. (I’m not familiar with Tensorflow either.)

I do agree with all your points, especially the first.

As for your last point, about downstream packages having to import qutip: This is indeed a potential issue. I’ll try to investigate this: maybe there is a solution whereby a callback executed on package load can be registered with importlib?