Decide if custom ops should continue to be optional

Right now TFE will work without custom ops such as secure randomness, falling back to TensorFlow’s insecure randomness. This means we can have some support on platforms that we do not publish official PyPI packages for, eg via source installations. Moreover, developers essentially only need to git clone to get started, and don’t have to go through a bootstrapping step first. However, as we add more functionality or optimizations, it could become hard to maintain these alternatives.

Options:

1. require custom ops to be present, making no attempts at falling back to alternatives; source code installations will still be possible but you will need a build environment

2. allow some functionality to simply be missing if the custom op is not there, causing an error if needed by the concrete application; this could eg be the case for protocols that heavily rely on custom ops such as HE.

3. continue to allowing different versions of functionalities, for instance falling back to a TensorFlow default if the custom op is missing; this could eg be the case for secure randomness

Note that in order to support third party MPC implementations we will in either case need some way of running with functionality missing, since these should not be bundled up in the main TFE pypi package (although potentially offered as extra pypi packages: tf-encrypted-scalemamba etc)

Child of #568

[mortendahl]

There is another option here, namely that TFE becomes free of protocols and they all move into sub-packages. This includes Pond and SecureNN. In this way TFE becomes somewhat more like Keras, and can likely stay free of custom-ops. It also puts all protocols on equal footing. All the ML op top of the protocols would still be in TFE, and probably a lot of common protocol functionality as well (orchestration, queues, etc). A middle-ground could be to include one or two pure-Python protocols with TFE (Pond?) but let any optimized versions with custom-ops be sub-packagess. That way users may still run on systems for which we don’t have binary packages.

cc @jvmancuso @justin1121 @ianlivingstone

[ianlivingstone]

After some thought, I’d suggest:

• Focus on reducing complexity as much as possible (both for maintainers and users).
• Our builds for each platform and operating system should contain a full suite of compiled operations – I shouldn’t have to do something extra to get the “base” working.
• A plugin system for more serious extensions makes sense which would require some sort of registration mechanism. Those plugins should conform to the previous point: they work out of the box for your platform and operating system to the best of our ability.
• Steer clear of duplicating functionality as much as possible – which means having a clear line around what’s included in the “base” by default and what is an extension.

With respect to licensing, if the license is copy-left, we should have those libraries outside of the primary repository with a clear warning regarding the underlying software. It’s important that copy-left licenses don’t end up inside the main package as it will be a barrier to company adoption.