[Discussion] Sub-distributions

We currently ship TFE as a single Python distribution, tf-encrypted, which includes both Python and C++ code. This makes the compilation process more involved than a simple git clone ... && pip install -e . as you need to have a range of build tools installed (including the right version of bazel). Especially MLers and cryptographers might benefit from doing (most of) their development in a pure-Python environment that can be compiled and tested using normal Python tools (and not bazel).

To get around this we could introduce a few sub-distributions, say:

• tf-encrypted-core: mixed code base containing e.g. custom ops
• tf-encrypted-dev: pure Python code base containing e.g. protocols and ML layers

We precompile and distribute both of these to PyPI, allowing developers to install tf-encrypted-core using pip and tf-encrypted-dev using git clone ... && pip install -e .... Both of them override any exiting installation of tf-encrypted, similarly to tf-nightly and tensorflow. Note that tf-encrypted-core will require e.g. bazel whereas tf-encrypted-dev can use standard Python tools.

We also continue to precompile and distribute tf-encrypted to PyPI since shipping a single distribution for non-development has some benefits, including mitigating versioning issues. I haven’t been able to find a Python process/tool for doing this by statically linking official dependencies, but as an alternative we could avoid official dependencies and instead simply pull in their codebases and compile everything together. This should be manageable using bazel

Note that in the future we may introduce e.g. tf-encrypted-runtime for the mixed code base related to the secure runtime. This fits nicely with the above approach.

[mortendahl]

Offline discussions lead to consensus. Key point was that we won’t allow fallback functionality in the future (ie around randomness) so there will never be e.g. non-compiled and pre-compiled functions that are expected to match.

[mortendahl]

Here’s a revision from discussions with @ianlivingstone.

This splits tf-encrypted-core into tf-encrypted-primitives (and potentially tf-encrypted-compiler and tf-encrypted-runtime down the line). It also reuses the tf-encrypted moniker for both the official package and what was tf-encrypted-dev above.

To install for non-development use: pip install tf-encrypted. This installs a standalone distribution without any dependencies (besides tensorflow and numpy). This will be the most common way of installing. It is built using bazel.

To install for development: git clone tf-encrypted && cd tf-encrypted && pip install -e .. This gives a pure-Python experience but you have to install additional distributions: for now just pip install tf-encrypted-primitives.

To pull this off we may need the following structure in the tf-encrypted repo:

./build/setup.py  <-- used for official standalone 'tf-encrypted'

./primitives/tf_encrypted/primitives/**  <-- mixed code
./primitives/setup.py  <-- used for 'tf-encrypted-primitives'
./primitives/BUILD

./tf_encrypted/protocols/**  <-- pure-Python code
./tf_encrypted/keras/**  <-- pure-Python code
./setup.py  <-- used for dev installation of 'tf-encrypted' (without primitives)
./BUILD
./WORKSPACE