Refactor package distribution

Currently OpenCensus Python is released as a monolithic package pip install opencensus which has the following disadvantages:

1. One will have to install all the dependencies even if most of them are not being used. This also adds concerns on deployment time/size and security.
2. Some of the exporters / integration with 3rd party libraries have to bind with the core OpenCensus Python release schedule. This is problematic for both the core OpenCensus Python and 3rd party integrations when it comes to the release / hotfix schedule.

The proposal is to refactor the current monolithic package using the namespace packaging approach https://packaging.python.org/guides/packaging-namespace-packages/. Specifically, https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages will be used given we need to support both Python 2.x and 3.x.

[c24t]

Following up from a conversation with @reyang:

We have two related problems here: how to split up the existing single distribution package (following the packaging docs, I’ll call this a distribution), and how/whether to split the code in this repo into multiple repos. For now I’m assuming that we want to keep the existing package/namespace structure, and it looks like namespace packages let us do exactly that while moving the code between distributions and repos.

It’s clear that we need a separate distribution for third party trace integrations (opencensus.trace.ext), but not clear whether we should move these into separate repos. For comparison, java client integrations are kept in the client repo, but in a separate contrib package. Keeping multiple distributions in the same repo is standard practice (see e.g. the GCP python client library repo, which includes a few dozen distributions), but it does make development more complicated.

We may also want to move the exporters (opencensus.trace.exporters, opencensus.stats.exporters) into a separate distribution, or move each exporter into its own distribution. Unless we decide that we want a single distribution per repo, I think there’s less reason to move these than third party integrations.

Then there are other components like the google cloud format propagator that are backend-specific, but can’t be easily extracted from the library. I don’t think we should touch this now, but worth keeping in mind if we want to prioritize having a small core library later.

[reyang]

Do you mean using a namespace package to make the package name consistent with the other integrations in the library? Or moving the code into the repo alongside the other integrations? If the former that sounds great too.

The former.

It looks like we can use environment markers to solve the problem of integrations importing different packages at runtime (see this blog for an example), but this may be a headache.

In any case I think it’s more important that we enforce a minimum version for e.g. flask than that we install the right version of flask on installing the OC integration.

Thank you! I’ll investigate.

@reyang do you have an opinion on which integrations should go in the client library repo and which should live in their own repo in https://github.com/opencensus-integrations?

I don’t have a very strong opinion. I believe the httplib/request and things that come with the default Python installation should be included into the core opencensus repo. For 3rd party things like MySQL, Flask, Django, I think it is mainly determined by the maintainer’s preference.