Release new version that's compatible with `tfx[kfp]==1.10.0`

Currently master branch has Depends on tensorflow>=1.15.5,<2 or tensorflow>=2.10,<2.11 from this commit. But this isn’t released yet.

Here’s a minimal repro of the dependency conflicts with tfx[kfp]==1.10.0.

❯ cat requirements.in
tfx[kfp]==1.10.0
tensorflow-ranking

❯ pip-compile requirements.in
Could not find a version that matches tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,<3,>=1.15.5,>=2.10.0 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
Tried: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.10.0
Skipped pre-versions: 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.7.0rc0, 2.7.0rc1, 2.8.0rc0, 2.8.0rc1, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.11.0rc0, 2.11.0rc1
There are incompatible versions in the resolved dependencies:
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tfx-bsl==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-model-analysis==0.41.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tensorflow-transform==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-data-validation==1.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow<3,>=2.10.0 (from tensorflow-serving-api==2.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))

Any rough timeline of when this will be released? Thanks!

[davidxia]

Thanks, I should clarify that I’m able to repro dep conflicts with just the following.

cat requirements.in
tfx[kfp]==1.10.0

pip-compile requirements.in

Could not find a version that matches tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,<3,>=1.15.5,>=2.10.0 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
Tried: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.10.0
Skipped pre-versions: 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.7.0rc0, 2.7.0rc1, 2.8.0rc0, 2.8.0rc1, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.11.0rc0, 2.11.0rc1
There are incompatible versions in the resolved dependencies:
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tfx-bsl==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-data-validation==1.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5 (from tensorflow-model-analysis==0.41.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow<3,>=2.10.0 (from tensorflow-serving-api==2.10.0->tfx[kfp]==1.10.0->-r requirements.in (line 1))
  tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 (from tensorflow-transform==1.10.1->tfx[kfp]==1.10.0->-r requirements.in (line 1))

The conflict is specifically from pip-compile thinking tensorflow has to be both <2.10 and >=2.10.0. The <2.10 restriction is from tensorflow-transform==1.10.1->tfx[kfp]==1.10.0 (hence this issue) and interestingly also from tfx[kfp]==1.10.0 itself (corresponding issue).

More curious is that despite pip-compile failing, pip install -r requirements.in works with resulting packages.

tensorflow==2.9.2
tensorflow-serving-api==2.9.2
tensorflow-transform==1.10.1

[davidxia]

The pip-compile error I had above was actually incorrect behavior from legacy pip-compile resolver logic. pip-compile ≥ 6.10.0 with --resolver backtracking successfully resolves the dependencies.

The legacy resolver will occasionally fail to resolve dependencies. The backtracking resolver is more robust, but can take longer to run in general.

— https://pypi.org/project/pip-tools/