Installing dbt on GCP Cloud Composer image composer-1.14.0-airflow-1.10.14 fails with requests package error

Describe the bug

Installing dbt==0.19.0 on Cloud Composer image composer-1.14.0-airflow-1.10.14 fails with:

ERROR: dbt-core 0.19.0 has requirement requests<2.24.0,>=2.18.0, but you’ll have requests 2.25.0 which is incompatible. ERROR: snowflake-connector-python 2.3.6 has requirement requests<2.24.0, but you’ll have requests 2.25.0 which is incompatible.

Steps To Reproduce

Create a GCP cloud composer instance (https://cloud.google.com/composer) using the image composer-1.14.0-airflow-1.10.14, navigate to the PYPI Packages tab and attempt to install dbt with version ==0.19.0

Expected behavior

should install properly without errors

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

Which database are you using dbt with? Bigquery

The operating system you’re using:

The output of python --version: Cloud Composer runs 3.6.6 by default: https://cloud.google.com/composer/docs/concepts/python-version

[svirs]

Yup that’s exactly the plan, found a nice demo of it as well over here: https://medium.com/analytics-vidhya/adapting-a-real-world-data-pipeline-for-my-pet-project-ca536d545409 https://gitlab.com/snowboard-analysis/analytics/-/tree/master/ https://gitlab.com/snowboard-analysis/data-image/-/tree/master

[jtcohen6]

@svirs The reason this doesn’t work is exactly as the error message states: dbt-snowflake==0.19.0 depends on snowflake-connector-python==2.3.6, which requires requests<2.24.0,>=2.18.0, whereas composer-1.14.0-airflow-1.10.14 requires requests==2.25.0.

I know that the snowflake-connector-python team have been working to remove upper bounds on popular packages, and 2.3.8 vendored pieces of requests and bumped the requirement to 'requests<3.0.0'. In the next minor version of dbt, we’ll bump the snowflake-connector-python pin to the latest available stable release, and that should ease installation for everyone.

~In the meantime, since you’re using dbt with BigQuery, you should be able to get around this by installing dbt-bigquery in particular:~

pip install dbt-bigquery==0.19.0