Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9

Describe the bug

This issue originally started when I merged a change into production to begin using the Shopify DBT package. The merge broke because our prod machine was still on dbt 17. So when I went to upgrade I hit the below issues.

Currently, I get the below stack trace when running dbt --version while trying to upgrade to dbt 18.1. I imagine this has something to do with my brew/python setup environment but am not quite sure exactly how to debug after about a day of brute forcing various solutions. Thanks for your help.

Steps To Reproduce

1. Install brew
2. Install Git and DBT via Brew
3. Run dbt --version

Expected behavior

dbt --version to return the updated version

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/bin/dbt", line 33, in <module>
    sys.exit(load_entry_point('dbt-core==0.18.1', 'console_scripts', 'dbt')())
  File "/home/linuxbrew/.linuxbrew/bin/dbt", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/main.py", line 11, in <module>
    import dbt.version
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/version.py", line 100, in <module>
    installed = get_installed_version()
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/version.py", line 29, in get_installed_version
    return dbt.semver.VersionSpecifier.from_version_string(__version__)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/semver.py", line 100, in from_version_string
    return cls.from_dict(matched)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 618, in from_dict
    cls.validate(data)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 957, in validate
    schema = _validate_schema(cls)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 183, in _validate_schema
    schema = schema_cls.json_schema()
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 942, in json_schema
    return cls._json_schema_recursive(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 913, in _json_schema_recursive
    schema = cls._schema_defs_from_cache(definitions)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 900, in _schema_defs_from_cache
    schema = cls._collect_json_schema(definitions)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 863, in _collect_json_schema
    properties[target_field], is_required = cls._get_field_schema(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 809, in _get_field_schema
    field_schema, required = cls._get_schema_for_type(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 783, in _get_schema_for_type
    raise ValidationError(f"Unable to create schema for '{type_name}'")
hologram.ValidationError: Unable to create schema for 'Optional'

The operating system you’re using: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1028-aws x86_64)

The output of python --version: Python 3.9.0 (I do have multiple installs of python which may be influencing the issue)

Additional context

$ python --version
Python 3.9.0

$ brew --version
Homebrew 2.5.8
Homebrew/linuxbrew-core (git revision 071501; last commit 2020-11-03)

My best assumption is the issue lies here with the billion python installs, but not sure.

$ whereis dbt
dbt: /home/linuxbrew/.linuxbrew/bin/dbt

$ whereis python
python: /usr/bin/python3.6 /usr/bin/python3.6-config /usr/bin/python2.7 /usr/bin/python2.7-config /usr/bin/python /usr/bin/python3.6m-config /usr/bin/python3.6m /usr/lib/python3.6 /usr/lib/python3.8 /usr/lib/python2.7 /usr/lib/python3.7 /etc/python3.6 /etc/python2.7 /etc/python /usr/local/lib/python3.6 /usr/local/lib/python2.7 /usr/include/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/python /home/linuxbrew/.linuxbrew/bin/python3.9 /home/linuxbrew/.linuxbrew/bin/python3.9-config /home/airflow/onedrop-scripts/transform/dbt/dbt-env/bin/python /usr/share/man/man1/python.1.gz

$ whereis brew
brew: /home/linuxbrew/.linuxbrew/bin/brew

$ which dbt
/home/linuxbrew/.linuxbrew/bin/dbt

$ which python
/usr/bin/python

$ which brew
/home/linuxbrew/.linuxbrew/bin/brew

Thank you so much

[jtcohen6]

Hey @Startouf, dbt-core==0.19.1 is compatible with py39, but one of the plugins included by pip install dbt (dbt-snowflake==0.19.1) is not compatible with py39. The next minor version (v0.20) will be.

I’m not positive if that clarifies the jinja issue you’re seeing, but suffice to say, it is expected that pip install dbt==0.19.1 + py39 don’t yet work together. If you know which adapter you’re using, however, and it isn’t Snowflake, you can install that in a py39 environment just fine, e.g. pip install dbt-postgres.

[jtcohen6]

The latest version of dbt (v0.18.1) does not support python 3.9. We plan to add py39 support (#2822) for the next release of dbt (v0.19.0). If it proves straightforward, I’d also like to release a patch version (v0.18.2) that is compatible with python 3.9 as well.

In the meantime, I am going to close this issue as a duplicate of #2822.