dbt 0.19.0 breaks dbt example and likely dagster-dbt

Summary

The following failure is occurring in dbt_example_tests/test_pipeline.py. It started failing on the same day that dbt 0.19.0 was released. Pinning dbt to <0.19.0 seems to fix it.

../../python_modules/libraries/dagster-dbt/dagster_dbt/cli/solids.py:195: in dbt_cli_run
--
  | cli_output = DbtCliOutput.from_dict(cli_output_dict)
  | ../../python_modules/libraries/dagster-dbt/dagster_dbt/cli/types.py:83: in from_dict
  | result=DbtResult.from_dict(d),
  | ../../python_modules/libraries/dagster-dbt/dagster_dbt/types.py:169: in from_dict
  | results = [NodeResult.from_dict(d) for d in check.is_list(d["results"], of_type=Dict)]
  | ../../python_modules/libraries/dagster-dbt/dagster_dbt/types.py:169: in <listcomp>
  | results = [NodeResult.from_dict(d) for d in check.is_list(d["results"], of_type=Dict)]
  | ../../python_modules/libraries/dagster-dbt/dagster_dbt/types.py:102: in from_dict
  | node = check.dict_elem(d, "node")
  | ../../python_modules/dagster/dagster/check/__init__.py:770: in dict_elem
  | CheckError("{key} not present in dictionary {ddict}".format(key=key, ddict=ddict))
  | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  |  
  | exc = CheckError("node not present in dictionary {'status': 'success', 'timing': [{'name': 'compile', 'started_at': '2021-01...ter_response': {'code': 'SELECT', 'rows_affected': 77}, 'unique_id': 'model.dbt_example_project.normalized_cereals'}",)
  | traceback = None
  |  
  | def raise_with_traceback(exc, traceback=Ellipsis):
  | if traceback == Ellipsis:
  | _, _, traceback = sys.exc_info()
  | >       raise exc.with_traceback(traceback)
  | E       dagster.check.CheckError: node not present in dictionary {'status': 'success', 'timing': [{'name': 'compile', 'started_at': '2021-01-27T21:59:57.615495Z', 'completed_at': '2021-01-27T21:59:57.665017Z'}, {'name': 'execute', 'started_at': '2021-01-27T21:59:57.675556Z', 'completed_at': '2021-01-27T21:59:57.793769Z'}], 'thread_id': 'Thread-2', 'execution_time': 0.18580341339111328, 'message': 'SELECT 77', 'adapter_response': {'code': 'SELECT', 'rows_affected': 77}, 'unique_id': 'model.dbt_example_project.normalized_cereals'}

Weirdly, none of the dagster-dbt tests are failing, but this seems like a failure that should not be specific to the example.

Reproduction

Dagit UI/UX Issue Screenshots

Additional Info about Your Environment

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We factor engagement into prioritization.

[chenbobby]

@sryza Weirdly, none of the dagster-dbt tests are failing, but this seems like a failure that should not be specific to the example.

I think this is because we pin dbt==0.17.* in our setup.py for testing. I changed it to dbt==0.19.0 and the dagster-dbt tests started failing on buildkite, with the same error that you reported.

dagster.check.CheckError: node not present in dictionary

The tricky part with dagster-dbt is that you can’t pin the version of dbt that you use with dagster-dbt. This is because dbt is installed as a CLI tool, not as a Python package dep in dagster-dbt.

---

@erik-attest Thanks for linking that issue from fishtown-analytics/dbt. Very useful 🙏

The release notes for dbt 0.19.0 indeed list out some GitHub issues and PRs that revamp their JSON schemas for dbt Artifacts.

Starting with 0.19.0, dbt is documenting their JSON schemas for dbt Artifacts. Unlike before, these JSON schemas will be versioned, which is very nice 👌. However, the 0.19.0 JSON schema is a breaking change from <0.19.0, which is less nice. For curious readers, the JSON schemas can be found here.

Currently, the Dagster solids in dagster-dbt tries (and fails) to parse the dbt run results JSON so it can yield Output and yield AssetMaterializations. As @abelsonlive commented above, you can rip out this logic to avoid this error.

---

Next Steps

The dbt Artifacts are a good use case for AssetMaterializations, so I don’t want to just rip that out. I’m going to work on a diff that can support the JSON schemas before and after dbt 0.19.0, but I would imagine that we eventually want to stop supporting JSON schemas from dbt <0.19.0.

I’d be open to hearing people’s opinions about how long we should keep supporting JSON schemas from dbt<0.19.0.

[chenbobby]

I’m closing this issue as commit 7bb8db4 has landed on master to fix this breakage.

There is still some schema compatibility work to be done, and I have created issue #3673 to track that work.