Cannot run dbt docs generate with JSON logs

Describe the bug

When running dbt docs generate with JSON logs enabled I receive an error: Encountered an error while generating catalog: Object of type DatabricksRelation is not JSON serializable. This occurs when using dbt-databricks 1.1.0 on all of locally (Windows), Docker (Linux) and the preview Databricks dbt task type. It does not occur in earlier versions. It does not occur with the default log format.

Steps To Reproduce

1. Run dbt --log-format json docs generate

Expected behavior

Doc site generates correctly, with JSON logs.

Screenshots and log output

{"code": "E044", "data": {}, "invocation_id": "4240ec3d-ef2f-4772-8540-d4d522a8c717", "level": "info", "log_version": 2, "msg": "Building catalog", "pid": 1628, "thread_name": "MainThread", "ts": "2022-05-25T14:22:49.779873Z", "type": "log_line"}
{"code": "Z046", "data": {"log_fmt": null, "msg": "Encountered an error while generating catalog: Object of type DatabricksRelation is not JSON serializable"}, "invocation_id": "4240ec3d-ef2f-4772-8540-d4d522a8c717", "level": "warn", "log_version": 2, "msg": "Encountered an error while generating catalog: Object of type DatabricksRelation is not JSON serializable", "pid": 1628, "thread_name": "MainThread", "ts": "2022-05-25T14:22:49.783320Z", "type": "log_line"}
{"code": "E041", "data": {"num_exceptions": 1}, "invocation_id": "4240ec3d-ef2f-4772-8540-d4d522a8c717", "level": "error", "log_version": 2, "msg": "dbt encountered 1 failure while writing the catalog", "pid": 1628, "thread_name": "MainThread", "ts": "2022-05-25T14:22:49.792754Z", "type": "log_line"}

System information

Windows:

Core:
  - installed: 1.1.0
  - latest:    1.1.1 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.1.0 - Up to date!
  - spark:      1.1.0 - Up to date!

The operating system you’re using: Windows/Linux/Databricks

The output of python --version: Windows: Python 3.9.0

Additional context

[jtcohen6]

@bilalaslamseattle Thanks for flagging this again.

We’ve seen multiple issues in this category, across multiple adapters, and we think there exists a general-purpose solution that will be the right move longer-term: https://github.com/dbt-labs/dbt-core/issues/5436

The work for that is definitely on our radar. If it appears that the general-purpose resolution will be too complex, we can put a one-off patch for this in dbt-spark, to unblock the user here.

(cc @nathaniel-may)

[VersusFacit]

Hiya people on the thread.

Per this core PR, this JSON serialization bug should be solved across all adapters. There’s a lot of layers of indirection in the logger call stack, so finding the root cause of this error took us some concerted time. I also threw on our PRs backlog tags, so in theory, you should be able to “seamlessly” integrate the fix into your env on the new release. (it’s also live in main)

I’d love to close this if (🤞) people report things working here.