I'm not able to run code developed using dbt directly in Databricks.

Describe the bug

I’m not able to run code developed using dbt directly in Databricks.

I’m trying to configure a CI/CD process using Databricks jobs to automatically run the SQL codes. We usually do this using Airflow and this is the first time we are configuring it directly in Databricks. However, when trying to connect, DBT (or Python, I don’t know) gives an error that it doesn’t have the logs function.

Steps To Reproduce

1. Create a venv with Python 3.8.10 py -m venv venv source venv/Scripts/activate

In this case, I’m on Windows 10, but I don’t think it will impact the next steps for reprodute.

1. Install and upgrade dbt-databricks

pip install dbt-databricks pip install --upgrade dbt-databricks

1. Generate a dummy project

dbt init jaffle_shop

1. Check if connection is working

dbt debug

1. Commit all changes in a repository test

2. Add repository in Repos Databricks and give a pull refresh

3. Create a notebook in same diretory for test

4. Install requirements and debug dbt In this case, we have only dbt-databricks to install

%pip install dbt-databricks %sh dbt --version

%sh dbt debug

Expected behavior

The expected result of dbt debug is the connection indicator working. Like this:

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

The output of dbt --version:

Additional context

We tried to install another version of dbt-databricks, but every time we try, the installed version always comes 1.0.4, as if it didn’t obey the command directed.

As happen in this case:

[bilalaslamseattle]

@ac-garcia @binhnefits @guizsantos I’m going to set up a “drop in” demo for next week. I’d love to get your feedback on the experience and, if you like it, we can enroll you in the preview right there.

Here are some times you can drop by next week:

EMEA-friendly time: Monday, May 23 at 15.00 CET. Google Meet link: https://meet.google.com/zzc-qpvo-vrn North America-friendly time: Tuesday, May 24 at 10.00 PT. Google Meet link: https://meet.google.com/zee-bdcj-eak

Looking forward to seeing you there!

[bilalaslamseattle]

@ac-garcia thanks for opening the issue. If you are trying to run a dbt project as a job, we are actually previewing this capability right now. Would you be interested in trying it out? Then there’s no need to run a notebook.