Compatability with dbtutils package.

Hi,

This is an amazing project and I am so excited to have stumbled upon it. My goal is to use this with DBT, but even after reading the documentation I am having trouble setting it up.

My environment:

• dbt_utils package installed in ./dbt_modules/dbt_utils/macros
• custom macros in ./macros/

If I run it with an empty config file:

$ sqlfluff lint models/01_source/is/src_is_other_tag_applications.sql                                                                            [15:03:45]
== [models/01_source/is/src_is_other_tag_applications.sql] FAIL
L:   - | P:   - |  TMP | Unrecoverable failure in Jinja templating:
                       | 'cast_is_datetime' is undefined. Have you configured
                       | your variables?
                       | https://docs.sqlfluff.com/en/latest/configuration.html
L:  10 | P:  12 |  TMP | Undefined jinja template variable: 'cast_is_datetime'
L:  11 | P:  12 |  TMP | Undefined jinja template variable: 'cast_as'
L:  12 | P:  12 |  TMP | Undefined jinja template variable: 'cast_as'
L:  27 | P:  12 |  TMP | Undefined jinja template variable: 'dbt_utils'

When I add a config with the macros path:

[sqlfluff:templater:jinja]
load_macros_from_path=macros

Then I get…

$ sqlfluff lint models/01_source/is/src_is_other_tag_applications.sql                                                                            [15:16:35]
== [models/01_source/is/src_is_other_tag_applications.sql] FAIL
L:   - | P:   - |  TMP | Unrecoverable failure in Jinja templating: 'dbt_utils'
                       | is undefined. Have you configured your variables?
                       | https://docs.sqlfluff.com/en/latest/configuration.html
L:  27 | P:  12 |  TMP | Undefined jinja template variable: 'dbt_utils'

When I add a config with the dbt_modules path:

[sqlfluff:templater:jinja]
load_macros_from_path=dbt_modules

Then I get an exception:

$ sqlfluff lint models/01_source/is/src_is_other_tag_applications.sql                                                                            [15:16:37]
Traceback (most recent call last):
  File "/Users/Kim/.pyenv/versions/3.8.2/bin/sqlfluff", line 11, in <module>
    load_entry_point('sqlfluff==0.3.4', 'console_scripts', 'sqlfluff')()
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/cli/commands.py", line 162, in lint
    result = lnt.lint_paths(paths, verbosity=verbose, ignore_non_existent_files=False)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/linter.py", line 890, in lint_paths
    result.add(self.lint_path(path, verbosity=verbosity, fix=fix,
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/linter.py", line 875, in lint_path
    self.lint_string(target_file.read(), fname=fname,
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/linter.py", line 692, in lint_string
    parsed, vs, time_dict = self.parse_string(s=s, fname=fname, verbosity=verbosity, config=config)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/linter.py", line 590, in parse_string
    s, templater_violations = self.templater.process(s, fname=fname, config=config or self.config)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/templaters.py", line 298, in process
    ctx.update(self._extract_macros_from_path(macros_path, env=env, ctx=live_context))
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/templaters.py", line 204, in _extract_macros_from_path
    macro_ctx.update(cls._extract_macros_from_path(
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/templaters.py", line 195, in _extract_macros_from_path
    cls._extract_macros_from_template(
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/sqlfluff/templaters.py", line 173, in _extract_macros_from_template
    for k in macro_template.module.__dict__:
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/environment.py", line 1189, in module
    return self._get_default_module()
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/environment.py", line 1172, in _get_default_module
    self._module = rv = self.make_module()
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/environment.py", line 1155, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/environment.py", line 1238, in __init__
    body_stream = list(template.root_render_func(context))
  File "<template>", line 24, in root
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/sandbox.py", line 460, in call
    if not __self.is_safe_callable(__obj):
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/sandbox.py", line 360, in is_safe_callable
    getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/runtime.py", line 753, in __getattr__
    return self._fail_with_undefined_error()
  File "/Users/Kim/.pyenv/versions/3.8.2/lib/python3.8/site-packages/jinja2/runtime.py", line 747, in _fail_with_undefined_error
    raise self._undefined_exception(self._undefined_message)
jinja2.exceptions.UndefinedError: 'ref' is undefined

I have also tried adding the default macros to my own config, but the result was the same

[sqlfluff:templater:jinja]
load_macros_from_path=dbt_modules

[sqlfluff:templater:jinja:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable) %}item{% endmacro %}

My questions

I think there may be a bug that causes the default macros to not be picked up.

And I am wondering whether it is possible to define multiple macro pathes. I think we need one for our custom macros path (./macros/) and one for the dbt_modules.

[alanmcruickshank]

Heya guys, thanks all for such detailed bug reports here. I’m aware that this is an issue.

Has anyone had this issue with any other plugins for dbt? Longer term I think sqlfluff needs a better way of rendering dbt queries that doesn’t involve just duplicating the dbt plugin logic within sqlfluff.

This feels like one of the biggest issues at the moment which is affecting the most people so I’ll try and find some time to work on this over the next few weeks. 👍

Has anyone delved into the code enough have an suggestions on how they would fix this best? Very open to suggestions. If not I’ll try a few things.

[NiallRees]

Hey @KimchaC thank you so so much for the testing on this! 🙇‍♂️

I’m going to close this original issue as it has been resolved by #508, but please raise any issues you’ve encountered separately, where they’ll be far more visible too.