load_macros_from_path should allow exceptions

Context

We are working with dbt in BigQuery and, as part of that, have created a new materialization strategy to handle some use cases that weren’t properly addressed by built-in dbt options. This means that we have a macro in our macros folder containing this line:

{% materialization incremental, adapter='bigquery' -%}

Issue

The materialization tag is not recognised, giving this error:

jinja2.exceptions.TemplateSyntaxError: Encountered unknown tag 'materialization'

Clearly, one solution would be to add a jinja extension that covers this tag, which I definitely would appreciate, but since there’s always a risk of this scenario cropping up it seems there should be an option to exclude certain paths from the load_macros_from_path statement in [sqlfluff:templater:jinja].

Alternatively, being able to specify multiple paths would be nice, since then we could add all the subfolders in our macros directory except for the one causing the error.

[pwildenhain]

I’m considering changing how we support the compiling of templates and actually getting dbt to do the rendering of queries, which would solve a load of these issues.

I’m 💯 in agreement with this. If people are using dbt anyway with this project, then I think we should just use dbt to compile the queries, and eliminate the need to double document macros

Could this be added as a config option? Should this be a separate plugin (like sqlfluff-dbt)?

[noahbruegmann]

I can confirm that we’ve been testing this on a few different projects now and it seems to be working well. Thanks!