L001 fix error: `NotImplementedError: Unexpected scenario during insert opcode!`

Query which generates the error:

{% set party_types = (
    'a', 'b', 'c'
) %}

select
    parties.id
    {% for party in party_types %}
        , case when sum(case when very_long_name_of_thing.party_type = '{{party}}' then 1 else 0 end) > 0 then false else true end as {{party_type}}_blah_blah_blah
    {% endfor %}
from parties

L:   5 | P:   1 | L001 | Unneccessary trailing whitespace.
L:   7 | P:   1 | L001 | Unneccessary trailing whitespace.
L:   9 | P:   1 | L001 | Unneccessary trailing whitespace.
L:  11 | P:   1 | L001 | Unneccessary trailing whitespace.

  File "/Users/.../venv/lib/python3.7/site-packages/sqlfluff/linter.py", line 302, in fix_string
    ("Unexpected scenario during insert opcode! Please report "
NotImplementedError: Unexpected scenario during insert opcode! Please report this issue on github with the query and rules you're trying to fix.

I think the issue here isn’t actually with the linter, but that the rule is being incorrectly triggered as there isn’t any trailing whitespace in the query.

[dmateusp]

The templating problem is an interesting one! My thoughts on this is that the solution should revolve around tagging templated segments.

It would allow us to:

• Return line numbers pre-templating and post-templating for issues that arise
• Know if errors were raised in templated segments and (optionally) ignore them, in the same vein as what @pwildenhain said. Like @NiallRees I don’t think removing templating segments make sense as it limits what we can validate.
• Potentially re-use the partially parsed file pre-templating instead of re-parsing the file again after templating (as re-parsing can be slow)

@NiallRees raised many great issues to implement DBT best practices through sqlfluff that are only possible if we read and understand the code pre-templating. (For example {{ ref("...") }} should be inside a CTE at the top of the file.)

I think it should look more like this:

1. Parse pre-templating and tag segments with templating code as “templating code”
2. Apply rules that are tagged to run on pre-templated code (like most of the DBT best practices, I imagine)
3. Expand the segments that are templated and parse these specific segments
4. Apply rules tagged to run on post-templating code (default)

Thanks for the ideas both.

I think pre-processing before we template could work, but we still produce several lint errors for potentially only one in this case which I’d really like to avoid. Thinking along the lines of as a human how I review a pull request I wonder if we could just lint the pre-templated raw code on a ‘best-effort’ basis, and then template and process for execution correctness.

In this world I imagine SQLFluff would run something along the lines of:

1. Lex and parse with jinja still present - so the raw file as suggested by @pwildenhain (let me know if this isn't what you mean!)

2. Apply linting rules for visual correctness only on a best-effort basis with regards to the jinja (like how we as humans review code) and return the result to the user

3. Apply templating

4. Lex and parse again

5. Verify SQL is valid for execution and return the result the user

I think we can actually avoid having to build another parser here and just use the same one for 1 and 4, or potentially subclass the SQL only parser to add an understanding of jinja.

I realise this is a fundamental redesign but the advantages are:

1. We can map lint errors 1-to-1 to the file line number in cases like the one in the issue

2. It makes it easy to address #464 and any other rules which apply to jinja

3. We can still ensure SQL correctness from an execution point of view (within the bounds of knowing nothing about the underlying data)

Disadvantages:

1. We end up writing our own code to parse jinja we want to understand (I think we'd have to do this to pre-process the templated code too though)

2. More complexity and therefore slowness

What do you think @alanmcruickshank?

– edit I’ve rewritten this comment about 3 times now the more I think about it, but hopefully in a positive direction!

[NiallRees]

This is the kind of problem where linting post templating becomes difficult, because as a user you’d never see the templated code and so don’t care what it looks like, but sqlfluff does. The goal is to make the pre-templated code readable, which in this case I think it is. I imagine this won’t be the only case where this crops up