Show unexpected values when accepted_values test fails

When an accepted_values test fails, print the unexpected values.

I am currently writing a test against a column with many possible values, where we want to know if any new values appear in the column that might affect our analytics.

When the accepted_values test fails, it shows a message like:

Got N results, expected 0.

However, it doesn’t display information about the unexpected results

Showing the unexpected results would make it easier to investigate and modify the test.

I can help contribute to this feature.

[jtcohen6]

I think this imagines us rewriting the default accepted_values test to be:

with all_values as (

    select
        {{ column_name }} as value_field,
        count(*) as count

    from {{ model }}
    group by 1

),

validation_errors as (

    select
        value_field,
        count

    from all_values
    where value_field not in (
        {% for value in values -%}
            {% if quote_values -%}
            '{{ value }}'
            {%- else -%}
            {{ value }}
            {%- endif -%}
            {%- if not loop.last -%},{%- endif %}
        {%- endfor %}
    )
)

select * from validation_errors

[jtcohen6]

@brylie What a cool idea! I think that the requisite pieces may just be falling into place, so that we could make this happen.

Back in #2593, I made a detailed proposal for how tests could store their (failing) results in database tables. To date, each schema test query has returned a single numeric value—count(*) or whatever else—so it would have been impossible to save and present an actual table of failing values. In order to make that happen, we’ve been working on creating a standard test “materialization” (#3181), and implementing it for ~schema~ generic tests as well (#3192). Those generic tests will now return a set of rows—whether that’s simply rows from the table where a column is null, or rows from an aggregate query that rolls up counts by unacceptable values.

All of which is to say, if we’re doing all that and can save those results in a database table, in addition to grabbing the total failure count/value, we could also print a few of those results to stdout using good old results.print_table():

Failure in test accepted_values__orders__order_status (models/resources.yml)
  Found 7 unacceptable values of orders.order_status, erroring because >0.
  
    | VALUE_FIELD |      COUNT |
    | ----------- | ---------- |
    | blue        | 17,452,443 |
    | red         |  6,105,277 |
    | green       |  1,111,547 |
  
    For full results:
    --------------------------------------------------------------------------------
    select * from dev_jcohen__dbt_test_results.accepted_values__orders__order_status
    --------------------------------------------------------------------------------

  compiled SQL at target/compiled/models/resources.yml/schema_test/not_null__orders__order_status.sql

I think we’d put in limitations like:

• Print top 5 rows, and up to 5 columns
• An optional property of generic test blocks that defines order_by (in the case above, order by 2 desc)

Let me know what you think about the above, and let’s see if we can sneak in this (or something like it) alongside the other test functionality coming in v0.20.0 😃