not_blank test

Describe the feature

A clear and concise description of what you want to happen. dbt has the not_null test to monitor for null values, but in certain datasets blank values are also problematic. However, there is no not_blank test that I found when searching utils or dbt-expectations.

Describe alternatives you’ve considered

A clear and concise description of any alternative solutions or features you’ve considered. Creating a custom test that errors if there are blanks.

Additional context

Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here. I think blank entries are a possibility on all databases

Who will this benefit?

What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly. Anyone needing to check data integrity to ensure there are no blank values in their dataset

Are you interested in contributing this feature?

Yes plz 🙏🏻

[joellabes]

@tigitz thanks for the feedback and the links to prior art!

Which raise the question on how the rule should behave when encountering a blank array for database that support them.

I would be surprised if we could validate both strings and arrays in the same test without knowing their type in advance (which I think is your point in recommending _string and _array variants?)

My suggestion would be to be explicit in the rules name and avoid grouping different kind of tests into a single one using parameter flags. That would cause confusion.

Can you say more about the confusion element? I’m pretty sure I disagree with you on this one, at least on tests of the same column type (e.g. empty string vs blank string). I consider it to be analogous to the inclusive param on accepted_range, and think that it’d be tidier/less cognitive overhead to have a single test whose behaviour you tweak as opposed to having two almost identical tests with suspiciously similar names.

(Accidental proving of my point: I wrote a lot of this response with blank and empty the wrong way around, and only discovered my mistake when I scrolled back up to your original comment before posting).

What shall we name the ~baby~ test?

A meme that gets in the way of my point

I would prefer that we go with not_empty_string, which conforms to the Java definition of empty (apart from the null stuff) and makes semantic sense at a glance. Then I’d have an optional parameter on it, trim_whitespace ¹ which is false by default.

¹ In an earlier version, I suggested ignore_whitespace. Coming to this with fresh eyes, that’s unclear - what’s doing the ignoring and how does ignore manifest?

Arrays

I could go either way on building a test to check whether arrays are empty or not - if either you or @epapineau wants to build one then I wouldn’t be against it, but I’m also happy to punt for a while.

[epapineau]

Thanks for all the incredible thought poured into this @joellabes and @tigitz. A quick first pass at the not_empty_string(trim_whitespace) version of the test is here.

Is there a preference on using length = 0 over column_name = ''?

I can add an array counterpart + tests to round out the PR in the next day or two 😃