Have compare_columns support BigQuery Arrays for equality test

Describe the feature

We would like to use the db_utils compare_columns to be able to check BigQuery clolumns with arrays in tests. At the moment, it does not appear to be supported.

Describe alternatives you’ve considered

Alternatives are to not use dbt to test for equality of arrays, and instead to test it via API call and comparing the results in Python, which requires extra work.

Additional context

BigQuery arrays are our use case, but I imagine this problem for any db with arrays and the solution might be generalizable to others.

Who will this benefit?

Any who writes tests and has databases with arrays.

Are you interested in contributing this feature?

Sure, I’d love to take a crack at this if someone points me in the right direction. I see the code here https://github.com/dbt-labs/dbt-utils/blob/37199d3901a97e813899dc25c1f72601714c2c3c/macros/generic_tests/equality.sql. It looks pretty straightforward, just have to support either user supplying a flag or maybe the datatype of the column being an array is inferred. Once there, then it’s a matter of just flattening the column/table (in BigQuery, with UNNEST, other dialects, not sure, hence, maybe this is a BQ specific solution?)

[jasonqng]

@joellabes Thanks for the pointers. I’ll take a crack at it later this week. I’ll probably need your help with tests and/or other polish, but hopefully I can at least get a prototype PR working for you all to look at.

[jasonqng]

@pishposh Haven’t had a chance to do so, and yeah, handling structs would be amazing too. Go for it! Look forward to helping to test and review if helpful!