Preserve column order in union macro

Describe the feature

I’d like to suggest a new feature for the union_relations macro: is it possible to preserve the order of the columns if either of the models does not have a column instead of sending it to the end of the table? Currently, if either of the table used as a source in the union macro does not have a column that appears in the other table, the column is sent to the right of the table instead of appearing as it does in the source table.

Ex:

table 1

id	name	email

table 2

id	email

unioned table (current display)

id	name	email

instead of (ideal display)

id	email	name

Describe alternatives you’ve considered

N/A-would defeat the purpose of using this macro

Who will this benefit?

Analysts building dbt models, including myself – the order of columns often matters (columns with similar topic are often grouped together)

Are you interested in contributing this feature?

Yes, happy to!

[chloe-lubin]

Hi @joellabes , thanks so much for sharing! I just tried it with the 2 models I’m looking to union and confirming that the columns appear in order after reversing the model references. Would it be possible to make a note in the documentation?

[joellabes]

Thanks for this @chloe-lubin! I just did some experimentation, and want to share a bit of context for how it currently works.

The macro iterates over each ref that’s passed into the macro, in order. I made two basic models:

--model1.sql
select '1' as col1, '2' as col2, '3' as col3, '4' as col4, '100' as col100

--model2.sql
select 'd' as col4, 'a' as col1, 'b' as col2, 'z' as col20

When passing them into the macro, model1 then model2, I get this result:

When doing model2 then model1, I get this:

Note that in the second example, col4 comes first, and the columns that don’t appear in model1 (col3 and col100) appear last.

Related: https://github.com/dbt-labs/dbt-utils/issues/395 proposes upcasting columns when they are of incompatible types; I have cast my numbers as strings for the sake of this demo.

So for your example, you could get the desired outcome by doing {{ dbt_utils.union_relations([ref('table2'), ref('table1')]) }} instead of table1 then table2.

From what I’ve seen so far, I’m pretty comfortable with the current behaviour, because users can get the behaviour they want pretty easily. If your example was a cut down to a minimum version to demonstrate the point, it’d be useful for you to share the actual case you’re trying to achieve (e.g. many more columns, or multiple relations being unioned together).

Let me know what you think!