no capability to add ORDER BY table models

a user in dbt Slack asks:

Hi all, is there a way to specify “sort” column(s) or order by for models created in SQL Server? When I put in an order by clause I get the following error:

[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
The ORDER BY clause is invalid in
views, inline functions, derived tables, subqueries, and common table expressions,
unless TOP, OFFSET or FOR XML is also specified.

Because we’re using the SELECT * INTO DML for our model generation, if you include an ORDER BY in the SELECT query of a model, you’ll get the above error. But I’m confused because the docs say it should be allowed, but isn’t guaranteed…

When used with a SELECT…INTO or INSERT…SELECT statement to insert rows from another source, the ORDER BY clause does not guarantee the rows are inserted in the specified order.

https://github.com/dbt-msft/dbt-sqlserver/blob/679b3515e9f056879c6609c78bb62791716c450a/dbt/include/sqlserver/macros/adapters.sql#L160-L161

[jeroen-mostert]

There is only one scenario where ORDER BY in an INSERT makes sense, and that’s for guaranteeing that IDENTITY columns are assigned values in order. In all other cases this does not do what you might want it to do – in particular, every time you SELECT from a table you must specify an ORDER BY if you want an order, otherwise the engine is free to rearrange things as it sees fit (and it will, if the table is a columnstore, or parallelism is involved, or it can reuse a scan in progress from another query), so whatever order you think you may have gotten from the insert will not be useful anyway (and that’s before we consider rows being moved through deletes and updates). People often fall into the trap of thinking there’s a “natural physical order” that you’re guaranteed to get if you do SELECT * with no ORDER BY, but there isn’t.

So this is probably not useful to enable unless there really is a use case of building models with IDENTITY columns (unlikely, since ROW_NUMBER() gives you predictable values as well), and even then, as Mikael notes, you’d need to not use a view. (In olden days SELECT TOP(100) PERCENT ... ORDER BY could be used as a trick to circumvent this restriction, but while that’s still allowed syntactically, the optimizer simply pretends you didn’t write an ORDER BY in that case.)

As a minor quality of life improvement, you could consider allowing ORDER BY in the SELECT so people have more convenient access to the results as they’re developing the model, but then strip out the ORDER BY when it’s used for an INSERT since it won’t do good anyway. But as this requires parsing the statement and is easy to get wrong, I’m not sure that’s worth the effort.

[mikaelene]

Closing this, as it is not really feasible, and not desirable.