Treat blank strings differently to nulls in surrogate_key()

Describe the bug

If I had a table like so:

field1 (STRING)	field2
NULL	1
‘’	1

running dbt_utils.surrogate_key(['field1', 'field2']) would produce the same value for both rows

This is because of the coalesce() that happens on each value during concatenation see here

DISTINCT and GROUP BY operations treat null different from ‘’, so should we do that as well when making a unique key? Moreover, what would be the proper way to implement this? You likely cannot use another string as a universal substitute and adding a randomly-generated int or string will break idempotency for surrogate keys.

System information

  - package: dbt-labs/dbt_utils
    version: 0.7.6

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

0.20.2

[joellabes]

Yes @callum-mcdata i think that works well.

Thanks everyone for weighing in, especially as I don’t know that I did a good job of explaining the second option.

Another attempt:

• The current surrogate_key macro will be fully deprecated and only throw an error message.
• The new macro, generate_surrogate_key, will default to treating nulls and blanks differently.
• Users can opt into the old setting at a project-wide level with a variable.

By doing this,

• the default path works correctly for all future users without ant additional config, which is the purpose of the v1.0 migration.
• We also avoid accidentally hosing people’s primary keys that they use in snapshots/incremental models, which a) causes data integrity issues and b) could cause a lot of additional spend on reprocessing data.

The cost of this approach is that anyone using the Cloud IDE who doesn’t have someone to do a local clone and update will be unable to upgrade to utils 1.0 without doing something like that GitHub action I linked above. If someone makes the action, we can link it in the changelog at some point.

[dbeatty10]

Resolved by #685