Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Use `md5_binary()` instead of `md5()` for `hash` function on Snowflake
See original GitHub issueDescribe the bug
Reportedly, Snowflake cannot properly micro-partition the UUID string generated by md5, and performs significantly better doing joins with md5_binary.
Steps to reproduce
Join a few large tables on surrogate keys in Snowflake.
Expected results
Performs something akin to a join on an integer id, or at the very least as well as a join across multiple columns (for instance an id and timestamp).
Actual results
Performs worse than joining on multiple columns, which give Snowflake information on partitioning.
System information
Which database are you using dbt with?
- snowflake
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:7 (3 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Why not make this optional via an argument? That way new implementation could use the binary key and older models wouldn’t break. The old string version could be deprecated at some point by switching the default to binary.
yep yep - i have some ideas around that – i want to test it on a big data set first and make sure it’s enough of an improvement to warrant it at all first though. open to any suggestions on that front if something good comes to mind.