Update HighLevelGraph documentation
See original GitHub issueI just came across this page of the docs which hasn’t been changed much since Dec 2018. I think it would be useful to update it to include a runnable example and some information about the Layer class.
Once that page is updated, it will probably make sense to update https://docs.dask.org/en/latest/phases-of-computation.html and https://docs.dask.org/en/latest/graphs.html to make it clearer how HLGs fit in.
I am happy to take on this work, but just wanted to put this here in case I am missing something.
Some resources that I am planning to draw on: Layer Issue Blog post about scheduling Scheduling workshop from summit
Pinging the people who I have seen working on HLG related things: @rjzamora @GenevieveBuckley @gjoseph92 @ian-r-rose @jrbourbeau
Issue Analytics
- State:
 - Created 2 years ago
 - Reactions:3
 - Comments:24 (22 by maintainers)
 
Top Results From Across the Web
High Level Graphs update - Dask Working Notes
Brief background; Blockwise layers progress; A high level graph for map ... The documentation page on Dask high level graphs is here: ...
Read more >Internal Design — Dask 2.23.0 documentation
__dask_graph__() <dask.highlevelgraph. ... In short, dask.array operations update Dask graphs, update dtypes, and track chunk shapes.
Read more >Dask Distributed produces AttributeError: 'HighLevelGraph ...
However, I'm not sure if you mean compatibility for a YARN cluster. If so, please refer to this resource from Dask docs. In...
Read more >GLog: A high level graph analysis system using MapReduce
With the rapid growth of graphs in different applications, it is inevitable to leverage existing distributed data processing frameworks in managing large ...
Read more >Source code for nvtabular.ops.categorify
[docs]class Categorify(StatOperator): """ Most of the data set will contain ... continuous-column stats with concat_groups=True") # Update tree_width tw ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

For what it’s worth, I also recently wrote up some notes on HLG pain points. It’s less detailed than Rick’s but perhaps still interesting
@martindurant @douglasdavis - The following gist is still very rough for now, but may be helpful for to-be layer developers: https://gist.github.com/rjzamora/3f76c6dcbde16a4185222eaadcd1d02f