question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Update HighLevelGraph documentation

See original GitHub issue

I just came across this page of the docs which hasn’t been changed much since Dec 2018. I think it would be useful to update it to include a runnable example and some information about the Layer class.

Once that page is updated, it will probably make sense to update https://docs.dask.org/en/latest/phases-of-computation.html and https://docs.dask.org/en/latest/graphs.html to make it clearer how HLGs fit in.

I am happy to take on this work, but just wanted to put this here in case I am missing something.

Some resources that I am planning to draw on: Layer Issue Blog post about scheduling Scheduling workshop from summit

Pinging the people who I have seen working on HLG related things: @rjzamora @GenevieveBuckley @gjoseph92 @ian-r-rose @jrbourbeau

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:3
  • Comments:24 (22 by maintainers)

github_iconTop GitHub Comments

2reactions
ian-r-rosecommented, Dec 9, 2021

For what it’s worth, I also recently wrote up some notes on HLG pain points. It’s less detailed than Rick’s but perhaps still interesting

1reaction
rjzamoracommented, Dec 9, 2021

@martindurant @douglasdavis - The following gist is still very rough for now, but may be helpful for to-be layer developers: https://gist.github.com/rjzamora/3f76c6dcbde16a4185222eaadcd1d02f

Read more comments on GitHub >

github_iconTop Results From Across the Web

High Level Graphs update - Dask Working Notes
Brief background; Blockwise layers progress; A high level graph for map ... The documentation page on Dask high level graphs is here: ...
Read more >
Internal Design — Dask 2.23.0 documentation
__dask_graph__() <dask.highlevelgraph. ... In short, dask.array operations update Dask graphs, update dtypes, and track chunk shapes.
Read more >
Dask Distributed produces AttributeError: 'HighLevelGraph ...
However, I'm not sure if you mean compatibility for a YARN cluster. If so, please refer to this resource from Dask docs. In...
Read more >
GLog: A high level graph analysis system using MapReduce
With the rapid growth of graphs in different applications, it is inevitable to leverage existing distributed data processing frameworks in managing large ...
Read more >
Source code for nvtabular.ops.categorify
[docs]class Categorify(StatOperator): """ Most of the data set will contain ... continuous-column stats with concat_groups=True") # Update tree_width tw ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found