question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Using dask to save frames in parallel

See original GitHub issue

I am opening a separate issue for this functionality (previously mentioned in #2).

Currently the most time intensive step in creating a movie/gif is the rendering of each frame, which is done in serial.

In an earlier test version I implemented a parallelization of the printing using dask. The speedup was very nice, but I encountered quite strange errors with some matplotlib elements: E.g. the colorbar label would shift around and other odd things that looked less than premium.

I would like to try to implement this again with the substantially refactored code. The proposed changes would replace the default save_frames with save_frames_parallel if a dask array is passed, so that mov.save(moviename.mp4) would work in parallel out of the box and serial rendering could be forced with something like mov.save(moviename.mp4, render_parallel=False).

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:2
  • Comments:23 (22 by maintainers)

github_iconTop GitHub Comments

1reaction
dcheriancommented, Oct 17, 2020

Map_blocks works really well

0reactions
jbuseckecommented, Nov 30, 2020

Its here: https://github.com/jbusecke/xmovie/tree/jbusecke_dasksave, but use at your own risk! It is quite old at this point!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Dask DataFrames — How to Run Pandas in Parallel With Ease
Yes — Dask Data Frames. Most of Dask API is identical to Pandas, but Dask can run in parallel on all CPU cores....
Read more >
Dask - How to handle large dataframes in python using ...
A Dask DataFrame is a large parallel DataFrame composed of many smaller Pandas DataFrames, split along the index. One Dask DataFrame is ...
Read more >
Dask Best Practices - Dask documentation
This is a short overview of Dask best practices. This document specifically focuses on best practices that are shared among all of the...
Read more >
Parallel computing in Python using Dask - Topcoder
dask.dataframe - Distributes data frames like Pandas. It is a large parallel data frame made of many smaller data frames. Installation.
Read more >
Dask — To handle large data frames using parallel computing
That means you can now use Dask to not only speed up computations on datasets using parallel processing, but also build ML models...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found