question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Implement parallel writes

See original GitHub issue

This is a big feature that would allow Etcher to be able to stream the same image to multiple drives at the same time. This feature should be implemented by extending the already existing etcher-image-write NPM module. The module currently accepts the path to a drive device as an argument to the main function. If the user passes an array of drives, the multi-writer logic should come in.

Some of the challenges for this task are:

  • Cross platform support.
  • Validating the drives at the moment the writing starts.
  • Handling validation correctly for all of them (there are some quirks to make validation work in certain operating systems, like in OS X, where we have to use OS X specific frameworks to prevent the drives from being mounted right when the flash ends, so the operating system doesn’t write files like .DS_Store that messes up the checksum).
  • Handling error conditions correctly (e.g: if a certain drive fails, the rest should continue).
  • Making sure the whole stream is streamed to all devices (e.g: no data is lost because it has been piped to another drive).
  • Imposing a reasonable limit of drives that can be written at the same time, depending on the capabilities of the operating system to prevent potential kernel panics, etc. If a user tries to write to more images than he should, we should make of use of a queue.

The module returns an EventEmitter instance which emits state information. When multiple writings are taking place, the emitted events could have a property containing the drive they correspond to.

It’s important that this feature contains extensive end to end tests.

This project should not contain native addons by any means, since it’s used by other projects that are installed by end users without necessarily having a toolchain in place.

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Reactions:6
  • Comments:10 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
jhermsmeiercommented, Mar 23, 2018

Resolved by #2124

0reactions
sonttrancommented, Jul 7, 2020

Same issue, same setup with @unphased on Etcher Version 1.5.100. Any way to improve speed to full in parrallel, @jviotti ? That would be great.

I’ve flashed about 20 cards now and I have noticed that the time taken more or less scales linearly with the transfer speed of one card. e.g. if I have 3 cards that are independently capable of flashing @21MB/s, flashing 3 of them simultaneously has them proceed at almost exactly 7MB/s, making it so that I have to wait 3 times as long for the batch. Now, this is a process improvement and already a big win, however I am confused as to why there is no speedup as you would see if you just ran concurrent dds on multiple drives. The reason I use etcher is so I can avoid building a dd runner and figuring out how to properly verify a disk, but this seems… not ideal.

This is using a MacBook with multiple usb-c to sd card adapters.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Parallel writes of a same value - c++ - Stack Overflow
Implementation detail answer: While the language standard classifies this as undefined behavior, you can actually feel quite safe as ...
Read more >
Writing Performant Parallel Python Code - Medium
To answer this question, we first need to discuss how the Python interpreter works. In the course of the story, when detailing how...
Read more >
Write a simple Parallel.ForEach loop - Microsoft Learn
This article shows how to use a Parallel.ForEach loop to enable data parallelism over any System.Collections.IEnumerable or System.Collections.
Read more >
Parallel read and write in SQLite - Skoumal
Unfortunately, simultaneous writing from more than one thread is not allowed. The simplest way to read in parallel with writing transaction is to...
Read more >
How Go Implements Parallel and Concurrent Programming
Like what you see? Check out the Programming for Lovers course project for more: http://compeau.cbd.cmu.edu/programming-for-lovers/We will ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found