Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use `Set` objects for dedupe

See original GitHub issue

Has there been any research done in improving the performance of dedupe by utilising a Set?

Sets are said to be faster^{[1] [2]} and are more idiomatic when dealing with lists of unique elements.

Issue Analytics

State:
Created 5 years ago
Comments:7

Top GitHub Comments

3reactions

newyork-anthonyngcommented, Sep 6, 2018

@dcousens I ran the JSPerf test for all available browsers I could get too.

Browser	Machine	Results
Chrome	Macbook Pro 2017	54% slower without using `Set`
Safari	Macbook Pro 2017	52% slower without using `Set`
Firefox	Macbook Pro 2017	54% slower without using `Set`
Mobile Chrome	iPhone 7	56% slower without using `Set`
Mobile Safari	iPhone 7	60% slower without using `Set`

3reactions

newyork-anthonyngcommented, Aug 24, 2018

I created a branch here that uses Set. I ran the benchmarks locally on my machine; see below.

* local#strings x 3,410,647 ops/sec ±0.82% (95 runs sampled)
*   npm#strings x 3,361,959 ops/sec ±0.31% (103 runs sampled)
* local/dedupe#strings x 768,729 ops/sec ±1.36% (90 runs sampled)
*   npm/dedupe#strings x 1,467,791 ops/sec ±0.23% (100 runs sampled)

> Fastest is local#strings

* local#object x 3,086,011 ops/sec ±0.22% (100 runs sampled)
*   npm#object x 3,078,249 ops/sec ±0.16% (102 runs sampled)
* local/dedupe#object x 1,084,332 ops/sec ±1.12% (99 runs sampled)
*   npm/dedupe#object x 2,091,594 ops/sec ±1.89% (98 runs sampled)

> Fastest is local#object

* local#strings, object x 2,756,612 ops/sec ±1.16% (99 runs sampled)
*   npm#strings, object x 2,641,684 ops/sec ±2.83% (96 runs sampled)
* local/dedupe#strings, object x 745,832 ops/sec ±3.45% (92 runs sampled)
*   npm/dedupe#strings, object x 1,335,123 ops/sec ±3.23% (96 runs sampled)

> Fastest is local#strings, object

* local#mix x 1,908,767 ops/sec ±3.32% (94 runs sampled)
*   npm#mix x 1,829,050 ops/sec ±3.97% (93 runs sampled)
* local/dedupe#mix x 494,316 ops/sec ±2.01% (98 runs sampled)
*   npm/dedupe#mix x 645,432 ops/sec ±0.56% (98 runs sampled)

> Fastest is local#mix

* local#arrays x 757,361 ops/sec ±1.58% (94 runs sampled)
*   npm#arrays x 815,488 ops/sec ±2.90% (98 runs sampled)
* local/dedupe#arrays x 384,666 ops/sec ±3.14% (95 runs sampled)
*   npm/dedupe#arrays x 717,931 ops/sec ±3.92% (96 runs sampled)

> Fastest is npm#arrays

It seems like using Set is slower in a node environment. However, I ran this test in JSPerf and it showed a 50% improvement in a browser environment.

Top Results From Across the Web

How to remove all duplicates from an array of objects?

Array.filter() removes all duplicate objects by checking if the previously mapped id-array includes the current id ( {id} destructs the object into ...

Removing duplicates in an Array of Objects in JS with Sets

Set will only allow unique values in it, so i'm going to pass it the ids of each object. If the loop tries...

Remove Duplicates from an Array of Objects in JavaScript

To remove all duplicates from an array of objects create an empty array that will store the unique object IDs. Call the `filter()`...

How to remove duplicates from an array of objects using ...

A new set is created by passing this array to the new set constructor. This step will remove all the duplicate elements as...

How to Remove Duplicates From an Array of Objects in ...

We can use the JavaScript array's filter method to remove duplicate entries from an array. ... The 2nd entry in arr is a...