question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add benchmark suite

See original GitHub issue

We should track method performance using a benchmark suite like @alimanfoo mentioned in https://github.com/pystatgen/sgkit/pull/36#issuecomment-658893949.

It would be ideal if this ran as a part of the build, but timings gathered that way are probably not very reliable. Apparently integration of ASV suites in builds is usually just to make sure they work: https://github.com/astropy/astropy/issues/6149. Presumably we should have a separate, scheduled process that runs on dedicated resources in order to benchmark accurately.

As @hammer mentions in https://github.com/related-sciences/gwas-analysis/issues/10, https://github.com/ornl-oxford/genben could also be a helpful resource (scikit-allel benchmarks).

Ideas for benchmarking:

Issue Analytics

  • State:open
  • Created 3 years ago
  • Reactions:1
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
hammercommented, May 6, 2021

Conbench from Ursa Labs could be useful for us: https://ursalabs.org/blog/announcing-conbench. cc @arunkk09 and @LiangdeLI

1reaction
ravwojdylacommented, Aug 13, 2020

Just wanna drop ann-benchmarks as an inspiration. From readme:

Doing fast searching of nearest neighbors in high dimensional spaces is an increasingly important problem, but so far there has not been a lot of empirical attempts at comparing approaches in an objective way. This project contains some tools to benchmark various implementations of approximate nearest neighbor (ANN) search for different metrics. We have pregenerated datasets (in HDF5) formats and we also have Docker containers for each algorithm. There’s a test suite that makes sure every algorithm works.

Read more comments on GitHub >

github_iconTop Results From Across the Web

benchmark.Suite.add JavaScript and Node.js code examples
Best JavaScript code snippets using benchmark.Suite.add(Showing top 15 results out of 351) · closure.js/add.add.on. · object_comparison.js/add.on. · test/benchmark ...
Read more >
goblint/bench: The benchmark suite - GitHub
Goblint benchmark suite. This contains the suite of benchmarks we currently use to evaluate Goblint. To run the benchmarks, make sure this repository...
Read more >
benchmark-suite - npm
A simple suite of benchmarking tests. Latest version: 0.1.8, last published: 5 months ago. Start using benchmark-suite in your project by ...
Read more >
HEP Benchmark Suite - CERN GitLab
Automates the programmatic execution of a number of benchmarks (HS06, SPEC CPU 2017, HEP-score, etc) Formerly known as cloud-benchmark-suite.
Read more >
UL Procyon Professional Benchmark Suite - UL Solutions
Flexible licensing lets you pick and choose the individual benchmarks that best meet your needs. Buy just one benchmark or add more in...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found