Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Benchmark regressions on master are not being tagged correctly

See original GitHub issue

Trying to build master at commit 98141a02e13b219ee80463ab6b5c94516681304d failed. More info at: https://travis-ci.com/Qiskit/qiskit-terra/jobs/225005209

Issue Analytics

State:
Created 4 years ago
Comments:7 (7 by maintainers)

Top GitHub Comments

1reaction

mtreinishcommented, Aug 19, 2019

OK it’s merged, now we just need to wait for the next asv release and we can update the CI jobs to run asv continuous with the --strict flag and it’ll have a return code of 2 if the benchmarks fail.

1reaction

mtreinishcommented, Aug 15, 2019

Good point. I had seen that asv wouldn’t raise if benchmarks failed on both on the base and branch but was hoping if there was a change in status (base ran but branch failed), asv might raise. Benchmarks aren’t unit tests, but if a commit breaks a benchmark, it be helpful to get a heads up. Maybe worth an issue in asv?

It looks like there is already an open issue for it: https://github.com/airspeed-velocity/asv/issues/449

I might give it a try if I get some spare time, it shouldn’t be super hard to fix.

(I thought the state tomography benchmarks had been out of service for awhile? Last commit in asv is 8609f7a .)

Well we haven’t done a backfill run Qiskit/qiskit#400 has merged the other day, but doesn’t work for any commits after #2414. I was planning to backfill the benchmark data for state tomography this weekend.