Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ci: Performance test failing randomly

See original GitHub issue

Meltano Version

2.5.0

Python Version

3.10

Bug scope

Other

Operating System

Windows

Description

The test in tests/meltano/cli/test_cli.py::TestLargeConfigProject::test_list_config_performance is failing for combinations of Python and OS:

source: https://github.com/meltano/meltano/runs/8078349477?check_suite_focus=true#step:11:687

Code

No response

Issue Analytics

State:
Created a year ago
Comments:6 (2 by maintainers)

Top GitHub Comments

2reactions

edgarrmondragoncommented, Sep 1, 2022

@aaronsteers Simple solution is to bump the tolerance.

Long term I wonder if this is the best way to run benchmarks. I looked around a bit during research for https://github.com/meltano/meltano/issues/6613, https://github.com/meltano/sdk/pull/887 and https://github.com/meltano/sdk/discussions/906.

It seems like we need something like https://github.com/marketplace/actions/continuous-benchmark that simply alerts us when time exceeds a certain threshold, and lets us keep a history of performance in key components.

1reaction

WillDaSilvacommented, Sep 2, 2022

The only case I can see where a retry would be the best/right call would be when root cause is an external service failure, and when we can’t reasonably mock the external service.

That’s essentially what this is. The Windows VMs occasionally perform worse, and we can’t control that or mock them performing better.

But I agree that resolving the immediate issue takes precedence, and increasing the timeout is the fastest/simplest way to resolve that. I’ll open a PR.