Run Pytest tests in parallel

https://meltano.slack.com/archives/CKHP6G5V4/p1653508630941889

Looks like pytest doesn’t take 45 minutes, not sure why I though it was taking that long before? Maybe on gitlab CI updated the title to reflect that it takes ~5-10 minutes to run pytest today.

1. Be sure https://github.com/meltano/meltano/pull/6155/files#diff-65c008e1065903299dfa0da25e1f18d0f87f387867ccbbf2b0888eb138bdb255L242 is implemented (This uses a seperate folder for each Test in your temp directory instead of the same folder over and over again)
2. poetry add -D pytest-xdist==1.*
3. poetry run pytest -n 20
4. Watch tests run extremely quickly

There’s still a large number of failures here as some tests depend on the other I believe? Not exactly sure what causes the failures, but it’s clear they weren’t all designed with parallel testing in mind. The gain is big enough it may be worth the time to figure out what causes the failures.

[WillDaSilva]

Good timing on the public beta for large GitHub runners: https://github.blog/changelog/2022-09-01-github-actions-larger-runners-are-now-in-public-beta/

I’ve added Meltano to the waitlist to be permitted to use them.

Seems like the larger runners also come with more memory, which we’ll need for running many tests in parallel. Particularly for the MSSQL tests, which create a very heavy Java process for each Pytest worker. The memory conveniently scales at the same rate as the processor cores.

[WillDaSilva]

We may want to upgrade our CI runners. They currently have 2 cores, but with more our CI pipelines would be far faster once we’re running the Python tests in parallel. GitHub charges linearly per-core for their CI runners, so as an example, if we want to use runners with 16 cores, it would cost us 8 times as much to execute the Python tests per CI minute, but my guess is that the tests would probably execute ~6 times faster. Seems like a worthwhile tradeoff.

https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners