`py38_images` tests failing after recent merge in Main
See original GitHub issue🐞 Problem
The Main branch & subsequently all rebased branches on the current main are failing the py38_images test. The problem seems to be that the images from test_porkchop_plotting in test_porkchop.py are different from those in the assertion … The can be seen in the code snippet below as well as this link
=================================== FAILURES ===================================
____________________________ test_porkchop_plotting ____________________________
Error: Image files did not match.
RMS Value: 23.55042462462195
Expected:
/tmp/tmpi6yv37y_/test_porkchop.test_porkchop_plotting/baseline.png
Actual:
/tmp/tmpi6yv37y_/test_porkchop.test_porkchop_plotting/result.png
Difference:
/tmp/tmpi6yv37y_/test_porkchop.test_porkchop_plotting/result-failed-diff.png
Tolerance:
2
💡 Possible solutions
We need to check for the reason why the test is failing in the first place & find the commit that was recently merged leading to failure of the required test. Appropriate changes need to be done if the test has wrong assertion, upload the correct image or revert the commit so as to restructure it with all the tests passing
Issue Analytics
- State:
- Created 2 years ago
- Comments:10 (10 by maintainers)
Top Results From Across the Web
If two individual branches pass unit tests, once they're merged ...
After the merge, the tests for function g will fail. ... So when branch A merges, the goalpost for the other branches move...
Read more >How to avoid broken master with Pipelines for Merged Results ...
With Pipelines for Merged Results, we can confidently say that MRs are continually tested against the latest master branch.
Read more >Git merge reports "Already up-to-date" though there is a ...
The message “Already up-to-date” means that all the changes from the branch you're trying to merge have already been merged to the branch...
Read more >Post-merge test automation failures - Katrina the Tester
Sometimes the second build will fail. 1. Failing tests after multiple PR merges to master. As the person who had driven sweeping test...
Read more >Defining the mergeability of pull requests - GitHub Docs
You can require pull requests to pass a set of checks before they can be merged. For example, you can block pull requests...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

These tests are just a way to check internal consistency. I propose that we regenerate the images with latest matplotlib so we can make the tests pass without changing anything else.
To regenerate the images: https://github.com/matplotlib/pytest-mpl/#with-baseline-images
Fair enough, but first we ought to understand what is the root cause. If we loosen up the tolerance, we might miss legitimate bugs.