question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`py38_images` tests failing after recent merge in Main

See original GitHub issue

🐞 Problem

The Main branch & subsequently all rebased branches on the current main are failing the py38_images test. The problem seems to be that the images from test_porkchop_plotting in test_porkchop.py are different from those in the assertion … The can be seen in the code snippet below as well as this link

=================================== FAILURES ===================================
____________________________ test_porkchop_plotting ____________________________
Error: Image files did not match.
  RMS Value: 23.55042462462195
  Expected:  
    /tmp/tmpi6yv37y_/test_porkchop.test_porkchop_plotting/baseline.png
  Actual:    
    /tmp/tmpi6yv37y_/test_porkchop.test_porkchop_plotting/result.png
  Difference:
    /tmp/tmpi6yv37y_/test_porkchop.test_porkchop_plotting/result-failed-diff.png
  Tolerance: 
    2

💡 Possible solutions

We need to check for the reason why the test is failing in the first place & find the commit that was recently merged leading to failure of the required test. Appropriate changes need to be done if the test has wrong assertion, upload the correct image or revert the commit so as to restructure it with all the tests passing

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (10 by maintainers)

github_iconTop GitHub Comments

2reactions
astrojuanlucommented, Mar 30, 2021

These tests are just a way to check internal consistency. I propose that we regenerate the images with latest matplotlib so we can make the tests pass without changing anything else.

To regenerate the images: https://github.com/matplotlib/pytest-mpl/#with-baseline-images

2reactions
astrojuanlucommented, Mar 30, 2021

Fair enough, but first we ought to understand what is the root cause. If we loosen up the tolerance, we might miss legitimate bugs.

Read more comments on GitHub >

github_iconTop Results From Across the Web

If two individual branches pass unit tests, once they're merged ...
After the merge, the tests for function g will fail. ... So when branch A merges, the goalpost for the other branches move...
Read more >
How to avoid broken master with Pipelines for Merged Results ...
With Pipelines for Merged Results, we can confidently say that MRs are continually tested against the latest master branch.
Read more >
Git merge reports "Already up-to-date" though there is a ...
The message “Already up-to-date” means that all the changes from the branch you're trying to merge have already been merged to the branch...
Read more >
Post-merge test automation failures - Katrina the Tester
Sometimes the second build will fail. 1. Failing tests after multiple PR merges to master. As the person who had driven sweeping test...
Read more >
Defining the mergeability of pull requests - GitHub Docs
You can require pull requests to pass a set of checks before they can be merged. For example, you can block pull requests...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found