Enfore no new warnings in test suite

Our integration tests emit quite a few warnings. From skimming them, they are all caused by testing deprecated behavior. We should do two things to clean this up:

1. Add some functionality to capture deprecation warnings, but let all others pass through. I’m thinking of a decorator than can be slapped onto a test case or test.
2. Configure pytest to fail on warnings.

[pmeier]

Hey Kush, awesome to have you here! Go ahead, pleased to have you working on this.

What do we mean by: “capturing deprecation warnings but let all others pass through”?

Let me illustrate this with an example. Let’s say we have a function foo. This function calls into torch and in a newer release they emit a warning that for example the default arguments will be changed in the future. This warning is legitimate and should not be suppressed by the test suite. On the other hand if we decide to deprecate foo it will emit some warning (more details below) until the next major release.

def foo():
    torch.bar()  # emits a warning 
    warnings.warn(...)

Since we have tests for foo in our test suite, we will now also see the deprecation warning in the test log. If we enforce the warnings equal failures rule, the test will also fail.

My idea is the following: we create a decorator, e.g. @deprecated, that we can slap on every test and that captures our deprecation warnings, i.e. suppresses them, while legitimate warnings will be emitted and in turn fail the test. We have very basic functionality like this already in

https://github.com/pystiche/pystiche/blob/4324f34f74f53ef0c143b7f0d6eac99ae082636c/pystiche/misc.py#L283

In my current understand we would ignore warnings

1. if they are emitted from pystiche and
2. if they are either a DeprecationWarning or a Userwarning containing the word deprecate.

We will probably need to play with the warnings package and its filters. pytest also has a recwarn fixture that captures all warnings. Maybe we could use that to filter after the test was run. We are doing something similar in our gallery tests:

https://github.com/pystiche/pystiche/blob/4324f34f74f53ef0c143b7f0d6eac99ae082636c/tests/galleries/test_galleries.py#L163

Hopefully, my intention is clearer now. Let me know if you still have questions.

[pmeier]

Hey @krshrimali, do you have an update on this?