Adding property-based tests for Astropy using Hypothesis

At SciPy 2019, I did a talk and tutorial on Hypothesis, and at the sprints I was encouraged to open an issue or PR showing how it could be applied to Astropy. So here it is!

(and for SciPy 2020 I wrote a paper on property testing for science with principles, examples, and advice)

Traditional unit tests involve checking that for some specific (usually simple) input, you get the expected output. With Hypothesis, your tests instead check that some property is true for any valid input - you describe the domain, and it generates examples. (See the docs for details, but we spend a lot of time making tests non-flaky too)

For example, the following test will demonstrate a variety of ways that floating-point imprecision can ruin your day when working with units. This is unlikely to happen in practice, but careful use of Fraction could prevent it entirely…

from hypothesis import given, strategies as st
import astropy.units as u
UNITS = sorted(u.get_current_unit_registry().all_units, key=repr)

@given(st.lists(st.sampled_from(UNITS), min_size=3))
def test_intermediate_decomposition_is_no_op(lst):
    decomp = compound = u.dimensionless_unscaled
    for unit in lst:
        decomp = (decomp * unit).decompose()
        compound *= unit
        assert decomp == compound.decompose(), (decomp, compound)

Other properties that were suggested:

• Check that round-tripping from image to sky coordinates and back is lossless for distortion-free mappings, and otherwise always below 10^-5 px.
• Take a moment in time, round-trip it through various frames, check it hasn’t changed or lost precision. See #9532 and #10373.
• Testing that IO routines losslessly round-trip data that they are expected to handle
• Check that any optimised routines calculate the same result as unoptimised, within tolerances
• Check broadcasting behaviour of custom ufuncs with mutually_broadcastable_shapes()

pytest-astropy pulls in Hypothesis automatically now, and Astropy has everything configured, so it’s just a matter of writing some more tests! While I’m not an astrophysicist, I remain willing to review and otherwise help out with any property-based testing PRs for open science projects - just ping me 😁

[pllim]

Might be a good sprint topic for the next Astropy coordination meeting. Will add to the suggestions. Thanks for the idea, @Zac-HD !

[astrofrog]

Just a thought - we might also want to provide strategies for other packages based on astropy, for example to generate random quantities, or random tables and so on.