Backwards incompatibility in fits.open

Description

The check for a missing SIMPLE card included in astropy v4.3 breaks backwards compatibility in fits.open. Archival data that contains the SIMPLE keyword, but in an unexpected format, produces an error when opened with default parameters in fits.open. Prior to v4.3, these files were opened and handled normally.

The associated pull request is: https://github.com/astropy/astropy/pull/10895.

This break in backwards compatibility will require us to change every fits.open command we have that might deal with non-standard archival data. This is hundreds of lines of code, across many separate packages. The data cannot easily be changed.

Expected behavior

fits.open opens files that can be read as FITS without throwing an error by default.

Actual behavior

The error “No SIMPLE card found, this file does not appear to be a valid FITS file” is produced and the file is not opened as an HDUList. The SIMPLE card is present and readable, though, and setting ignore_missing_simple=True allows the file to be opened and handled normally.

Steps to Reproduce

This issues an error with a minimal FITS file containing: ‘SIMPLE = T / does conform to FITS standard’

from astropy.io import fits
hdul = fits.open('test.fits')

This succeeds:

from astropy.io import fits
hdul = fits.open('test.fits', ignore_missing_simple=True)

System Details

macOS-10.15.7-x86_64-i386-64bit Python 3.8.2 (default, Mar 26 2020, 10:43:30) [Clang 4.0.1 (tags/RELEASE_401/final)] Numpy 1.20.3 astropy 4.3.1 Scipy 1.7.1 Matplotlib 3.4.2

[kYwzor]

This occurs because io.fits expects this exact sequence of characters to be within the first 30 bytes of the file:

SIMPLE  =                    T

This check obviously fails if the file’s spacing is non-standard. As I mentioned in the pull request, I propose replacing: https://github.com/astropy/astropy/blob/c6cafcb763c9a2cf6cbbc1989d7fbb6dbdb6932f/astropy/io/fits/hdu/hdulist.py#L1084-L1085 with: match_sig = re.match(rb"SIMPLE *= *[T|F]", simple)

This would still allow io.fits to terminate early when no SIMPLE card is present at the beginning of the file, while being tolerant of non-standard spacing.

[maxnoe]

This is also not an addition to recent standards, it goes back all the way to FITS 1.0 from 1993.