FITS file corruption via fits.open(mode='update')

Hello,

I typically use mode='update' for modifying FITS data and headers with great success, but I have found a file where this fails. After opening for an update and adding 7 new keywords to the header, the updated file is double the original size and the data is corrupted:

from astropy.io import fits

with fits.open('update-corruption.fits', mode='readonly') as hdu:
    print('Image mean = ', hdu[0].data.mean())

print('Add keywords')
with fits.open('update-corruption.fits', mode='update') as hdu:
    for i in range(7):
        hdu[0].header['kw{:02d}'.format(i)] = 'foo'

with fits.open('update-corruption.fits', mode='readonly') as hdu:
    print('Image mean = ', hdu[0].data.mean())

Saving the above as corrupt.py and executing:

$ cp original-data.fits update-corruption.fits
$ ls -l original-data.fits update-corruption.fits 
-rw-rw-r-- 1 msk msk 75542400 Jan 30 07:42 original-data.fits
-rw-rw-r-- 1 msk msk 75542400 Jan 30 07:54 update-corruption.fits
$ python3 corrupt.py
Image mean =  123.848496001
Add keywords
Verify data
WARNING: VerifyWarning: Error validating header for HDU #1 (note: Astropy uses zero-based indexing).
    'ascii' codec can't decode byte 0x80 in position 2: ordinal not in range(128)
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
Image mean =  0.0
$ ls -l original-data.fits update-corruption.fits 
-rw-rw-r-- 1 msk msk  75542400 Jan 30 07:42 original-data.fits
-rw-rw-r-- 1 msk msk 151067520 Jan 30 07:57 update-corruption.fits

I checked the documentation and the FAQ, but did not find any information on update mode limitations.

As a workaround, I can open the file with readonly and overwrite it after making the same changes. If having this file helps: original-data.fits.

Thanks for the help, Mike Kelley

[mkelley]

My workaround is to open the file in readonly mode, modify the header, and overwrite the file, which is fine.

Just now I created a clean Python 3.5 virtual environment, and installed astropy, etc. via pip. The FITS file is updated without any problems. But that probably isn’t surprising since astropy wasn’t compiled on my machine.

Thanks for the help so far!

[pllim]

@mkelley , since this is pretty specific to your machine setup and you found a fix/workaround, I am closing this issue.