FITS headers don't update immediately?

This is an issue noted on https://github.com/astropy/astropy/issues/5396, but is independent of that issue.

In short, given a FITS header with an invalid entry (e.g., a newline within the 'ORIGIN' value), the following will happen:

fh = fits.open(fn)
fh[0].header.tostring() # produces exception
fh[0].header.tostring() # does not produce exception

>>> h = fits.open('test1n.fits')
>>> h[0].header['origin'] = 'goodval'
>>> w = wcs.WCS(h[0].header)  # <- crashes
# a repeat attempt here would (probably) not crash

see: https://github.com/astropy/astropy/issues/5396#issuecomment-252687514, https://github.com/astropy/astropy/issues/5396#issuecomment-252699025, https://github.com/astropy/astropy/issues/5396#issuecomment-252700815, https://github.com/astropy/astropy/issues/5396#issuecomment-252701891

[saimn]

Just to confirm that there is clearly an issue :

In [1]: %astropy
Numpy 1.12.1
Astropy 2.0.dev18422

In [2]: c = fits.Card.fromstring('FOO     = BAR\n   ')

In [3]: c.value
---------------------------------------------------------------------------
VerifyError                               Traceback (most recent call last)
<ipython-input-3-d335c3962011> in <module>()
----> 1 c.value

/home/simon/dev/astropy/astropy/io/fits/card.py in value(self)
    279             value = self._value
    280         elif self._valuestring is not None or self._image:
--> 281             self._value = self._parse_value()
    282             value = self._value
    283         else:

/home/simon/dev/astropy/astropy/io/fits/card.py in _parse_value(self)
    738         if m is None:
    739             raise VerifyError("Unparsable card ({}), fix it first with "
--> 740                               ".verify('fix').".format(self.keyword))
    741 
    742         if m.group('bool') is not None:

VerifyError: Unparsable card (FOO), fix it first with .verify('fix').

In [4]: c.value = 'FIXED'

In [5]: c.value
Out[5]: 'FIXED'

In [6]: c
Out[6]: ('FOO', 'FIXED', '')

In [7]: c.verify('fix')
WARNING: VerifyWarning: Verification reported errors: [astropy.io.fits.verify]
WARNING: VerifyWarning: Card 'FOO' is not FITS standard (invalid value string: 'BAR').  Fixed 'FOO' card to meet the FITS standard. [astropy.io.fits.verify]
WARNING: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.
 [astropy.io.fits.verify]

In [8]: c.value
Out[8]: 'BAR'

It looks like the private Card._image is not updated when setting a new value, and this attribute is used at least when verifying the card which brings back the old value ! 😨

[mwcraig]

That is indeed the fix. I had several hundred images with newline characters in the header, fixed them with this snippet (your workaround is in the last few lines):

from ccdproc import ImageFileCollection
from astropy.io.fits import VerifyError

ic = ImageFileCollection('.')

for hdu in ic.hdus(save_location='foo'):
    h = hdu.header
    bads = []
    for i, c in enumerate(h.cards):
        if isinstance(c[1], str) and '\n' in c[1]:
           bads.append(i)
    for b in bads:
        new = h[b].replace('\n', '~~')
        h[b] = new
    try:
        h.tostring()
    except VerifyError:
        # Yes, seriously, this.
        h.tostring()
    h.update()