FITS Verification throws exception even after fixing an invalid header value

Description

Using astopy.io.fits, I was trying to fix a header card that has non-ASCII characters in its value, by simply replacing its value with empty string. Though the hdu.header['XYZ'] prints '' after assigning the same to it. But on writing it to file (to save my changes) the same exception which was thrown during reading the file is thrown, as if no change has taken place in the hdu.

Expected behavior

Since the value in header card have changed, the .writeto() with default output_verify='exception should not lead to the exception.

Actual behavior

After my changes on 1st HDU i.e. hdul[0], when I do hdul.writeto() for saving the file or `hdul.verify(‘exception’), the following exception occurs:

VerifyError: 
Verification reported errors:
HDU 0:
    Card 16:
        Card 'XYZ' is not FITS standard (invalid value string: "'\x00\x04-\x00?\x05-\x00\x02\x00\x0c9\x02\x00/\x17$\x0c'          /").
Note: astropy.io.fits uses zero-based indexing. 

Steps to Reproduce

hdul = fits.open('abc.fits') 

hdul.info()
# Filename: abc.fits
# No.    Name      Ver    Type      Cards   Dimensions   Format
#  0  PRIMARY       1 PrimaryHDU      17   (1024, 1024)   int16

hdul[0].header
# ValueError: FITS header values must contain standard printable ASCII characters; "'\x00\x04-\x00?\x05-\x00\x02\x00\x0c9\x02\x00/\x17$\x0c'          /" contains characters not representable in ASCII or non-printable characters.

hdul[0].verify('exception') # to identify which key has problem
# VerifyError: 
# Verification reported errors:
# Card 16:
#    Card 'XYZ' is not FITS standard (invalid value string: "'\x00\x04-\x00?\x05-\x00\x02\x00\x0c9\x02\x00/\x17$\x0c'          /").
# Note: astropy.io.fits uses zero-based indexing.

hdul[0].header['XYZ']="" # it's non-important so replacing value by "" is enough

hdul[0].header['XYZ'] # check if value has changed
# '' 

hdul.writeto('fixed_abc.fits') # or hdul.verify('exception')
# VerifyError: 
# Verification reported errors:
# HDU 0:
#     Card 16:
#         Card 'XYZ' is not FITS standard (invalid value string: "'\x00\x04-\x00?\x05-\x00\x02\x00\x0c9\x02\x00/\x17$\x0c'          /").
# Note: astropy.io.fits uses zero-based indexing. 

I’m guessing that changes made by header key assignment in the hdu in memory, is not the hdu which is being used for verification?

And surprisingly, I found a strange workaround that if I print complete header before saving hdul, it works fine:

hdul[0].header
# SIMPLE  =                    T          /                                       
# BITPIX  =                   16          /                                       
# NAXIS   =                    2          /                                       
# NAXIS1  =                 1024          /                                       
# NAXIS2  =                 1024          /    
# ...
# XYZ     = ''

hdul.writeto('fixed_abc.fits') # works!!

Is this something related how hdu is modified in memory by printing full header?

System Details

Linux-5.4.0-67-generic-x86_64-with-glibc2.31 Python 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46) [GCC 9.3.0] Numpy 1.20.1 astropy 4.2 Scipy 1.6.1 Matplotlib 3.3.4

[embray]

I agree this is a duplicate of #11108. Thanks for the test case @jaladh-singhal , I was able to create an invalid FITS file like the one you described, and confirmed that your test case is now fixed.

[pllim]

Maybe something to do with lazy loading, @saimn ?