Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

io.fits issue reading valid 1-bit binary table

See original GitHub issue

Hi All,

So there is a fits file that is used for testing in the pulsar world (and which came from the CSIRO ATNF archive) that fv and fitsverify both like (and fitsverify thinks it is valid with no errors), but which astropy.io.fits is not liking because of some buffer size issue. I strongly suspect the issue is that the 'DATA' column is in bit array (i.e. format ‘X’) format.

Basically, none of the table data can be read (including the other columns that are not in X format). I’ve poked around with some of the internal size numbers and nothing strikes me as obviously wrong.

Here is the input file (18MB in size)

The following code will trigger the issue both in v4.2 and in v4.3.dev75+g02fe1e16f

import sys
import astropy
import astropy.io.fits as pf

print("Python  version: ", sys.version)
print("Astropy version: ", astropy.__version__)
with pf.open("Parkes_70cm_PSR.fits", mode='readonly') as ff:
    print(ff.info())
    print(ff['SUBINT'].columns)
    x = ff['SUBINT'].data[0]

That produces the following:

Filename: Parkes_70cm_PSR.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      60   ()      
  1  HISTORY       1 BinTableHDU     57   1R x 22C   [24A, 80A, 8A, 8A, 1I, 1I, 1I, 1I, 1D, 1D, 1I, 1D, 1I, 1I, 1I, 1I, 32A, 32A, 32A, 32A, 32A, 32A]   
  2  SUBINT        1 BinTableHDU     76   342R x 18C   [1D, 1D, 1D, 1D, 1D, 1D, 1D, 1D, 1E, 1E, 1E, 1E, 1E, 256E, 256E, 256E, 256E, 393216X]   
None
ColDefs(
    name = 'INDEXVAL'; format = '1D'
    name = 'TSUBINT'; format = '1D'; unit = 's'
    name = 'OFFS_SUB'; format = '1D'; unit = 's'
    name = 'LST_SUB'; format = '1D'; unit = 's'
    name = 'RA_SUB'; format = '1D'; unit = 'deg'
    name = 'DEC_SUB'; format = '1D'; unit = 'deg'
    name = 'GLON_SUB'; format = '1D'; unit = 'deg'
    name = 'GLAT_SUB'; format = '1D'; unit = 'deg'
    name = 'FD_ANG'; format = '1E'; unit = 'deg'
    name = 'POS_ANG'; format = '1E'; unit = 'deg'
    name = 'PAR_ANG'; format = '1E'; unit = 'deg'
    name = 'TEL_AZ'; format = '1E'; unit = 'deg'
    name = 'TEL_ZEN'; format = '1E'; unit = 'deg'
    name = 'DAT_FREQ'; format = '256E'; unit = 'deg'
    name = 'DAT_WTS'; format = '256E'
    name = 'DAT_OFFS'; format = '256E'
    name = 'DAT_SCL'; format = '256E'
    name = 'DATA'; format = '393216X'; unit = 'Jy'; dim = '(1,256,1,1536)'
)
Traceback (most recent call last):
  File "astropy_1bit_fits_problem.py", line 6, in <module>
    x = ff['SUBINT'].data[0]
  File "/home/sransom/python_venvs/py3/lib/python3.8/site-packages/astropy/utils/decorators.py", line 758, in __get__
    val = self.fget(obj)
  File "/home/sransom/python_venvs/py3/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py", line 399, in data
    data = self._get_tbdata()
  File "/home/sransom/python_venvs/py3/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py", line 171, in _get_tbdata
    raw_data = self._get_raw_data(self._nrows, columns.dtype,
  File "/home/sransom/python_venvs/py3/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 520, in _get_raw_data
    return self._file.readarray(offset=offset, dtype=code, shape=shape)
  File "/home/sransom/python_venvs/py3/lib/python3.8/site-packages/astropy/io/fits/file.py", line 330, in readarray
    return np.ndarray(shape=shape, dtype=dtype, offset=offset,
TypeError: buffer is too small for requested array

I’ll keep poking around to see if I can figure out what is going awry.

Here is the system info: Linux-5.8.0-45-generic-x86_64-with-glibc2.32 Python 3.8.6 (default, Jan 27 2021, 15:42:20) [GCC 10.2.0] Numpy 1.20.1 astropy 4.2 Scipy 1.6.1 Matplotlib 3.3.4

Thanks,

Scott