FITS Table columns read as 2D arrays

Description

The following code:

from astropy.io import fits
from astropy.table import Table

with fits.open('whatever.fits') as f:
    d = Table.read(f)
print(d['SUPRA'].shape)

outputs (1294, 1) in v4.2.1 and (1294,) in v3.1.2.

This makes quite a bit of our previous code non-backwards compatible.

Expected behavior

I expected (1294,) as in previous versions.

Actual behavior

Output is (1294, 1) with v4.2.1.

Steps to Reproduce

See code above.

System Details

Linux-5.11.0-25-generic-x86_64-with-glibc2.29
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0]
Numpy 1.20.2
astropy 4.2.1
Scipy 1.6.2
Matplotlib 3.4.1

[maxnoe]

So the behavior of astropy here is correct and following the standard.

[maxnoe]

The Problem is not TFORM, but TDIM.

An empty TDIM is a 0D element, so the whole column is 1D. A TDIM of (1) means that each element is a 1d array of 1 element, so the whole column becomes 2D.

Please fix the code producing your files if having 2d columns is not the intention. The FITS standard is pretty clear on this:

TDIMn keywords. Thevalue field of this indexedkeywordshall contain a character string describing how to interpret the contents of Field n as a multi-dimensional array with a format of ’(l,m,n…)’, where l, m, n, …are the dimensions of the array. The data are ordered such that the array index of the first dimension given (l) is the most rapidly varying, and that of the last dimension given is the least rapidly varying. The total number of elements in the array equals the product of the dimensions specified in the TDIMn keyword