Support nulls in ECSV boolean columns?

Description

The ECSV format is described, and as far as I know defined, in https://github.com/astropy/astropy-APEs/blob/master/APE6.rst. That document does not discuss in any detail the encoding of blank values, but e.g. an empty string value in an int32-typed column can be read by the Astropy implementation, and is represented by a masked value. However an empty string value in a bool-typed column causes a read error. If file refers to the following text:

# %ECSV 0.9
# ---
# datatype: [
#   { name: AA, datatype: int32   },
#   { name: BB, datatype: bool    },
#   { name: CC, datatype: float64 },
# ]
AA BB CC
1 True 2.1
2 False 5.4
"" "" nan
9 True -9.9

then astropy.table.Table.read(file, format='ascii.ecsv') gives me a ValueError: Column BB failed to convert: bool input strings must be only False or True. This is Astropy 4.0, Python 3.6.

Is this intended behaviour? There are cases where a boolean column could contain nulls as well as Trues and Falses (I came across this myself when trying to represent certain Gaia data in ECSV - the table in question had boolean columns containing null values, which meant the Astropy reader could not read my ECSV serialisation of the table). APE6 does say “Boolean fields are represented as the case-sensitive string False or True”, but since nulls are not discussed elsewhere (e.g. it doesn’t mention nulls in integer columns) it’s not clear whether this counts as a declaration that nulls are intentionally outlawed for the bool type.

The ECSV I/O handlers I have written for STIL/TOPCAT do work with blanks in bool-typed columns. I request that the Astropy ECSV implementation considers doing the same.

[mbtaylor]

Sure. What I’d expect/like to see is like this:

>>> from astropy.table import Table
>>> t2 = Table.read(s, format='ascii.ecsv')
>>> t2
  AA    BB     CC
int32  bool float64
----- ----- -------
    1  True     2.1
    2 False     5.4
   --    --     nan
    9  True    -9.9
>>> t2[2][0]
masked
>>> t2[2][1]
masked
>>> type(t2[1][1])
<class 'numpy.bool_'>
>>> type(t2[2][1])
<class 'numpy.bool_'>

For the io.ascii.read option, the BB column header does not report type bool and the cell types are not reported as numpy.bool_.

[mbtaylor]

Great! Many thanks for quick service.