stitch() across QLP and SPOC lightcurves result in error

Problem description

When I try to use LightCurveCollection.stitch() across QLP and SPOC lightcurves, it result in the following error.

ValueError: TessLightCurve object is invalid - expected 'time' as the first columns but found 'time'

I encountered the issue in Lightkurve v2.1.1 . It used to work with Lightkurve 2.0.10. So I suspect the issue is likely related to the Astropy 5.0 upgrade that comes with Lightkurve v2.1.

Example

import lightkurve as lk

sr = lk.search_lightcurve("TIC198394082", mission="TESS")
# sector 26 is from QLP , sector 40 is from SPOC
lcc = sr[(sr.mission == "TESS Sector 26") | (sr.mission == "TESS Sector 40")].download_all(download_dir="data")
lcc.stitch()

Expected behavior

In Lightkurve 2.0.10 (with Astropy 4.2.1), the stitch() works (along with some warnings). In Lightkurve 2.1.1 (with Astropy 5.0.2), it results in the following error.

--------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [5], in <cell line: 6>()
      4 # sector 26 is from QLP , sector 40 is from SPOC
      5 lcc = sr[(sr.mission == "TESS Sector 26") | (sr.mission == "TESS Sector 40")].download_all(download_dir="data")
----> 6 lcc.stitch()

File C:\dev\lightkurve\src\lightkurve\collections.py:217, in LightCurveCollection.stitch(self, corrector_func)
    212     warnings.warn(
    213         f"The following columns will be excluded from stitching because the column types are incompatible: {columns_to_remove}",
    214         LightkurveWarning,
    215     )
    216     lcs = [lc.copy() for lc in lcs]
--> 217     [
    218         lc.remove_columns(columns_to_remove.intersection(lc.columns))
    219         for lc in lcs
    220     ]
    222 # Need `join_type='inner'` until AstroPy supports masked Quantities
    223 return vstack(lcs, join_type="inner", metadata_conflicts="silent")

File C:\dev\lightkurve\src\lightkurve\collections.py:218, in <listcomp>(.0)
    212     warnings.warn(
    213         f"The following columns will be excluded from stitching because the column types are incompatible: {columns_to_remove}",
    214         LightkurveWarning,
    215     )
    216     lcs = [lc.copy() for lc in lcs]
    217     [
--> 218         lc.remove_columns(columns_to_remove.intersection(lc.columns))
    219         for lc in lcs
    220     ]
    222 # Need `join_type='inner'` until AstroPy supports masked Quantities
    223 return vstack(lcs, join_type="inner", metadata_conflicts="silent")

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1\lib\site-packages\astropy\timeseries\core.py:32, in autocheck_required_columns.<locals>.decorator_method.<locals>.wrapper(self, *args, **kwargs)
     29 @wraps(method)
     30 def wrapper(self, *args, **kwargs):
     31     result = method(self, *args, **kwargs)
---> 32     self._check_required_columns()
     33     return result

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1\lib\site-packages\astropy\timeseries\core.py:79, in BaseTimeSeries._check_required_columns(self)
     73     raise ValueError("{} object is invalid - expected '{}' "
     74                      "as the first column{} but time series has no columns"
     75                      .format(self.__class__.__name__, required_columns[0], plural))
     77 elif self.colnames[:len(required_columns)] != required_columns:
---> 79     raise ValueError("{} object is invalid - expected '{}' "
     80                      "as the first column{} but found '{}'"
     81                      .format(self.__class__.__name__, required_columns[0], plural, self.colnames[0]))
     83 if (self._required_columns_relax
     84         and self._required_columns == self.colnames[:len(self._required_columns)]):
     85     self._required_columns_relax = False

ValueError: TessLightCurve object is invalid - expected 'time' as the first columns but found 'time'

Note: stitch() call also generates the following warning. It happened in Lightkurve 2.0.10 as well, and is not directly related to the error.

C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1\lib\site-packages\lightkurve\collections.py:212: LightkurveWarning: The following columns will be excluded from stitching because the column types are incompatible: {'flux_err', 'sap_bkg', 'flux', 'sap_bkg_err', 'sap_flux'}
  warnings.warn(

Environment

• platform (e.g. Linux, OSX, Windows): Windows
• lightkurve version (e.g. 1.0b6): 2.1.1
• Astropy version: 5.0.2
• installation method (e.g. pip, conda, source): conda

[orionlee]

Root cause identified. It is related to the use of MaskedQuantity with Astropy 5.0

In the example given with Lightkurve 2.1.1, the stitch() call incorrectly tries to remove flux and flux_err columns, both of which are required. The Astropy exception was trying to say required columns were being dropped (the actual exception message was misleading).

...\lightkurve\collections.py:212: LightkurveWarning: The following columns will be excluded from stitching because the column types are incompatible: 
{'flux_err', 'sap_bkg', 'flux', 'sap_bkg_err', 'sap_flux'}
  warnings.warn(

(My bad for not noticing the difference in the warning.)

The logic for determining if a column is compatible (between LC objects) for stitching breaks down with Astropy 5.0

https://github.com/lightkurve/lightkurve/blob/ad8eb62ae0f6b9d7ecf4183c391fe9891a62a969/src/lightkurve/collections.py#L202-L208

In the example given, flux column from QLP is a a regular Quantity, while the flux column from SPOC is a MaskedQuantity, they are incorrectly deemed as incompatible by the above logic.

# lcc[0] is from QLP, lcc[1] is from SPOC
>>> lc_qlpn, lc_spocn = lcc[0].normalize(), lcc[1].normalize()
>>> lc_qlpn["flux"].__class__, lc_spocn["flux"].__class__
(astropy.units.quantity.Quantity,
 astropy.utils.masked.core.MaskedQuantity)
>>> issubclass(lc_qlpn["flux"].__class__, lc_spocn["flux"].__class__)
 False

[barentsen]

Update: a fix for this issue has been included in the Lightkurve v2.2.0 release.