CBVCorrector fails?

Problem description

CBVCorrector fails executing a minimal example from the tutorial.

Example

Input Script

import lightkurve as lk
from lightkurve.correctors import CBVCorrector

import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl

print(lk.__version__)
lc = lk.search_lightcurve('TIC 737578805',sector=32,exptime=120)[0].download().remove_nans()
print('downloaded')

cbvCorrector = CBVCorrector(lc)
print('initialized corrector - using download')

# run cbv correction
# Select which CBVs to use in the correction
cbv_type = ['SingleScale', 'Spike']

# Select which CBV indices to use
# Use the first 8 SingleScale and all Spike CBVS
cbv_indices = [np.arange(1,9), 'ALL']

# Perform the correction
print('Correcting!')
cbvCorrector.correct(cbv_type=cbv_type, cbv_indices=cbv_indices);

Output

(lk) d-i184-54-49:lightkurve benjaminpope$ python lktest.py
2.0.10
WARNING: dropping mask in Quantity column 'flux': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'flux_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'centroid_col': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'centroid_row': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_flux': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_flux_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_bkg': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_bkg_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pdcsap_flux': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pdcsap_flux_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr1': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr1_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr2': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr2_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr1': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr1_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr2': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr2_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pos_corr1': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pos_corr2': masked Quantity not supported [astropy.table.table]
downloaded
initialized corrector - using download
Correcting!
WARNING: dropping mask in Quantity column 'flux': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'flux_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'centroid_col': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'centroid_row': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_flux': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_flux_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_bkg': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'sap_bkg_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pdcsap_flux': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pdcsap_flux_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr1': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr1_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr2': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'psf_centr2_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr1': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr1_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr2': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'mom_centr2_err': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pos_corr1': masked Quantity not supported [astropy.table.table]
WARNING: dropping mask in Quantity column 'pos_corr2': masked Quantity not supported [astropy.table.table]
WARNING: File may have been truncated: actual file length (458752) is smaller than the expected size (1895040) [astropy.io.fits.file]
Traceback (most recent call last):
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/timeseries/sampled.py", line 366, in read
    return super().read(filename, format=format, *args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/table/connect.py", line 61, in __call__
    out = registry.read(cls, *args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/registry.py", line 527, in read
    data = reader(*args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/io/tess.py", line 34, in read_tess_lightcurve
    lc = read_generic_lightcurve(filename, flux_column=flux_column, time_format="btjd")
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/io/generic.py", line 40, in read_generic_lightcurve
    tab = Table.read(hdulist[ext], format="fits")
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/table/connect.py", line 61, in __call__
    out = registry.read(cls, *args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/registry.py", line 527, in read
    data = reader(*args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/fits/connect.py", line 234, in read_table_fits
    data = table.data
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/utils/decorators.py", line 767, in __get__
    val = self.fget(obj)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py", line 402, in data
    data = self._get_tbdata()
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py", line 172, in _get_tbdata
    raw_data = self._get_raw_data(self._nrows, columns.dtype,
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 529, in _get_raw_data
    return self._file.readarray(offset=offset, dtype=code, shape=shape)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/io/fits/file.py", line 324, in readarray
    return np.ndarray(shape=shape, dtype=dtype, offset=offset,
TypeError: buffer is too small for requested array

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lktest.py", line 30, in <module>
    cbvCorrector.correct(cbv_type=cbv_type, cbv_indices=cbv_indices);
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/correctors/cbvcorrector.py", line 458, in correct
    minimize_result = minimize_scalar(self._goodness_metric_obj_fun, method='Bounded',
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/scipy/optimize/_minimize.py", line 804, in minimize_scalar
    return _minimize_scalar_bounded(fun, bounds, args, **options)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/scipy/optimize/optimize.py", line 2024, in _minimize_scalar_bounded
    fx = func(x, *args)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/correctors/cbvcorrector.py", line 811, in _goodness_metric_obj_fun
    underMetric = self.under_fitting_metric()
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/correctors/cbvcorrector.py", line 606, in under_fitting_metric
    metric = underfit_metric_neighbors (corrected_lc, 
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/correctors/metrics.py", line 174, in underfit_metric_neighbors
    lc_neighborhood, lc_neighborhood_flux = _download_and_preprocess_neighbors(
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/memoization/caching/plain_cache.py", line 45, in wrapper
    result = user_function(*args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/correctors/metrics.py", line 333, in _download_and_preprocess_neighbors
    lcfCol = search.download_all(flux_column=flux_column)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/utils.py", line 555, in wrapper
    return f(*args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/search.py", line 423, in download_all
    self._download_one(
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/search.py", line 290, in _download_one
    return read(path, quality_bitmask=quality_bitmask, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/io/read.py", line 92, in read
    return TessLightCurve.read(path_or_url, format="tess", **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/lightkurve/lightcurve.py", line 3078, in read
    return super().read(*args, **kwargs)
  File "/Users/benjaminpope/opt/anaconda3/envs/lk/lib/python3.8/site-packages/astropy/timeseries/sampled.py", line 373, in read
    raise ValueError("``time_column`` should be provided since the default Table readers are being used.")
ValueError: ``time_column`` should be provided since the default Table readers are being used.

Expected behavior

Expected to replicate tutorial at https://docs.lightkurve.org/tutorials/2-creating-light-curves/2-3-how-to-use-cbvcorrector.html .

Environment

All in Python 3.8 fresh install from Conda, in a fresh environment with no other packages installed other than a pip install of lightkurve. I’m separately testing a fork of the dev version for CBV IO in #1122, and a similar bug happens there.

• platform: OSX
• lightkurve version: 2.0.10
• installation method: pip

[benjaminpope]

Hi @jcsmithhere, thanks, the latest dev version of lightkurve works. It was both a problem with an incomplete fits file (it progressed further when I cleared the cache) and the problem fixed by PR #1120 (in the sense that it didn’t progress to completion without this). Fixed!

[jcsmithhere]

Yes, that error should have been fixed with PR #1120. @benjaminpope, with that fix can you run to completion? If so, was the issue due to the incomplete FITS file in your local cache?

Regarding the search_neighbors code, yes it only searches neighbors where the author is ("Kepler", "K2", "SPOC"). Is this causing problems? The idea being we only want to find official light curves from the Kepler SOC and TESS SPOC. There is a uniformity of processing of those light curves which we have control over.