How to save custom aperture masks?

Imagine you painstakingly create a custom aperture mask for aperture photometry. Perhaps you’ve done so with our slick new .interact() method. Or maybe you’ve done so programmatically by some other means that may be difficult to reproduce. What are your options for saving your work?

Chiefly, the options come down to:

1. Do nothing— leave it up to the user to save masks with their desired format (status quo)
2. Save just the mask (in some format-to-be-named-later)
3. Save the lightcurve with e.g. the .to_fits() method with an HDU extension for the mask
4. Save a new target pixel file with a new HDU extension for the mask

[christinahedges]

@gully, @nksaunders and I have had a chat about this.

We’ve weighed up the options from @gully’s first post:

1. Do nothing— leave it up to the user to save masks with their desired format (status quo)
2. Save just the mask (in some format-to-be-named-later)
3. Save the lightcurve with e.g. the .to_fits() method with an HDU extension for the mask
4. Save a new target pixel file with a new HDU extension for the mask

Some take aways from our discussion is below.

Note: @barentsen and I are adding a 4th option, which I’m describing below.

1. Save a fits file that would be opened by a GenericLightCurveFile class. The aperture information would be accessible via GenericLightCurveFile.hdu[-1].data.

Option 1. Save just the mask

Masks would be fits files with keywords describing the way they were generated and the file they apply to.

Pros:

• Small file size
• No change to lightkurve infrastructure needed.

Cons:

• Have to carry around mask and TPF to use.
• Have to include a way to read in masks in lightkurve (i.e. tpf.to_lightcurve(aperture_mask=parse_mask('mask.fits')))
• Masks could be applied to the wrong file.

Option 2. Save a fits file with a light curve and an aperture extension

Pros:

• Small file size
• Easy to reproduce/share with collaborators
• No intermediate steps
• Will work for TESS regardless of postage stamp/tpf/two minute cadence etc
• Potentially adds infrastructure for use when loading in HLSPs.

Cons:

• Not inherently reproducible. To reproduce the light curve contained in the fits file you would also need the TPF.
• Requires a new attribute aperture_mask in all LightCurve, which is populated or produces a descriptive warning in all subclasses. This is a fairly large change to a major class in lightkurve.
• The aperture_mask attribute must be writeable in the LightCurve.to_fits()
• Custom light curves fits files from tpf.interact will not behave like Light Curve Files. They must still be readable by lightkurve, into KeplerLightCurve objects (or TESSLightCurve objects). This is new behavior which needs to be written and unit tested.

Watch out for:

• Need to ensure that the fits file generated by tpf.interact() has no information loss compared with the equivalent lcf.SAP_FLUX() object.

• Have to ensure that the fits file generated by the tpf.interact() button can be read back into lightkurve. (i.e. lk.open('custom_lc_file.fits') must produce the same object with the same attributes as the equivalent lcf.SAP_FLUX() object.) Gully refers to this as round tripping above. This is necessary for this button to really be useful. If we include this button in lightkurve v1.0, this issue must be addressed.

• lk.open('custom_lc_file.fits') must return a KeplerLightCurve or TESSLightCurve class, not a generic LightCurve class. (Otherwise information has been lost.)

• This option will require that the KeplerLightCurve returned by lk.open('custom_lc_file.fits') will have a new attribute aperture_mask. This requires changing the base class LightCurve to have a blank aperture_mask attribute that will be able to be set. When the attribute is accessed from different objects it should always provide a reasonable result, see below. All of these scenarios must be unit tested

tpf = open('targetpixelfile.fits')
tpf.aperture_mask     # equivalent to tpf.pipeline_mask

lc = tpf.to_lightcurve()
lc.aperture_mask      # provides pipeline aperture

lc = LightCurve([1,2,3,4], [1,1,1,1])
lc.aperture_mask      # raises a helpful exception saying this attribute has not been set

lcf = open('lightcurvefile.fits')
lcf.aperture_mask   # MUST either produce a useful warning/exception, or fetch the aperture mask
lcf.SAP_FLUX.aperture_mask   # MUST either produce a useful warning/exception, or fetch the aperture mask

lcf.SAP_FLUX.to_fits('my_hlsp_file.fits')
my_hlsp_lc = open('my_hlsp_file.fits')
my_hlsp_lc.aperture_mask   # Has no aperture extension?

lc = open('custom_light_curve.fits')
lc.aperture_mask     # provides pipeline aperture
lc.to_fits('my_custom_file.fits')
my_custom_lc = open('my_custom_file.fits')
my_custom_lc.aperture_mask   # Does have an aperture extension?

Option 3. Save a new target pixel file with a new HDU extension for the mask

Pros:

• Provides all the information for reproducible light curves, can share a single file with collaborators.
• Requires no infrastructure change for lightkurve.

Cons:

• Large file size, irresponsible to fill up users hard drives
• Inconvenient for power users, who only want the aperture mask.

Option 4 Save a fits file that would be opened by a GenericLightCurveFile class.

@barentsen and I have been chatting and like a fourth option. Here we would create a new GenericLightCurveFile class which would attempt to read in any fits file. This would be able to have any extensions we liked. Some behaviors would include:

lcf = open('GenericLightCurveFileExample.fits')
lc = lcf.SAP_FLUX

aperture_mask = lcf.hdu[-1].data

If the fits file has the keyword MISSION = KEPLER, lc.SAP_FLUX would return a KeplerLightCurve. If the fits file has the keyword MISSION = TESS, lc.SAP_FLUX would return a TESSLightCurve. If the fits file has an unrecognized or no MISSION keyword then it would return a LightCurve.

All my concerns about ensuring no information is lost from option 2 still apply. The output from tpf.interact() #SAVE\ lc = open('custom_file.fits').SAP_FLUX must be exactly equivalent to tpf.to_lightcurve()

Pros:

• Small file size
• Clearly a custom file
• Requires a small amount of lightkurve changes which are easy to unit test and support
• Will enable future reading in of HLSPs

Cons:

• Requires a new subclass of LightCurveFile
• Requires open() to have a new file type to open.
• It is not pleasant to access the mask, but it is doable.

Some decisions

@gully has made lots of headway on option 2 in PR #329. I’m writing my concerns about option two in detail here. I find option 1 to probably on balance be the most palatable, although I don’t love the idea of lots of dangling mask files that I’m easily going to lose.

If all my concerns under option 2 can be addressed then it is the best solution. However, it looks like quite a bit of work, especially before we release v1.0.

[barentsen]

Below I copy/paste the exact specification from the Kepler archive manual and the TESS documentation. They are different but they both give the same meaning to bits 1 and 2, so I think we may want to stick to those!