Table Merge Error

Description

We are trying to take an astropy table with missing values and merge it with another data table to replace those missing values with. The bug here is there is a masked column that doesn’t want to take in those values.

Expected behavior

Allow us to merge two data tables and be able to replace missing values with the ones provided.

Actual behavior

We get a TableMerge Error and don’t know how to go about fixing it.

Steps to Reproduce

from astropy.table import Table, join
import pyvo as vo
from astropy.table import Table
import astropy.io.votable
from astropy.io.votable import parse

 # In Online Mode (default), we will query the latest catalog from the Exoplanet Archive.
nasa_service = vo.dal.TAPService("https://exoplanetarchive.ipac.caltech.edu/TAP")
nasa_search_query = "SELECT pl_name, hostname,sy_snum,sy_pnum, pl_orbper, pl_orbpererr1, pl_orbpererr2, pl_orbperlim, pl_bmassj, \
                        pl_orbeccen,pl_orbeccenerr1,pl_orbeccenerr2, \
                        pl_orbeccenlim,st_spectype,st_rad,st_raderr1,st_raderr2, st_radlim,st_mass,st_masserr1, \
                        st_masserr2,st_masslim,st_age,st_ageerr1,st_ageerr2,st_agelim \
                        FROM pscomppars "
nasa_results = nasa_service.search(nasa_search_query)
nasa_table_data = vo.dal.TAPResults.to_table(nasa_results)
        
        # In Online Mode (default), we will query the latest catalog from the eu catalog for missing data .
eu_service =vo.dal.TAPService("http://voparis-tap-planeto.obspm.fr/tap")
eu_search_query = "SELECT target_name, star_name, mass, mass_error_min, mass_error_max, mass_sin_i, mass_sin_i_error_min, mass_sin_i_error_max \
                    period, period_error_min, period_error_max, \
                    eccentricity,eccentricity_error_min, eccentricity_error_max FROM exoplanet.epn_core"   
eu_results = eu_service.search(eu_search_query)
eu_table_data = vo.dal.TAPResults.to_table(eu_results)
        
       
        #merge nasa_table_data & eu_table_data tables here and name it merged_table using Archive colunm names
names = ('target_name','star_name','period', 'period_error_min', 'period_error_max', 'eccentricity','eccentricity_error_min', 'eccentricity_error_max')
new_names = ('pl_name','host_name', 'pl_orbper', 'pl_orbpererr2','pl_orbpererr1', 'pl_orbeccen','pl_orbeccenerr2','pl_orbeccenerr1')
eu_table_data.rename_columns(names, new_names)
#print(t)
        

print(join(nasa_table_data, eu_table_data, join_type='right'))


---------------------------------------------------------------------------
TableMergeError                           Traceback (most recent call last)
/tmp/ipykernel_802/2606867453.py in <module>
     31 
     32 
---> 33 print(join(nasa_table_data, eu_table_data, join_type='right'))

/opt/conda/lib/python3.9/site-packages/astropy/table/operations.py in join(left, right, keys, join_type, uniq_col_name, table_names, metadata_conflicts, join_funcs)
    382 
    383     col_name_map = OrderedDict()
--> 384     out = _join(left, right, keys, join_type,
    385                 uniq_col_name, table_names, col_name_map, metadata_conflicts,
    386                 join_funcs)

/opt/conda/lib/python3.9/site-packages/astropy/table/operations.py in _join(left, right, keys, join_type, uniq_col_name, table_names, col_name_map, metadata_conflicts, join_funcs)
   1125                                       .format(arr_label, name))
   1126             if hasattr(arr[name], 'mask') and np.any(arr[name].mask):
-> 1127                 raise TableMergeError('{} key column {!r} has missing values'
   1128                                       .format(arr_label, name))
   1129 

TableMergeError: Left key column 'pl_orbper' has missing values

System Details

Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-glibc2.31

Python 3.9.5 | packaged by conda-forge | (default, Jun 19 2021, 00:32:32) [GCC 9.3.0] Numpy 1.21.0 astropy 4.2.1 Scipy 1.7.0 Matplotlib 3.3.4

[eerovaher]

If the inputs of join() contain matching non-key column names then the output will indeed contain columns from both tables, which are renamed to be distinguishable. It is possible to have some control over how the renaming gets done, see the documentation for details: https://docs.astropy.org/en/v4.2/table/operations.html#column-renaming

Let column_names be the the column names of your primary data and replacement_column_names the corresponding column names of the replacement values. If a column contains missing values then its mask attribute tells where the missing values are. So you can do something like

for col, replacement_col in zip(column_names, replacement_column_names):
    mask = joined_table[col].mask
    joined_table[col][mask] = joined_table[replacement_col][mask]
    joined_table.remove_column(replacement_col)    # If it's not needed anymore it can be removed

This has to be done using the joined table because join() makes sure the rows are aligned.

[astro-alice]

Testing it out still, but will do!