question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

QTable ignores dtype, tries converting string column with unit to Quantity

See original GitHub issue

Description

I have a table that (ab)uses the “unit” metadata for some additional info on non-numeric strings, mostly by putting the String string there:

[…]
TTYPE10 = 'TEMPLATE'                                                            
TFORM10 = '30A     '                                                            
TUNIT10 = 'String  '                                                            
[…]

astropy.table.QTable.read() raises a type error when trying to read this file, so that I need to revert to Table, which gives other unexpected problems (#11521).

/usr/lib/python3/dist-packages/astropy/table/table.py in _convert_data_to_col(self, data, copy, default_name, dtype, name)
   1081             raise ValueError('unable to convert data to Column for Table')
   1082 
-> 1083         col = self._convert_col_for_table(col)
   1084 
   1085         return col

/usr/lib/python3/dist-packages/astropy/table/table.py in _convert_col_for_table(self, col)
   3649             # identified in the unit (such as u.mag()).
   3650             q_cls = getattr(col.unit, '_quantity_class', Quantity)
-> 3651             qcol = q_cls(col.data, col.unit, copy=False)
   3652             qcol.info = col.info
   3653             qcol.info.indices = col.info.indices

/usr/lib/python3/dist-packages/astropy/units/quantity.py in __new__(cls, value, unit, dtype, copy, order, subok, ndmin)
    380             not (value.dtype.kind == 'O' and
    381                  isinstance(value.item(0), numbers.Number))):
--> 382             raise TypeError("The value must be a valid Python or "
    383                             "Numpy numeric type.")
    384 

TypeError: The value must be a valid Python or Numpy numeric type.

It would be good if these columns could just kept as-is in case of non-numeric columns (where units make no sense anyway), at least with an additioal flag in QTable.read().

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
olebolecommented, Apr 12, 2021

The following code shows it:

from astropy.table import Table, QTable

t = Table({'foo': list('*'*100)})  # just a sample string column
t['foo'].unit = 'String'
qt = QTable(t)                     # convert to QTable

which gives the exception shown above.

0reactions
mhvkcommented, Apr 16, 2021

See #11585 for a possible fix.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python Pandas Convert String to int/float - Stack Overflow
I have tried pd.to_numeric(), but without success. I think my problem is with the 'hours' tacked on the end of each string. Is...
Read more >
Column — Astropy v1.0.4
Print a formatted string representation of column values. to(unit[, equivalencies]), Converts this table column to a Quantity object with the requested units.
Read more >
Constructing a Table — Astropy v5.2
This will allow the column to be stored in the table as a native Quantity and bring the full power of Units and...
Read more >
Cleaning Up Currency Data with Pandas
This article includes tips on how to clean up messy currency data in pandas so that you may convert the data to numeric...
Read more >
IO tools (text, CSV, HDF5, …) — pandas 1.5.2 documentation
Function to use for converting a sequence of string columns to an array of ... dtype=object) In [17]: df Out[17]: a ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found