question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

non-dates are often converted to datetime

See original GitHub issue

Described
issue where non-dates are often converted to datetime.

Environment OS: Windows 10 Python: 3.8.2 IDE: Scite

Package versions
pandasgui==0.2.11 pandas==1.1.4



I changed utility.py line  to

`            return pd.to_datetime(s, format="%b %d %Y")`
and now I only see
_PandasGUI WARNING — pandasgui.utility — In metadata_df, converted columns to datetime: RecordingDate_
which is desired.

It does convert everything else to a string (like float columns), but that is not a big issue for me, visually.
Here are the first two lines and header which are a result of `df.to_csv('metadata_df.csv')` which I do just before `show(metadata_df)`
[metadata_df.txt](https://github.com/adamerose/PandasGUI/files/6625776/metadata_df.txt)


Firstly, when I upgraded to 0.2.11 it seems to need a new module
`>pip install pandasgui==0.2.11`
I got `ModuleNotFoundError: No module named 'qtstylish'`

python Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32

import pandas pandas.version ‘1.1.4’ import pandasgui Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “C:\Python38\lib\site-packages\pandasgui_init_.py”, line 15, in <module> from pandasgui.gui import show File “C:\Python38\lib\site-packages\pandasgui\gui.py”, line 20, in <module> import qtstylish ModuleNotFoundError: No module named ‘qtstylish’ import pandasgui pandasgui.version ‘0.2.11’

but it did install fine: `pip install qtstylish`, so now I have

python Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32

import pandas pandas.version ‘1.1.4’ import pandasgui pandasgui.version ‘0.2.11’


I did see the simple call in utility.py parse_dates_series(), so it appears to me now it is a pandas "thing", but maybe the question was why is pandasgui trying to recast columns at all?


So, it is likely that reading from CSV is not a proper test here.
The data was strings read from report forms, and the df types as created are

… Side object Location object NIHSScore float64 TICIScore object IV_tPA_timing object EVT_timing object RecordingDate object RecordingStartTime object RecordingStopTime object Gender object Age object SystolicBP object DiastolicBP object


my calling code saves that df to csv, then opens pandasgui on the dataframe for a look
    metadata_df.to_csv('metadata_df.csv')
    try:
        show(metadata_df)
    except ValueError:
        pass
When I call show() I get:
![a](https://user-images.githubusercontent.com/20799053/121825595-3737c900-cc68-11eb-9ac0-550e9e204e55.jpg)



I might need to generate a pickle of metadata_df for you to test, if you'd like; if I "filter" the df  through CSV, I get a result very similar to your screen cap, but with floats instead of ints
![b](https://user-images.githubusercontent.com/20799053/121825591-31da7e80-cc68-11eb-9d1f-594aed5f982c.jpg)


Small data pkl in a zip:
[head.zip](https://github.com/adamerose/PandasGUI/files/6644870/head.zip)

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
rjsdotorgcommented, Jun 26, 2021

Very nice, thanks!

0reactions
adamerosecommented, Jun 26, 2021

The columns being converted with warnings are object type in the pkl, rather than the newer StringDtype

Ah yeah you’re right, I convert columns to date if they’re object type because that’s often the type of columns containing strings, and I believe there’s no way to differentiate between typical string columns and other types of object column like dict or whatnot, so I just try converting every object column to a date. StringDtype is actually unaccounted for, I’ll fix that.

I loaded your pickle and ran parse_dates and these are the columns that got turned into dates on my end.

# This is good
RecordingDate          datetime64[ns]
RecordingStartTime     datetime64[ns]
RecordingStopTime      datetime64[ns]
# This is wrong. These `object` columns are all NaN and pd.to_datetime still converts them to datetime
TICIScore              datetime64[ns]
IV_tPA_timing          datetime64[ns]
EVT_timing             datetime64[ns]
Consciousness          datetime64[ns]
> pd.to_datetime(pd.Series([pd.np.nan, pd.np.nan, pd.np.nan, pd.np.nan]))
Out[3]: 
0   NaT
1   NaT
2   NaT
3   NaT
dtype: datetime64[ns]

So here are the issues I see:

  1. parse_dates is too aggressive and sometimes converts things to datetime when not desired. SOLUTION - I added an if-statement to check if the whole column is NaN and skip it in that case
  2. StringDtype doesn’t get parsed into a date SOLUTION - Check for this as well as object
  3. Any logic for identifying if a column should be a date will not be perfect SOLUTION - I’m removing parse_dates from the automatic DataFrame cleaning on load, and instead adding it to the Edit menu as an optional action. I’ve also added this option to the context menu for column actions that pops up on right clicking headers
Read more comments on GitHub >

github_iconTop Results From Across the Web

pandas.to_datetime — pandas 1.5.2 documentation
This function converts a scalar, array-like, Series or DataFrame /dict-like to a pandas datetime object. Parameters. argint, float, str, datetime, list, tuple, ...
Read more >
Skipping non-date time values when converting to datetime in ...
I have a dataframe where one of the columns has data that I want to turn into datetime. But there are ...
Read more >
Easily Converting Strings to Times and Dates in R with flipTime
Need to convert strings or characters to time and dates in R? I'll discuss some useful packages, the common pitfalls and give some...
Read more >
Format Dates in String Array - Help - UiPath Community Forum
I need all my dates in a array even after conversion. ... always use a ForEach loop with an assign inside, but my...
Read more >
10 Tricks for Converting Numbers and Strings to Datetime in ...
Often, you'll find that dates are represented as strings. In Pandas, strings are shown as object, it's the internal Pandas lingo for the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found