Getting TypeError while parsing a dataframe

I’m facing TypeError while using pandas-profiling on a data as follows:

Summarize dataset: 2% 1/59 [00:09<09:20, 9.66s/it, Describe variable:project] -->>
.....
.......
~\Anaconda3\lib\site-packages\pandas\core\algorithms.py in isin(comps, values)
    441         # If the the values include nan we need to check for nan explicitly
    442         # since np.nan it not equal to np.nan
--> 443         if np.isnan(values).any():
    444             f = lambda c, v: np.logical_or(np.in1d(c, v), np.isnan(c))
    445         else:
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Please help. I have no idea what this is about.

[vperiyasamy]

I checked out the pandas source and found the root cause: Here is where that 1 million threshold is coming from, and in the version of pandas I’m using (1.1.3) checks this with np.isnan instead of np.isna; as the OP mentioned above, np.isna is the more robust check. pandas==1.1.4+ includes this fix and resolves the issue for me.

[vperiyasamy]

Here’s a snippet of what I’m using (Spark but converted to pandas):

from pandas_profiling import ProfileReport

spark_df = spark.range(10000000).selectExpr("id AS f1", "CONCAT('string id: ', id) AS f2", "'constant string' AS f3", "(id%3) == 0 AS f4", "(id+1) % 2 AS label")
df = spark_df.toPandas()
df_small = df[:1000001]

df_profile = ProfileReport(df_small, interactions={"continuous": False}, title="Profiling Report", progress_bar=False)
profile_html = df_profile.to_html()