Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Py4JJavaError: An error occurred while calling o1446.filter.

See original GitHub issue

Hi, I have a proc_cnt which koalas df with column count, (coding in databricks cluster)

thres = 50
drop_list = list(set(proc_cnt.query('count >= @thres').index))
ks_df_drop = ks_df[ks_df.product_id.isin(drop_list)]

My query function throws ParseException

org.apache.spark.sql.catalyst.parser.ParseException: 
Py4JJavaError                             Traceback (most recent call last)
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:

Py4JJavaError: An error occurred while calling o1446.filter.
: org.apache.spark.sql.catalyst.parser.ParseException: 
extraneous input '@' expecting {'(', 'SELECT', 'FROM...………...

May I know the issue Thanks

Issue Analytics

State:
Created 3 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

2reactions

HyukjinKwoncommented, Apr 9, 2020

Can you check the documentation of https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.query.html? We don’t currently support @ properly. There is a workaround documented there as well.

1reaction

HyukjinKwoncommented, Apr 14, 2020

Oh, you mean the Python built-in function set, right? It wouldn’t work out of the box. You might want to check https://github.com/databricks/koalas/pull/1420

Top Results From Across the Web

Py4JJavaError: An error occurred while calling - Stack Overflow

My code is only doing some filtering and joins. I keep getting errors regarding py4J. Any help would be useful, and appreciated. from...

How To Fix - "Py4JJavaError: An Error Occurred While Calling ...

Make sure data fields or columns are correctly attributed with null or not-null property as appropriate for the use case. Check null values....

PySpark python issue: Py4JJavaError: An error occurred while ...

Hi,. the "An error occurred while calling o65.sql" message is only a symptom, not a cause, and the true error lies somewhere above...

Py4JJavaError: An error occurred while calling - Microsoft Q&A

Py4JJavaError: An error occurred while calling. I am running notebook which works when called separately from a databricks cluster.

An error occurred while calling o5082.csv. : org.apache.spark ...

How to solve Py4JJavaError: An error occurred while calling o5082.csv. : org.apache.spark.SparkException: Job aborted. when writing to csv. Screenshot 2022-03 ...