Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
OverflowError
See original GitHub issueUsing python 3.8, I am getting an OverflowError running apply_parallel:
OverflowError Traceback (most recent call last)
<ipython-input-5-a78fd5119887> in <module>
37 grouped_df = df.groupby("id")
38
---> 39 grouped_df.parallel_apply(lookahead) \
40 .to_parquet(output_location_look_ahead, compression='snappy', engine='pyarrow')
41
../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in closure(data, func, *args, **kwargs)
429 queue = manager.Queue()
430
--> 431 workers_args, chunk_lengths, input_files, output_files = get_workers_args(
432 use_memory_fs,
433 nb_requested_workers,
../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in get_workers_args(use_memory_fs, nb_workers, progress_bar, chunks, worker_meta_args, queue, func, args, kwargs)
284 raise OSError(msg)
285
--> 286 workers_args = [
287 (
288 input_file.name,
../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in <listcomp>(.0)
293 progress_bar == PROGRESS_IN_WORKER,
294 dill.dumps(
--> 295 progress_wrapper(
296 progress_bar >= PROGRESS_IN_FUNC, queue, index, chunk_length
297 )(func)
../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in wrapper(func)
203 def wrapper(func):
204 if progress_bar:
--> 205 wrapped_func = inline(
206 progress_pre_func,
207 func,
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
32 raise SystemError("Python version should be 3.{5, 6, 7, 8}")
33
---> 34 return function(*args, **kwargs)
35
36 return wrapper
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in inline(pre_func, func, pre_func_arguments)
485
486 func_instructions = tuple(get_instructions(func))
--> 487 shifted_func_instructions = shift_instructions(
488 func_instructions, len(b"".join(pinned_pre_func_instructions_without_return))
489 )
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
32 raise SystemError("Python version should be 3.{5, 6, 7, 8}")
33
---> 34 return function(*args, **kwargs)
35
36 return wrapper
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in shift_instructions(instructions, qty)
301 If Python version not in 3.{5, 6, 7}, a SystemError is raised.
302 """
--> 303 return tuple(
304 shift_instruction(instruction, qty)
305 if bytes((instruction[0],))
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in <genexpr>(.0)
302 """
303 return tuple(
--> 304 shift_instruction(instruction, qty)
305 if bytes((instruction[0],))
306 in (
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
32 raise SystemError("Python version should be 3.{5, 6, 7, 8}")
33
---> 34 return function(*args, **kwargs)
35
36 return wrapper
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in shift_instruction(instruction, qty)
291 """
292 operation, *values = instruction
--> 293 return bytes((operation,)) + int2python_bytes(python_ints2int(values) + qty)
294
295
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
32 raise SystemError("Python version should be 3.{5, 6, 7, 8}")
33
---> 34 return function(*args, **kwargs)
35
36 return wrapper
../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in int2python_bytes(item)
69
70 nb_bytes = 2 if python_version.minor == 5 else 1
---> 71 return int.to_bytes(item, nb_bytes, "little")
72
73
OverflowError: int too big to convert
Issue Analytics
- State:
- Created 3 years ago
- Comments:7 (2 by maintainers)
Top Results From Across the Web
Python OverflowError | Working of Overflow Error in ... - eduCBA
In Python, OverflowError occurs when any operations like arithmetic operations or any other variable storing any value above its limit then there occurs...
Read more >OverflowError: (34, 'Result too large') - python - Stack Overflow
Python floats are neither arbitary precision nor of unlimited size. When k = 349, 16.**k is much too large - that's almost 2^1400....
Read more >OverflowError Exception in Python - Pylenin
An OverflowError exception is raised when an arithmetic operation exceeds the limits to be represented. This is part of the ArithmeticError ...
Read more >Built-in Exceptions — Python 3.11.1 documentation
The base class for those built-in exceptions that are raised for various arithmetic errors: OverflowError , ZeroDivisionError , FloatingPointError .
Read more >Python Exception Handling - OverflowError - Airbrake Blog
Just like most other programming languages, the OverflowError in Python indicates that an arithmetic operation has exceeded the limits of the ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Me too! pandarallel 1.5.2 and Python 3.8.1 It occurs when
progress_bar=TrueAlso got the same error with Python 3.7.6
Same problem with python 3.7.5, only when
progress_bar=True