Handling non-pickleable exceptions

Not sure if you would agree that this is a bug rather than unsupported behavior.

Google libraries tend to include non-pickleable Client objects in the exceptions they raise (who knows why). So the following flow run locally gives a nice clear error:

from google.cloud import bigquery

@task
def bq_fail():
    bq = bigquery.Client()
    bq.query('select 1 from nonexistent_table').result()
    return True

with Flow('f') as f:
    bq_fail()

f.run()

...
google.api_core.exceptions.BadRequest: 400 Table name "nonexistent_table" missing dataset while no default dataset is set in the request.

(job ID: 69b11ec8-312b-49c4-91c9-aae701b7b8cf)

  -----Query Job SQL Follows-----

    |    .    |    .    |    .    |
   1:select 1 from nonexistent_table
    |    .    |    .    |    .    |
[2020-03-20 18:21:39,312] INFO - prefect.TaskRunner | Task 'bq_fail': finished task run for task with final state: 'Failed'
INFO:prefect.TaskRunner:Task 'bq_fail': finished task run for task with final state: 'Failed'
[2020-03-20 18:21:39,313] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
INFO:prefect.FlowRunner:Flow run FAILED: some reference tasks failed.

but running on a Dask client gives a much crazier failure:

import distributed
from prefect.engine.executors import DaskExecutor
client = distributed.Client()
f.run(executor=DaskExecutor(client.scheduler.address))

...
Could not serialize object of type Failed.
Traceback (most recent call last):
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 38, in dumps
    result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
AttributeError: Can't pickle local object 'if_exception_type.<locals>.if_exception_type_predicate'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 191, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 58, in pickle_dumps
    return {"serializer": "pickle"}, [pickle.dumps(x)]
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 51, in dumps
    return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 1125, in dumps
    cp.dump(obj)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 482, in dump
    return Pickler.dump(self, obj)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/google/cloud/client.py", line 144, in __getstate__
    "Clients have non-trivial state that is local and unpickleable.",
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.

This is actually a very abbreviated version of the actual error: the real output contains like 10 variations of the above and sort of swamps all other output in the logs.

I understand that Google is the underlying culprit here, but it seems like Prefect could handle this case more gracefully, maybe by just catching a PickleError and re-raising as something more informative…?

[jcrist]

No progress yet - someone will comment here when things have changed.

[jcrist]

It’s not just the latest exception that is being stored, but rather some kind of exception chain that includes the previous boto exception. (I have no idea whether this is even a possibility)

Ah, good catch. Python supports chained exceptions, so the previous exception would leave stuff on the __cause__ and __context__ attributes of the raised MyBotoException. Can you try this one?

from functools import wraps

def fix_boto_exceptions(func):
    @wraps(func)
    def inner(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except BaseBotoException as exc:    # I don't know what this class is called, but you probably do
            raise MyBotoException(str(exc)) from None
    return inner

@task
@fix_boto_exceptions
def mytask(...):
    some_boto_thing()
    some_other_boto_thing()

Still not ideal, but I believe this should work for you.