convert() and fieldmap() swallow errors silently by default

This has led to a lot of difficult-to-debug scenarios in my experience. As you’re developing an ETL pipeline, you’re often iterating on your conversion functions quickly, which often leads to exceptions being raised. Currently petl eats those errors silently, making it far more difficult to navigate this process. In the regular operation of an ETL pipeline, it’s also undesirable for new, unexpected errors to pass silently.

The current workaround is to set failonerror=True in every call to fieldmap() and the like. I suggest you make failonerror=True be the default. This might be something that would require bumping to version 2.0 since it will likely cause a lot of exceptions to be raised that were previously ignored (for better or for worse!).

[alimanfoo]

Apologies, I closed this issue too hastily. Personally I’d be happy to change the default to failonerror=True but that would need a major version bump. I need to make a quick release for a downstream project, and so was taking the opportunity to merge in some PRs, but didn’t want to introduce any changes that might surprise people.

How about we do a quick release 1.3 which includes the new petl.config.failonerror config option, set to False by default (current behaviour). Then follow up with a 2.0 release shortly after which switches the default to True, but precede that with a message on the mailing list to see if anyone has any objections to the change.

[bmaggard]

If failonerror=False, petl doesn’t eat errors silently, it returns ‘errorvalue’. So, how about also adding a ‘returnexceptions’ option, akin to:

In [1]: import petl as etl

In [2]: t1 = etl.wrap([ ['foo'], ['A'], [1] ]).convert('foo', 'lower', errorvalue=Exception)

In [3]: t1
Out[3]:
+---------------------+
| foo                 |
+=====================+
| 'a'                 |
+---------------------+
| <class 'Exception'> |
+---------------------+

… but to have the original exceptions returned instead? This would enable categorizing failures directly.