`elt` pipeline is exiting with an error code of `-9` and a blank error message

I’ve recently started seeing our pipeline fail with the following output:

2022-06-30T17:09:39.480320Z [error ] Extraction failed code=-9 message=

The command I’m running is: meltano elt tap-postgres-zenapi target-snowflake-zb --state-id=sync-zenapi

Here is a concatenated version of my meltano.yml that contains the config for the offending tap/target pipeline:

plugins:
  extractors:
  - name: tap-postgres-zenapi
    inherit_from: tap-postgres
    config:
      filter_schemas: zb
      itersize: 100000
      default_replication_method: INCREMENTAL
    select:
      - zb-*
    metadata:
      zb-*:
  loaders:
  - name: target-snowflake-zb
    inherit_from: target-snowflake
    variant: transferwise
    pip_url: pipelinewise-target-snowflake
    config:
      add_metadata_columns: true
      batch_size_rows: 100000
      primary_key_required: false
      default_schema: SOURCE

Some further context:

This pipeline is running in a ephemeral Kubernetes pod orchestrated by Airflow, created from an image with Meltano v2.0.1 as the base image found in Dockerhub.

[tayloramurphy]

@cbarnes-zb thanks for the update! I’m okay to close - if you have further suggestions on how we can improve error messages or state management, I’d love another issue on that. Thanks!

[cbarnes-zb]

@aaronsteers I was finally able to track this down last week, or at least what I think is the root cause. We moved state management from the system db to a postgres instance recently as part of ongoing infrastructure changes and updates and I think that somewhere along the way, as careful as we were, the state became corrupted. The payload field contained escaped quotes in a few places. I’m not sure exactly how this happened, but falling back on the payload from a successful run eliminated the error references by this issue.

I’ll leave this open for now so I don’t step on any closing procedures, but I thought I’d let you know this error has been eliminated from our pipe.