Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Can't make incremental work with tap-mongodb

See original GitHub issue

Hi,

Trying to do the following

enviroment variables

MELTANO_DATABASE_URI=...

meltano elt tap-mongodb target-postgres --transform=skip --state-id=mongo-to-postgres --catalog=catalog-pd.json

looking into the tap’s code it seems it not loading the state

it’s like always start from scratch

2022-08-13T01:17:23.793367Z [info     ] Environment 'dev' is active
2022-08-13T01:17:26.327980Z [info     ] Running extract & load...      name=meltano run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres
2022-08-13T01:17:26.590014Z [info     ] Found catalog in /project/catalog-pd.json
2022-08-13T01:17:27.588487Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Connected to MongoDB host: db-mongodb-nyc3-50445-c65cd427.mongo.ondigitalocean.com, version: 5.0.8 cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.603508Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Starting incremental sync for go-admin-groups cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.606004Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Querying go-admin-groups with: {'find': {}} cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.621951Z [info     ] time=2022-08-13 01:17:27 name=target_postgres level=INFO message=Table '"groups"' exists cmd_type=loader name=target-postgres run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.675500Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Syncd 1 records for go-admin-groups cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.680784Z [info     ] time=2022-08-13 01:17:27 name=singer level=INFO message=METRIC: {"type": "timer", "metric": "job_duration", "value": 0.06659770011901855, "tags": {"job_type": "sync_table", "database": "go-admin", "table": "groups", "status": "succeeded"}} cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.684226Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Starting incremental sync for go-admin-users cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.685218Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Querying go-admin-users with: {'find': {}} cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.701308Z [info     ] time=2022-08-13 01:17:27 name=target_postgres level=INFO message=Table '"users"' exists cmd_type=loader name=target-postgres run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.704995Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Syncd 26 records for go-admin-users cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.709079Z [info     ] time=2022-08-13 01:17:27 name=singer level=INFO message=METRIC: {"type": "timer", "metric": "job_duration", "value": 0.03786444664001465, "tags": {"job_type": "sync_table", "database": "go-admin", "table": "users", "status": "succeeded"}} cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.716568Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Starting incremental sync for go-admin-products cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.718235Z [info     ] time=2022-08-13 01:17:27 name=tap_mongodb level=INFO message=Querying go-admin-products with: {'find': {}} cmd_type=extractor name=tap-mongodb run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr
2022-08-13T01:17:27.761226Z [info     ] time=2022-08-13 01:17:27 name=target_postgres level=INFO message=Table '"products"' exists cmd_type=loader name=target-postgres run_id=765f5b21-a10b-46ef-b139-c59fc2bb29f9 state_id=mongo-to-postgres stdio=stderr

it’s doing the find {} always and not passing the date.

in catalog i have the replication defined

{
  "streams": [
    {
      "table_name": "groups",
      "stream": "groups",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": [
              "_id"
            ],
            "database-name": "go-admin",
            "row-count": 1,
            "is-view": false,
            "valid-replication-keys": [
              "_id"
            ],
            "selected": true,
            "replication-method": "INCREMENTAL",
            "replication-key": "updatedAt"
          }
        }
      ]

https://github.com/transferwise/pipelinewise-tap-mongodb/blob/master/tap_mongodb/sync_strategies/incremental.py#L91

any help will be very appreciated