Bug: Null config values don't get passed to plugins, preventing Mappers from working as expected

Trying to replicate the behavior here

https://sdk.meltano.com/en/latest/stream_maps.html

{
    "stream_maps": {
        "customers": {
            // exclude these since we're capturing them in the pii stream
            "email": null,
            "full_name": null
        },
        "customers_pii": {
            "__source__": "customers",
            // include just the PII and the customer_id
            "customer_id": "customer_id",
            "email": "email",
            "full_name": "full_name",
            // exclude anything not declared
            "__else__": null,
        },
    },
}

So if you have a meltano.yml like

config:
  stream_maps:
    customers_pii:
      "email": "email"
      __else__: null

And then run meltano invoke --dump=config tap-name the else token will not exist.

It’s not clear that this is a bug as I did some digging here and as a part of https://gitlab.com/meltano/meltano/-/issues/2334 (more specifically https://gitlab.com/meltano/meltano/-/merge_requests/1898 ) it looks like passing null values was removed as it was causing a different issue

UPDATE (AJ, 2022-08-30)

As noted below in https://github.com/meltano/meltano/issues/6382#issuecomment-1208353612, the bug is that Meltano excludes/removes config input values if they are null.

I can confirm this is still the behavior

visch@visch-ubuntu:~/git/tap-abc$ meltano --version
meltano, version 2.4.0

      config:
        stream_maps:
          customers_pii:
            "email": "email"
            __else__: null

visch@visch-ubuntu:~/git/tap-abc$ meltano invoke --dump=config tap-abc 
{
  "stream_maps": {
    "customers_pii": {
      "email": "email"
    }
 }

[pnadolny13]

@tayloramurphy logging here that we had another user (and myself too) get stuck on this bug in slack today https://meltano.slack.com/archives/C01TCRBBJD7/p1671134681823139.

From my view mappers and stream maps are a differentiating meltano feature that gives flexibility and allows users to hack around broken taps/targets, especially since we are constantly recommending it in slack. It feels like we’re very close to something super powerful if it was cleaned up and documented.

After skimming this issue I see that its a challenge to fix in meltano but it is really a meltano bug 😄 . It helps if its fixed in the SDK but all taps/targets need to be upgraded to get support back, even though it already works today and meltano’s bug causes the null values to disappear.

[visch]

Verified that this works with mappers but not taps (Really hard to just get the config for mappers but here we go)

meltano.yml

version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
project_id: 8339108a-ba30-48ae-9dcc-8c50702ff803
plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-csv.git
    config:
      stream_maps:
        customers_pii:
          email: email
          __else__:
      files:
        - entity: "in"
          path: "in.csv"
          keys: ["id"]
          delimiter: ","
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  mappers:
  - name: transform-field
    variant: transferwise
    pip_url: pipelinewise-transform-field
    executable: transform-field
    mappings:
    - name: pipedrive-exclude-custom-fields
      config:
        stream_maps:
          activities:
            id: id
            deal_title: deal_title
            __else__:
        transformations:
          - field_id: "commit"
            tap_stream_name: "commits"
            field_paths: ["author/email", "committer/email"]
            type: "HASH"

meltano run tap-csv pipedrive-exclude-custom-fields target-jsonl

(monkey patch in a sleep in target-jsonl)

}visch@visch-ubuntu:~/git/meltano-projects/test_null_key_with_tap/.meltano/run/transform-field$ ls
mapper.90aa6695-2412-4909-975c-f1901713d331.config.json
visch@visch-ubuntu:~/git/meltano-projects/test_null_key_with_tap/.meltano/run/transform-field$ cat mapper.90aa6695-2412-4909-975c-f1901713d331.config.json
{
  "stream_maps": {
    "activities": {
      "id": "id",
      "deal_title": "deal_title",
      "__else__": null
    }
  },
  "transformations": [
    {
      "field_id": "commit",
      "tap_stream_name": "commits",
      "field_paths": [
        "author/email",
        "committer/email"
      ],
      "type": "HASH"
    }
  ]