[firestore-bigquery-export] gen-schema-view script - Duplicate entries created when using array types in schema

• Extension name: firestore-bigquery-export
• Extension version: 0.1.16

Amazing extension, this is exactly what I was looking for.

I’m using the gen-schema-view script with and it works great as long as I don’t have array field types. When I do, it creates duplicate entries in the BigQuery table.

This schema create 116 entries per every firestore document:

{
  "fields": [
    {
      "name": "field1",
      "type": "string"
    },
    {
      "name": "field2",
      "type": "number"
    },
    {
      "name": "field3",
      "type": "array"
    },
    {
      "name": "field4",
      "type": "array"
    },
    {
      "name": "field5",
      "type": "map",
      "fields": [
        {
          "name": "field5nested1",
          "type": "string"
        },
        {
          "name": "field5nested2",
          "type": "boolean"
        },
        {
          "name": "field5nested3",
          "type": "array"
        },
        {
          "name": "field5nested4",
          "type": "array"
        }
      ]
    }
  ]
}

This one doesn’t

{
  "fields": [
    {
      "name": "field1",
      "type": "string"
    },
    {
      "name": "field2",
      "type": "number"
    },
    {
      "name": "field5",
      "type": "map",
      "fields": [
        {
          "name": "field5nested1",
          "type": "string"
        },
        {
          "name": "field5nested2",
          "type": "boolean"
        }
      ]
    }
  ]
}

I should add that field3 and field4 are array of firestore references, and field5nested3, field5nested4 are array of maps.

[dackers86]

Base on that schema I have produced case with the following results…

  await ref.set({
    field1: "test",
    field2: 1000,
    // field3: [ref1, ref2],
    // field4: [ref1, ref2],
    field5: {
      field5nested1: "field5nested1_example",
      field5nested2: true,
      field5nested3: [
        { field5nested3_array_1: "field5nested3__array1__example" },
        { field5nested3_array_2: "field5nested3__array2__example" },
        { field5nested3_array_3: "field5nested3__array3__example" },
      ],
      field5nested4: [
        { field5nested4_array_1: "field5nested4_array1__example" },
        { field5nested4_array_2: "field5nested4_array2__example" },
        { field5nested4_array_3: "field5nested4_array3__example" },
      ],
    },
  });

![image](https://user-images.githubusercontent.com/2060661/142433516-0d1c51c7-286a-424e-965a-c4a24b01a009.png)

I suppose the confusion is that this returns n amount of rows from a complex structure into a flat version of the data.

Does this adequately replicate the data that you are seeing from your own database?

[alexcastillo]

Yes, it’s good to know that’s the intended behavior. Thank you!