Data misplaced when reading a table that does not have the same field positions as the spark schema

I am trying to create a dataset from bigquery table. The table has the same fields as case class but not in the same order. When creating the dataset, we get columns mapped to the wrong fields.

Given this table:

When loading dataset

case class NestedClass(
  int3: Int,
  int1: Int,
  int2: Int)

case class ClassA(str: String, l1: NestedClass)

 val schema = Encoders.product[ClassA].schema

    val ds2 = spark.read
      .schema(schema)
      .option("table", "customers_sale.test_table")
      .format("com.google.cloud.spark.bigquery")
      .load()
      .as[ClassA]

    ds2.map(_.l1).show(false)

NB: notice that NestedClass has the same fields as table but in different order: (int3, int1, int2) instead of (int1, int2, int3)

we got this:

+----+----+----+
|int1|int2|int3|
+----+----+----+
|2   |3   |1   |
+----+----+----+

We expect to get this

+----+----+----+
|int1|int2|int3|
+----+----+----+
|1   |2   |3   |
+----+----+----+

=> The connector does not use fields name to assign values but correlates field position in case class with the same position in table.

[davidrabinowitz]

Fixed by PR #391

[LaurentValdenaire]

We indeed have an issue with nested structs