Cannot send lineage data

Have a very simple test that creates and materializes 1 dataframe using the following SQL query : select 123 as id

I get the following error

15:25:49 [ScalaTest-run-running-LineageTest] WARN  org.apache.spark.sql.util.ExecutionListenerManager Error executing query execution listener
java.lang.RuntimeException: Cannot send lineage data to http://localhost:8080/producer/execution-plans
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher.sendJson(HttpLineageDispatcher.scala:57)

rest gateway is returning :

{
    "error": "JSON parse error: ; nested exception is com.twitter.finatra.json.internal.caseclass.exceptions.CaseClassMappingException: \nErrors:\t\tcom.twitter.finatra.json.internal.caseclass.exceptions.CaseClassValidationException: operations.other.childIds: field is required\n\n"
}

Execution plan sent to the rest gateway by spline agent has indeed a missing childIds field for operations.other with id=3

{
    "id": "070e41dd-7d70-422e-a8f7-305ab4fcdd92",
    "operations": {
        "write": {
            "outputSource": "file:/c:/tmp/instrument",
            "append": false,
            "id": 0,
            "childIds": [
                1
            ],
            "params": {
                "path": "c:\\tmp\\instrument"
            },
            "extra": {
                "name": "InsertIntoHadoopFsRelationCommand",
                "destinationType": "Parquet"
            }
        },
        "other": [
            {
                "id": 3,
                "extra": {
                    "name": "OneRowRelation"
                }
            },
            {
                "id": 2,
                "childIds": [
                    3
                ],
                "schema": [
                    "ae29dfe3-39fe-4271-8482-f6826b2c00b5"
                ],
                "params": {
                    "projectList": [
                        {
                            "_typeHint": "expr.Alias",
                            "alias": "id",
                            "child": {
                                "_typeHint": "expr.Literal",
                                "value": 123,
                                "dataTypeId": "129f2969-214f-43dd-8f13-ebf285c6cb5f"
                            }
                        }
                    ]
                },
                "extra": {
                    "name": "Project"
                }
            },
            {
                "id": 1,
                "childIds": [
                    2
                ],
                "schema": [
                    "1e8bf5b0-1863-4674-b802-f6c238a5cf90"
                ],
                "params": {
                    "name": "`instrument`"
                },
                "extra": {
                    "name": "SubqueryAlias"
                }
            }
        ]
    },
    "systemInfo": {
        "name": "spark",
        "version": "2.4.1"
    },
    "agentInfo": {
        "name": "spline",
        "version": "0.4.0"
    },
    "extraInfo": {
        "appName": "runner_test",
        "dataTypes": [
            {
                "_typeHint": "dt.Simple",
                "id": "129f2969-214f-43dd-8f13-ebf285c6cb5f",
                "name": "integer",
                "nullable": false
            }
        ],
        "attributes": [
            {
                "id": "ae29dfe3-39fe-4271-8482-f6826b2c00b5",
                "name": "id",
                "dataTypeId": "129f2969-214f-43dd-8f13-ebf285c6cb5f"
            },
            {
                "id": "1e8bf5b0-1863-4674-b802-f6c238a5cf90",
                "name": "id",
                "dataTypeId": "129f2969-214f-43dd-8f13-ebf285c6cb5f"
            }
        ]
    }
}

Adding an empty array childIds field and sending the same json with an API testing tool works fine.

[wajda]

@shubhluck, from your logs it looks like you are using an old snapshot Spline version. The config parameter spline.server.rest_endpoint doesn’t exist in Spline 0.4.0. Please make sure you are running a correct version.

[wajda]

Our tests passed. Closing the issue. Please open another one if the error persists.