[FEATURE REQUEST]: UDF to return custom business objects

I have a C# component which is presently running in Azure Data lake and i am planning to move to Spark and reuse the same component. My example scenario

C# takes an input of Manager Dataset like

mgrId	name
11	ABC
22	DEF

C# component returns a List of Reportee, where Reportee is Defined as Class { public int EmpId; public string Name; public string Role; public int MgrId; }

Reportee dataset

empId	name	role	mgrId
100	pqr	admin	11
200	stu	reader	11
300	wxy	reader	22

intended UDF var udf = Udf<int, List<Reportee>>((mgrId) => return component.Execute(mgrId); });

for each row in my Manager dataset, i have to call UDF to get final result in spark as

mgrId	mgrname	empname	empid	Role
11	ABC	pqr	100	admin
11	ABC	stu	200	reader
22	DEF	wxy	300	reader

[imback82]

The UDT (user-defined type) as a return type of UDF will not be supported. (UDT API in Spark became private since 2.0, and not much traction in PR, etc.)

However, we plan to achieve something similar using StructType. This is how it’s done in PySpark.

I did a quick prototype, and it looks like the following:

var schema = new StructType(new[] {
    new StructField("col1", new IntegerType()),
    new StructField("col2", new StringType()) });
// The schema is hard-coded inside Udf<> for POC, but "Row" class will be used to embed
// schema and object[].
var udf = Udf<string, object[]>((str) => new object[] { 1, "abc" });

// Assume that we have a df that has:
//+----+-------+
// | age|   name|
// +----+-------+
// |null|Michael|
// |  30|   Andy|
// |  19| Justin|
// +----+-------+
var udfDf = df.Select(udf(df["name"]).As("udf_col"));

// PrintSchema() prints:
// root
// |-- udf_col: struct (nullable = true)
// |    |-- col1: integer (nullable = false)
// |    |-- col2: string (nullable = true)
udfDf.PrintSchema();

// Show() prints:
// +--------+
// | udf_col|
// +--------+
// |[1, abc]|
// |[1, abc]|
// |[1, abc]|
// +--------+
udfDf.Show();

// Flatten nested column as follows:
// +----+----+
// |col1|col2|
// +----+----+
// |   1| abc|
// |   1| abc|
// |   1| abc|
// +----+----+
udfDf.Select(udfDf["udf_col.col1"], udfDf["udf_col.col2"]).Show();
// or
udfDf.Select("udf_col.*").Show();

This feature be available in coming weeks.

[imback82]

Why not something like Udf_SomeOtherName<T>(T t, Schema s) for UDFs that return Row? It’s awkward to allow schema for almost all other types that don’t even need it.

So we have two options:

1. Have a different name for UDF that returns Row type. The API is clear, but the downside is we need a separate set; you cannot specialize on generic types in C#.
2. Allow specifying schema for UDF, but we need to make sure there is no confusion on Row and GenericRow, especially if you are going from an object with less info to more info (GenericRow -> Row, how can it logically be possible?)

We can explore both options and get some early feedback.