[FEATURE REQ] Add sdk support for DataFactoryExpression

Library name

This applies to management plane and dataplane DataFactory and Synapse libraries.

Please describe the feature.

Summary

In the datafactory API many of the properties have the ability to either be a constant value or an expression which will be evaluated at runtime. For more information on expressions go here. The structure of an expression is different that a constant value for example the FolderPath property of an AzureBlobDataset can either be a “string (or Expression with resultType string)”.

Json representation

Constant

"folderPath": "foo/bar"

Expression

"folderpath": {
  "type": "Expression",
  "value": "foo/bar-@{ pipeline().TriggerTime}"
}

In this example when the pipeline is run in the first case the folder is always foo/bar, but in the second case the service will append the time the pipeline kicked off to the folder name.

Because the property can have multiple types the only real way to express this in openapi is to set it to an object which gets generated as object for dotnet dataplane and BinaryData for mgmtplane. This makes it difficult for customers to know how they are supposed to construct these.

If modeled as object

Constant

  azureBlobDataset.FolderPath = "foo/bar";

Expression

  Dictionary<string, object> properties = new Dictionary<string, object>();
  properties.Add("type", "Expression");
  properties.Add("value", "foo/bar-@{ pipeline().TriggerTime}");
  azureBlobDataset.FolderPath = properties;

If modeled as BinaryData

Constant

  azureBlobDataset.FolderPath = BinaryData.FromObjectAsJson("foo/bar");

Expression

  azureBlobDataset.FolderPath = BinaryData.FromObjectAsJson(new { type = "Expression", value = "foo/bar-@{ pipeline().TriggerTime}"});

Solution

The main challenge from a client side is that we need an object that can dynamically serialize and deserialize a primitive or an object based on its current value. We also should restrict the primitive type in strongly typed languages that can be assigned to the property since because its an object I can easily assign an integer to a field that expects a string.

To solve for this we should introduce a new type DataFactoryExpression<T> which defines which type can be assigned as well as which type the expression must evaluate to aligning with the documentation of the API but moving it from documentation to actual API shape.

This would allow us to use the exact same code whether we are assigning a constant or an expression

If modeled as DataFactoryExpressoin<T>

Constant

  azureBlobDataset.FolderPath = new DataFactoryExpression<string>("foo/bar");

Expresion

  azureBlobDataset.FolderPath = new DataFactoryExpression<string>("foo/bar-@{ pipeline().TriggerTime}");

There a lot of considerations on the API shape of DataFactoryExpression<T> that need to be worked out, but conceptually this would allow us to provide a much more user friendly SDK for libraries like DataFactory and Synapse who use these expression objects.

[m-nash]

This would be for all properties that can be a primitive or expression. There might be other properties that are untyped schema that wouldn’t fall into this category.

The only way to achieve the syntax above in strongly typed languages would be to add an implicit cast

public static implicit operator DataFactoryExpression<T>(T value) => new DataFactoryExpression<T>(value);

For dotnet I don’t think this violates any of the recommendations.

The only odd behavior would be for DataFactoryExpression<string> since both the expression and the T are the same type so it might look like you can assign either an expression or a value via the implicit. This matters because the internal storage on the class needs to be set up correctly in order for serialization to know the difference. We might have to check the incoming value to see if it has an @ and if so treat it as an expression otherwise treat it as a value. I don’t think any of the other T types have this issue.

[m-nash]

yes I meant to say its set to any which allows primitives and objects