Temporal Tables : add support for set operations

Currently set operations are not supported when arguments are mapped to temporal tables. This is due to ambiguity:

ctx.Customers.TemporalAsOf(new DateTime(2000, 1, 1)).Concat(ctx.Customers.TemporalAsOf(new DateTime(2010, 1, 1))).Select(x => x.Orders)

should orders come from the first source or the second? However, if the query only contains set operation (i.e. no navigation afterwards), we should be able to translate it correctly. However currently we don’t retain the information about the sources of set operation, so at the time we encounter the navigation in Select, we don’t know if the sources were temporal or not (and therefore can’t block the scenario properly at that point).

As a result we block the scenario at the time of set operation itself, which we should fix. However this requires some refactoring in nav expansion.

[smitpatel]

Set operations are - Union. Intersection, Except, Concat. Remember Concat doesn’t remove duplicate and Intersection/Except can have their own non-duplicating method in database though it doesn’t exist in BCL LINQ. No other operation is defined as set operation. Join is not a set operation.

Any navigation configured in model will work as long as it is in a query which is distinct point in time and target of navigation is also temporal table. The target entity will automatically fetch from the same point in time so user don’t need to write it (hence AsOf operation is on DbSet and cannot be put on navigation. This restriction comes because navigations in EF Core are connected by referential integrity constraints and they are only valid for point in time and not across the time.

The very first query posted in original issue is about combination of above 2 as a set operation can combine 2 different point in time data into one. Trying to expand navigation afterwards doesn’t work. The issue tracks what to do with such scenario and we are most likely to throw error for that. Users can also apply the Select before performing set operation and that should work (if it doesn’t work rn, we will enable it in future)

With all above, is about using navigations configured in EF Core in query since we convert them to appropriate LINQ structure. But that doesn’t restrict users to write any query the way they want without using navigations and performing manual joins as needed. The first query is last comment is how you write it and it should generate similar SQL. LINQ is infinite problem space, we have laid out what are current restrictions with the system, so anything outside of it should be considered valid and should work so best thing to do is write different query and see what SQL/result they generate and it is expected. If you run into any bugs where EF Core throws exception without a message or you believe error message doesn’t clearly say what is wrong in the query then file a new bug with detailed repro steps.

I have mentioned above what this ticket represent and it is not relating to question you are referring to, so we are talking unrelated things here. If you have further questions please utilize https://github.com/dotnet/EntityFramework.Docs/issues/3353 @maumar filed or file a new issue with detailed information of what query you wrote and what results EF Core generated.

[jzabroski]

normal navigations work fine as long as you use TemporalAsOf operation, rather than TemporalAll, or any of the range ops.

So, how do I write the following query in EF?

-- Get name changes
SELECT DISTINCT c.CustomerId
FROM dbo.Customer FOR SYSTEM_TIME CONTAINED IN (@StartDate, @EndDate) ch
  JOIN dbo.Customer c ON ch.CustomerD = c.CustomerID
WHERE ISNULL(c.Name, '') <> ISNULL(ch.Name, '');

So I write:

var c = ctx.Customers;
var ch = ctx.Customers.TemporalContainedIn(startDate, endDate);
var customersWithNameChanges = (from history in ch
  join current in c
    on history.CustomerId equals current.CustomerId
  where history.Name ?? string.Empty != current.Name ?? string.Empty
  select c.CustomerId)
  .ToList();

Will that work? Note: This is just a dumbed down example of a broad class of problems I described in the original issue: “Queries to determine what fields have changed”, of which the Net Changes problem is the most general (I did not expect EFCore to actually tackle Net Changes, but I posed it as a gambit to consider an architecture that would allow sufficiently writing C# extension methods to solve the problem in multiple steps). The abstract version of the above query is:

context.Entity1
.TemporalTableContainedIn(startDate, endDate, typeof(Entity1))
.Join(context.Entity1, history => history.Id, current => current.Id, (history, current) => new
{
    Item = current,
    OldValue = history.Field1,
    NewValue = current.Field1
})
.Where(x => x.OldValue != x.NewValue)
.Select(x => x.Item)

By set operations I mean Concat, Union, Except etc - operations that take 2 queryables and return one queryable.

Is Join a set operation?

Also, I’m not sure I fully understood this disclosure:

normal navigations work fine as long as you use TemporalAsOf operation, rather than TemporalAll, or any of the range ops.

Are you saying normal navigations only work for TemporalAsOf, because it’s a distinct point in time, and any query that returns a time series is not supported? In which case I would infer that the Net Changes problem I posed above is unsolvable, which is disappointing. Does this ticket then address that?