Translate SkipWhile and TakeWhile

Feature Request

In order to facilitate my use case of implementing cursor based pagination I would find the LINQ .SkipWhile() method useful.

I would imagine this expression to result in something like the following SQL statement:

(DbSet<T>)
    .OrderBy(q => q.DepartureTime)
    .ThenBy(q => q.ArrivalTime)
    .SkipWhile(q => q.Id != "84CFC944-B64E-4C7D-80CD-053C771DC478")
    .Take(50);

SELECT * FROM [Flights] ORDER BY [DepartureTime], [ArrivalTime]
OFFSET (
    SELECT TOP 1 rn
    FROM (
        SELECT Id, ROW_NUMBER() OVER (ORDER BY [DepartureTime], [ArrivalTime]) as rn FROM [Flights]
    ) T
    WHERE T.Id = '84CFC944-B64E-4C7D-80CD-053C771DC478'
) rows
FETCH NEXT 50 ROWS ONLY

Until now I have been unable to find a reusable and composable way to achieve this behaviour by using the FromSql() extension method for multiple models.

Related Issues

https://github.com/aspnet/EntityFrameworkCore/issues/12046

[d00ML0rDz]

Just wondering if there is any update to this or any idea when we may see this implemented in EF Core?

[roji]

I’ve recently done some research into pagination, and I don’t think SkipWhile is the right building block for an efficient “cursor-oriented” pagination mechanism. In a nutshell, offset or rownumber-based approaches do not use indexes, making them inherently inefficient. In addition, in many cases these approaches also lack robustness in the face of concurrent updates, as rows are skipped or duplicated (although the approach detailed above doesn’t suffer from this, since the ID is used as the cursor, rather than a raw offset).

For efficient cursor pagination, simple keyset pagination seems to be the best fit. In other words, assuming we’re ording by an Id, pagination would simply use a WHERE clause to get rows after the last Id fetched from the previous page; compared to the above approach of an OFFSET subquery with ROW_NUMBER, this is simple and uses indexes. If more than one pagination key is being used, row value comparison can be used WHERE (Date, Id) > (@date, @id) (see #26822), or the expanded version of it (WHERE Date > @date OR Date = @date AND Id > @id).

Assuming we’ve excluded the use of SkipWhile for pagination purposes, there’s still the question of non-pagination uses. There are some characteristics of relational databases which limit the usefulness of SkipWhile:

1. Rows in relational database have no ordering. Unless ORDER BY is specified, ordering is non-deterministic.
2. Doing SkipWhile over a non-deterministic set doesn’t seem to have any value. Doing Take over unordered results can be useful to retrieve any matching X rows, but it’s not clear why someone would want to do SkipWhile over a non-deterministic set.
3. For SkipWhile to be deterministic, it must be done after ordering by all columns specified in it (e.g. .OrderBy(b => b.CreationDate).SkipWhile(b => b.CreationDate < @date)). The effect of this is very similar to Where, with the difference being that Where would eliminate all rows not matching the filter, whereas SkipWhile eliminates only the rows at the beginning.
4. However, since the rows are already sorted, in the above example Where is guaranteed to return the same results as SkipWhile. It’s possible to come up with scenarios where SkipWhile has different behavior from Where, but these usually seem contrived.

So I’m leaving this issue open in the backlog to track translating SkipWhile and TakeWhile for non-pagination purposes, given non-contrived real-world scenarios which would benefit from it.