[QUERY] How do we query the TOP record in Azure.Data.Tables

Library name and version

Azure.Data.Tables 12.6.1

Query/Question

With Microsoft.WindowsAzure.Storage.Table, we used to do as below:

var query = new TableQuery<TEntity>()
            .Where(partitionFilter)
            .Take(limit);
return await ExecuteQuery(table, query);

How do we achieve the same using Azure.Data.Tables. For now, we are getting all the data and then doing top in-memory but that seems to be problematic as the data grows.

Environment

Microsoft Visual Studio Enterprise 2022 (64-bit) - Current Version 17.2.6

[fhtino]

Hi @christothes, regarding the documentation, you linked a different API. However the point is the same: “maxPerPage”. In your case “tables”, in my case “entities”. In my opinion the documentation is clear and also the parameter name “maxPerPage” is correct. Instead this issue discuss the lack of a “TopN” functionality. It would be very usesul to have a Query<T>(filter, . . . . . , topN)

My local implementation (wrapper), that cover my needs is:

public List<T> RunQuery<T>(Expression<Func<T, bool>> filter, int? topN = null) where T : class, ITableEntity, new()
{
    // Currently Azure.Data.Tables does not implement "TopN"
    // https://github.com/Azure/azure-sdk-for-net/issues/30985

    if (filter == null)
    {
        // this generates a query without a "where" condition
        filter = item => true;
    }

    TableClient tableClient = this.GetAzureTableClient<T>();

    // Remember: every underlying REST API call returns 1000 items (maximum).
    // Giving an explicit maxPerPages is not strictly required.
    // But if TopN < 1000, it would be a waste of resource retrieving more items than required.            
    int? maxPerPages = (topN.HasValue && topN < 1000) ? topN : null;

    var query = tableClient.Query<T>(filter, maxPerPages);

    var outputBuffer = new List<T>();

    foreach (var page in query.AsPages())
    {
        outputBuffer.AddRange(page.Values);

        if (topN.HasValue && outputBuffer.Count >= topN.Value)
            break;
    }

    if (topN.HasValue)
        outputBuffer = outputBuffer.Take(topN.Value).ToList();

    return outputBuffer;
}

[fhtino]

@christothes If I remember correctly, the old WindowsAzure.Storage.Table Take only queried and returned the requested number of elements. With the new Azure.Data.Tables if I specify maxPerPages it only limits the number of elements of each rest api call. E.g. If I have a table with 1 million items and I specify 50, I will get all the elements, running 20.000 api call to azure table rest api.

If I really want to fetch only required data, I must do something like this:

            int? maxPerPages = (topN.HasValue && topN < 1000) ? topN : null;
            var result = tableClient.Query<T>(filter, maxPerPages);
            var buffer = new List<T>();
            foreach (var page in result.AsPages())
            {
                buffer.AddRange(page.Values);
                if (topN.HasValue && buffer.Count >= topN.Value)
                    break;
            }
            if (topN.HasValue)
                buffer = buffer.Take(topN.Value).ToList();
            return buffer;

I think the library should implement a “TopN” feature. Many users are not aware of the behavior of the underlying rest api.