[BUG] Digital Twins: Service is returning duplicate results when paging

Library name and version

Azure.DigitalTwins.Core 1.2.2

Describe the bug

When using the pageSizeHint parameter of Pageable<T>.AsPages if a value which is lower than the expected result set is used (essentially forcing a page) then duplicate records are returned. I’m not sure that this is specifically the .NET SDK though as I’ve tried this against the API directly and the same issue occurs.

Expected behavior

Regardless of the max-items-per-page value, the full set of results should be returned without duplicates

Actual behavior

In an example where 22 results are expected from the supplied query, setting the pageSizeHint or the max-items-per-page header to a value such as 10 or as low as 2 results in only a handful of twins being returned, but repeatedly. The actual number of records returned is still 22, but may consist of only 5 unique records instead of the 22 unique records expected.

Reproduction Steps

The following is a quick .NET 6 console app I wrote to reproduce the issue (settings need changing to actual values obviously).

using System.Text.Json;
using Azure.DigitalTwins.Core;
using Azure.Identity;

namespace DigitalTwinTest;

internal class Program
{
    private static void Main()
    {
        var twinUrl = "https://my-twin-instance.api.reg.digitaltwins.azure.net";
        var clientId = "abc-123";
        var clientSecret = "super-secret-value";
        var tenantId = "abc-123-def-456";

        var credential = new ClientSecretCredential(tenantId, clientId, clientSecret);
        var client = new DigitalTwinsClient(new Uri(twinUrl), credential);

        var query = @"SELECT level FROM digitaltwins company JOIN building RELATED company.owns JOIN level RELATED building.isPartOf WHERE company.$dtId IN ['Comp1', 'Comp2'] AND IS_OF_MODEL(level, 'dtmi:digitaltwins:rec_3_3:core:Level;1')";

        var results = client.Query<BasicDigitalTwin>(query);

        foreach (var page in results.AsPages(pageSizeHint: 5))
        {
            foreach (var twin in page.Values)
            {
                Console.WriteLine(((JsonElement)twin.Contents["level"]).GetProperty("$dtId").GetString());
            }
        }
    }
}

The same is also repeatable using the same steps but through a tool such as postman to play multiple requests using the continuation token.

Environment

DotNet Info

.NET SDK (reflecting any global.json): Version: 6.0.301 Commit: 43f9b18481

Runtime Environment: OS Name: Windows OS Version: 10.0.22000 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.301\

Host (useful for support): Version: 6.0.6 Commit: 7cca709db2

.NET SDKs installed: 3.1.420 [C:\Program Files\dotnet\sdk] 5.0.214 [C:\Program Files\dotnet\sdk] 5.0.400 [C:\Program Files\dotnet\sdk] 5.0.403 [C:\Program Files\dotnet\sdk] 5.0.408 [C:\Program Files\dotnet\sdk] 6.0.100 [C:\Program Files\dotnet\sdk] 6.0.101 [C:\Program Files\dotnet\sdk] 6.0.102 [C:\Program Files\dotnet\sdk] 6.0.106 [C:\Program Files\dotnet\sdk] 6.0.202 [C:\Program Files\dotnet\sdk] 6.0.203 [C:\Program Files\dotnet\sdk] 6.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

IDE Version

Visual Studio Code

Version: 1.68.1 (user setup) Commit: 30d9c6cd9483b2cc586687151bcbcd635f373630 Date: 2022-06-14T12:48:58.283Z Electron: 17.4.7 Chromium: 98.0.4758.141 Node.js: 16.13.0 V8: 9.8.177.13-electron.0 OS: Windows_NT x64 10.0.22000

JetBrains Rider

Version: 2022.1.2 Build #RD-221.5787.36, built on June 1, 2022

[johngallardo]

@dazfuller - we have identified what we believe to be the root cause, and are working on mitigations at multiple places within our services. In the meantime, we believe this only impacts scenarios where max-page-size < 100.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @johngallardo, @efriesner, @abhinav-gha, @Aashish93-stack, @sjiherzig, @Satya-Kolluri.

Issue Details

---

Library name and version

Azure.DigitalTwins.Core 1.2.2

Describe the bug

When using the pageSizeHint parameter of Pageable<T>.AsPages if a value which is lower than the expected result set is used (essentially forcing a page) then duplicate records are returned. I’m not sure that this is specifically the .NET SDK though as I’ve tried this against the API directly and the same issue occurs.

Expected behavior

Regardless of the max-items-per-page value, the full set of results should be returned without duplicates

Actual behavior

In an example where 22 results are expected from the supplied query, setting the pageSizeHint or the max-items-per-page header to a value such as 10 or as low as 2 results in only a handful of twins being returned, but repeatedly. The actual number of records returned is still 22, but may consist of only 5 unique records instead of the 22 unique records expected.

Reproduction Steps

The following is a quick .NET 6 console app I wrote to reproduce the issue (settings need changing to actual values obviously).

using System.Text.Json;
using Azure.DigitalTwins.Core;
using Azure.Identity;

namespace DigitalTwinTest;

internal class Program
{
    private static void Main()
    {
        var twinUrl = "https://my-twin-instance.api.reg.digitaltwins.azure.net";
        var clientId = "abc-123";
        var clientSecret = "super-secret-value";
        var tenantId = "abc-123-def-456";

        var credential = new ClientSecretCredential(tenantId, clientId, clientSecret);
        var client = new DigitalTwinsClient(new Uri(twinUrl), credential);

        var query = @"SELECT level FROM digitaltwins company JOIN building RELATED company.owns JOIN level RELATED building.isPartOf WHERE company.$dtId IN ['Comp1', 'Comp2'] AND IS_OF_MODEL(level, 'dtmi:digitaltwins:rec_3_3:core:Level;1')";

        var results = client.Query<BasicDigitalTwin>(query);

        foreach (var page in results.AsPages(pageSizeHint: 5))
        {
            foreach (var twin in page.Values)
            {
                Console.WriteLine(((JsonElement)twin.Contents["level"]).GetProperty("$dtId").GetString());
            }
        }
    }
}

The same is also repeatable using the same steps but through a tool such as postman to play multiple requests using the continuation token.

Environment

DotNet Info

.NET SDK (reflecting any global.json): Version: 6.0.301 Commit: 43f9b18481

Runtime Environment: OS Name: Windows OS Version: 10.0.22000 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\6.0.301\

Host (useful for support): Version: 6.0.6 Commit: 7cca709db2

.NET SDKs installed: 3.1.420 [C:\Program Files\dotnet\sdk] 5.0.214 [C:\Program Files\dotnet\sdk] 5.0.400 [C:\Program Files\dotnet\sdk] 5.0.403 [C:\Program Files\dotnet\sdk] 5.0.408 [C:\Program Files\dotnet\sdk] 6.0.100 [C:\Program Files\dotnet\sdk] 6.0.101 [C:\Program Files\dotnet\sdk] 6.0.102 [C:\Program Files\dotnet\sdk] 6.0.106 [C:\Program Files\dotnet\sdk] 6.0.202 [C:\Program Files\dotnet\sdk] 6.0.203 [C:\Program Files\dotnet\sdk] 6.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.24 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

IDE Version

Visual Studio Code

Version: 1.68.1 (user setup) Commit: 30d9c6cd9483b2cc586687151bcbcd635f373630 Date: 2022-06-14T12:48:58.283Z Electron: 17.4.7 Chromium: 98.0.4758.141 Node.js: 16.13.0 V8: 9.8.177.13-electron.0 OS: Windows_NT x64 10.0.22000

JetBrains Rider

Version: 2022.1.2 Build #RD-221.5787.36, built on June 1, 2022

Author:	dazfuller
Assignees:	-
Labels:	Service Attention, customer-reported, question, needs-team-attention, Service, Digital Twins
Milestone:	-