Unable to execute queries with distinct keyword

We are running into a problem while executing a query that has distinct keyword and orders by a field. When the result set fetched during the execution of the query does not match the size limit specified and if we have a continuation token that is not null, we are noticing this issue.

The continuation token generated for us has an attribute called “lastHash” which is not being accepted as a valid token attribute. Here is the error message reported by the sdk,

Failed to query items; nested exception is {\"ClassName\":\"BadRequestException\",\"userAgent\":\"azsdk-java-cosmos/4.38.1 MacOSX/10.16 JRE/1.8.0_232\",\"statusCode\":400,\"resourceAddress\":null,\"innerErrorMessage\":\"One of the input values is invalid.: DistinctContinuationToken is malformed. DistinctContinuationToken can not have a 'lastHash', when the query type is not ordered (ex SELECT DISTINCT VALUE c.blah FROM c ORDER BY c.blah).\",\"causeInfo\":null,\"responseHeaders\":\"{}\"}

The query we were trying to execute,

SELECT distinct c.originalMessageId, c.messageTimestamp FROM c WHERE c.testFlag = @testFlag AND (@originalMessageId = null OR c.originalMessageId = @originalMessageId) AND ARRAY_CONTAINS(@partitionIds, c.partitionId) AND (c.messageTimestamp between @rangeStart and @rangeEnd) ORDER BY c.messageTimestamp ASC

The page number and page size were set using the CosmosPageRequest object. Initially we had it set to 0 and 100 respectively.

https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-spring-data-cosmos/src/main/java/com/azure/spring/data/cosmos/core/CosmosTemplate.java#L826

The query we issued was able to fetch only 20 records in the first iteration and along with the following continuation token,

{"lastHash":"kMmLddiTVKs4TqCdtIm7Uw==","sourceToken":"{\"compositeToken\":\"{\\\"token\\\":\\\"+RID:~NEwdAJ-YNKKtWhsAAABABg==#RT:1#TRC:100#RTD:Svj6cyIw1f7CCts4bwYlBTMxMzMuMTouMzhVMjo7MjU7MzovOjM5WwA=#ISV:2#IEO:65567#QCF:8#FPC:AghtAAAAABkAAHYCAAAAGQAAbQAAAAAZAAAGADEawP/AZW4AAAAAGQAAAgAAQG8AAAAAGQAAAgAAQHAAAAAAGQAAAgAAQHEAAAAAGQAAAgAAQHIAAAAAGQAAAgAAQHMAAAAAGQAAAgAAQHQAAAAAGQAAAgAAQHUAAAAAGQAAAgAAQHYAAAAAGQAAAgAAQHcAAAAAGQAAAgAAQHgAAAAAGQAAAgAAQHkAAAAAGQAAAgAAQHoAAAAAGQAABACBRAEAbgEAAAAZAAAGAIEN/v9wcm8BAAAAGQAAAgAAQHABAAAAGQAAAgAAQHEBAAAAGQAAAgAAQHIBAAAAGQAAAgAAQHMBAAAAGQAAAgAAQHQBAAAAGQAAAgAAQHUBAAAAGQAAAgAAQHYBAAAAGQAAAgAAQHcBAAAAGQAAAgAAQHgBAAAAGQAAAgAAQHkBAAAAGQAAAgAAQHoBAAAAGQAAAgAAQHsBAAAAGQAAAgAAQHwBAAAAGQAAAgAAQH0BAAAAGQAAAgAAQH4BAAAAGQAAAgAAQH8BAAAAGQAAAgAAQIABAAAAGQAAAgAAQIEBAAAAGQAAAgAAQIIBAAAAGQAAAgAAQIMBAAAAGQAAAgAAQIQBAAAAGQAAAgAAQIUBAAAAGQAAAgAAQIYBAAAAGQAAAgAAQIcBAAAAGQAAAgAAQIgBAAAAGQAAAgAAQIkBAAAAGQAAAgAAQIoBAAAAGQAAAgAAQIsBAAAAGQAAAgAAQIwBAAAAGQAAAgAAQI0BAAAAGQAAAgAAQI4BAAAAGQAAAgAAQI8BAAAAGQAAAgAAQJABAAAAGQAAAgAAQJEBAAAAGQAAAgAAQJIBAAAAGQAAAgAAQJMBAAAAGQAAAgAAQJQBAAAAGQAAAgAAQJUBAAAAGQAAAgAAQJYBAAAAGQAAAgAAQJcBAAAAGQAAAgAAQJgBAAAAGQAAAgAAQJkBAAAAGQAAAgAAQJoBAAAAGQAAAgAAQJsBAAAAGQAAAgAAQJwBAAAAGQAAAgAAQJ0BAAAAGQAAAgAAQJ4BAAAAGQAAAgAAQJ8BAAAAGQAAAgAAQKABAAAAGQAAAgAAQKEBAAAAGQAAAgAAQKIBAAAAGQAAAgAAQKMBAAAAGQAAAgAAQKQBAAAAGQAAAgAAQKUBAAAAGQAAAgAAQKYBAAAAGQAAAgAAQKcBAAAAGQAAAgAAQKgBAAAAGQAAAgAAQKkBAAAAGQAAAgAAQKoBAAAAGQAAAgAAQKsBAAAAGQAABAARef8DbwIAAAAZAAAGACE2AMDQSXACAAAAGQAAAgAAQHECAAAAGQAAAgAAQHICAAAAGQAAAgAAQHMCAAAAGQAAAgAAQHQCAAAAGQAAAgAAQHUCAAAAGQAAAgAAQHYCAAAAGQAABAChdB8A\\\",\\\"range\\\":{\\\"min\\\":\\\"05C1B7FFFFFFF0\\\",\\\"max\\\":\\\"05C1BFFFFFFFF0\\\"}}\",\"orderByItems\":[{\"item\":\"2022-09-27T19:14:27.928Z\"}],\"rid\":\"NEwdAJ-YNKI2WhsAAABABg==\",\"inclusive\":true}"}

Since the continuation token has a property called “lastHash”, the next iteration to fetch more records was encountering an exception reported earlier. This is the version of the library we are using,

<dependency> <groupId>com.azure</groupId> <artifactId>azure-spring-data-cosmos</artifactId> <version>3.29.1</version> </dependency>

[kushagraThapar]

@trande4884 please take a look at this, thanks!

[ViswanathVeerappan]

@trande4884 thank you for taking a look at it.