Going from SerializerOptions to a custom Serializer makes it impossible to read the data back

Describe the bug I started a project using the following CosmosClientOptions:

new CosmosClientOptions
{
    ConnectionMode = ConnectionMode.Gateway,
    SerializerOptions = new CosmosSerializationOptions
    {
        PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
    },
}

But eventually, I needed more control over the serialization of my data so I switched to a custom serializer:

new CosmosClientOptions
{
    ConnectionMode = ConnectionMode.Gateway,
    Serializer = new CustomCosmosSerializer()
}

Since then, all queries that require composite indexes fail with missing indexes. It looks like the database thinks that my properties are in PascalCase. Changing my indexes to pascal case and trying the queries again yields empty arrays to my serializer instead of triggering an error. It never gets a chance to deserialize any data.

I suspect that my serializer is not used for metadata queries on the Database and Collection objects and cannot participate in the serialization of the query itself. This means that I lost to ability to use LINQ to request objects with camel case properties. Since I cannot use both SerializerOptions and Serializer at the same time to control that internal serializer, I have no way to work around this issue without rewriting a large chunk of code and adding attributes everywhere, which is what I was trying to avoid by using my own serializer in the first place.

To Reproduce

• Create a command-line app that connects to CosmosDb by setting the PropertyNameingPolicy to camel case like in the previous snippet.
• Use the client to create some documents.
• Create a query using the following form to force the use of a composite index:

container.GetItemLinqQueryable<T>()
    .OrderBy(c => c.Prop1)
    .ThenBy(c => c.Prop2)

• After validating that it does return some results replace the SerializerOptions property with the second snippet, using the following dummy serializer:

public class DummyJsonCosmosSerializer : CosmosSerializer
    {
        public override T FromStream<T>(Stream stream)
        {
            return default;
        }

        public override Stream ToStream<T>(T input)
        {
            return new MemoryStream();
        }
    }

Expected behavior The stream passed to the serializer should contain the same set of results as if I was using SerializerOptions. In other words, the use of a custom serializer should not change the outcome of the query and the serializer should see the same data in the stream in both scenarios.

Actual behavior The internal serializer reverts to pascal case causing queries that depends on custom indexes (like an OrderBy().ThenBy() query) to fail with the following message: BadRequest (400) - The order by query does not have a corresponding composite index that it can be served from. Queries that do not require specialized indexes will simply call the serializer with an empty array all the time.

This means that it is currently impossible to switch between SerializerOptions with camel case and a custom Serializer without dumping the database and recreating it from scratch and converting all custom indexes.

Environment summary SDK Version: 3.12.0 OS Version (e.g. Windows, Linux, MacOSX) Windows 10 build 1909

Notes I feel like most of this could be avoided if both properties could be used at the same time. At least, I would still be able to read my data and it would be up to me to ensure that my serializer is compatible with the SerializerOptions. The best case scenario would be to use the custom serializer everywhere, but I’m sure there is a reason why it isn’t the case already.

Also, no data is lost in this process. You can always switch back to the old version of the code and get everything working again… As long as no documents have been written with the new serializer before you notice that you can’t read anything back. Those documents will probably have to be hand-edited to read them with SerializerOptions again. I haven’t tried this scenario.

[kipusoep]

@j82w is it not possible to use camelCase querying support globally instead of having to defined it in every singe place where the CosmosDB is queried?

[BrandonWilliamsCS]

@j82w I must insist that this issue is NOT solved. The linked PR provides a workaround, not a solution. Further, it’s not even an obvious or even well-documented workaround! The documentation acts as if the new argument is there to override options, but the truth is that (when using a custom Serializer) we must use this argument to provide any options whatsoever.

As @kipusoep mentioned, what we actually need is a way to define these options in the same place as we define the custom serializer, as both need to behave the same way. Having to repeat the configuration for every linq query is problematic.

The two suggestions presented in the original question will do the job perfectly, so please reopen this issue and consider one of the proposed solutions.