Provide a way to ignore attributes in JSON when decoding/deserialising an event

I am experimenting with CloudEvents and have decided to persist my events to Azure Cosmos. I had a few issues with serialisation but eventually came up with something like this which successfully writes to Cosmos.

public override async Task<CloudEvent> AddAsync(CloudEvent item, CancellationToken cancellationToken = default)
{
	var container = CosmosContainerFactory.GetContainer(ContainerName);

	try
	{
		var formatter = new JsonEventFormatter();
		var bytes = formatter.EncodeStructuredModeMessage(item, out _);

		await using var stream = BinaryData.FromBytes(bytes).ToStream();
		var response = await container.CreateItemStreamAsync(
			stream,
			GetPartitionKey(item),
			cancellationToken: cancellationToken
		);

		response.EnsureSuccessStatusCode();
		return item;
	}
	catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.Conflict)
	{
		throw new CosmosItemAlreadyExistsException(IdProvider.GetId(item));
	}
}

I am using the SystemTextJson JSON formatter here to encode the message. It feels a bit janky but this seemed to be the only way to simply serialise the event without using any of the ToHttpContent stuff (which I don’t want/need). Firstly, is there any better way of doing this?

Next, my main issue is that when retrieving an event from my Cosmos table, the deserialisation fails due to the Cosmos system properties containing underscores (which I can see that the CloudEvent spec and validation prohibits). I can see from looking at the JsonEventFormatter code that these properties are being treated as event attributes. I want to be able to ignor ethese properties but there doesn’t seem to be any way of doing this because:

1. CloudEvent is a sealed class so I can’t do any fancy inheritance and [JsonIgnore] combination.
2. The event formatter does not expose any way of ignoring properties. It just treats any property which isn’t a ‘standard’ property as a custom event attribute.

FWIW, here is my code for fetching an event from Cosmos:

public override async Task<CloudEvent?> GetByIdAsync(
	string id,
	PartitionKey partitionKey,
	CancellationToken cancellationToken = default)
{
	var container = CosmosContainerFactory.GetContainer(ContainerName);
	
	try
	{
		var response = await container.ReadItemStreamAsync(
			id,
			partitionKey,
			cancellationToken: cancellationToken
		);
		
		var formatter = new JsonEventFormatter();
		var cloudEvent = await formatter.DecodeStructuredModeMessageAsync(response.Content, new ContentType("application/json"), null);
		return cloudEvent;
	}
	catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
	{
		return null;
	}
}

[jskeet]

Wrapping the CloudEvent sounds like a decent suggestion but also having those APIs to parse/format to JsonElement would be awesome!

The latter would make it easier to do that wrapping, of course 😃

Tomorrow, I’ll close this issue but file a new one as a feature request for those additional methods. (I can see this being a pattern to apply to the XML format as well…)

[jskeet]

I understand that this is an issue mostly caused by Cosmos but I wondered if we could have a bit of flexibilty in the CloudEvents API to handle situations like this.

I really don’t think so. The JSON you’ve provided simply isn’t a valid structured mode JSON format CloudEvent. It violates the specification, so I think it’s absolutely correct for it to fail to parse. As soon as we start accepting bad data, it’s a slippery slope - next it’ll be “please just lower-case the property names” or “don’t worry about JSON that isn’t actually valid”. I’m a big fan of parsing in a really strict manner, to avoid broken data being presented incompatibly to other code.

If you can’t stop Cosmos from modifying the data you’re storing, I’d strongly recommend creating a wrapper object instead - put the CloudEvent JSON into a cloudEvent property or similar, then when you fetch the stream, parse it and extract that property. We could potentially add public methods to deserialize a JsonElement, to avoid having to serialize back and forth a lot. (We’ve got code like that in the ProtobufEventFormatter, for example - ConvertToProto and ConvertFromProto. Similar methods for JSON formatters could be fine.)

An alternative is to parse the JSON and strip out the extra properties yourself before deserializing. But that sounds a bit risky to me - what’s to stop Cosmos from (at some point in the future) modifying your JSON to override the id property, for example?

Side question: is there a different place you’d recommend persisting CloudEvents?

Anywhere that doesn’t modify your data silently? 😉 Seriously, without knowing what you’re trying to achieve, it’s very hard to suggest anywhere else.