size limit for metadata?

Hi!

I have a Client and a Server in two Winform apps. I send a data request command from the Client to the Server:

metadata.Add("Command", "GetDateData");
Client.Connect();
SyncResponse serverResponse = Client.SendAndWait(ClientTimeout, "", metadata);

Then, at the Server side, I check the command metadata, and send back a Json serialized List of class objects (DfileProperties), from the methode “private SyncResponse SyncRequestReceivedServer(SyncRequest req) { … }”:

replyMetadata.Add("Command", "GetDateData");
string json = JsonConvert.SerializeObject(DfileProperties.GetRange(0, 4), Formatting.None);
replyMetadata.Add("ServerDayData", json);

return new SyncResponse(req, replyMetadata, "");

The interesting thing is that, if the object list DfileProperties contains like 5000 elements, the data never arrives, I get timeout error at the SendAndWait() method. If I limit the List size to like 3 items, the data comes through. Is there a size limit? The Class is not huge, only contains some basic type props, like 25 int, string and DateTime values.

Any idea what can be the problem? Thanks! 😃

[jchristn]

Hi @bukkideme sorry for taking a long time to get back to you on this. The reason this is happening is as follows:

1. Metadata is packaged as part of each message frame
2. The message frame’s demarcation from the payload (data) is \r\n\r\n
3. To build a message, Watson reads from the underlying socket one byte at a time and persists that to a header buffer
4. The header buffer’s tail is evaluated for the pattern in 2, and if present, the end of the header has been reached. If not, it continues reading one byte at a time

There isn’t a way AFAIK of peeking ahead in a network stream to find the first occurrence of \r\n\r\n, which is why it’s reading one byte at a time and evaluating the buffer each time.

So with a lot of metadata, let’s say 1MB, that’s 1024 * 1024 (roughly) evaluations of the end of the buffer to see if the end of the headers has been reached.

I did a simple test using the Test.Metadata project (which I’ll be committing momentarily). With 5 key/value pairs in the metadata, is was fast. 50, still ok, 500 it started becoming a problem (148ms), and 5000 was really slow (16.3 seconds).

I’ll add some guidance in the README to make sure this is cleared up for others. Hope this helps!

[jchristn]

Glad you got it working @bukkideme - I will spend some time testing large-ish metadata to see if I can find a way to reproduce. Cheers!