Look into data rejected by ET Elasticsearch due to changed type

What should we do?

We have received emails this morning from Google error reporting that the ET production Elasticsearch has rejected some modeler error data, specifically the json.message.product.internals.deployment.error field that contained a string instead of a long.

The reason behind this is that elasticsearch is trained every morning depending on which data comes in first on this day. Today this happens to be the value 20 of type long. So today, only values of this type are going to be accepted by elasticsearch for this specific field. Tomorrow, this could change back to a string if the first value that Elasticsearch receives is of type string again.

Since you have sent a string (mostly DIAGRAM_PARSE_ERROR) for this field for 498 of the last 500 records, we want to make you aware of this and suggest you look into whether or not there is anything that needs to be changed in the way the errors are sent to ET.

Here is a link to the ET production Kiban where you can check out what these error messages have looked like in the past weeks up until this morning.

/cc @casulond (Thanks for pointing this out and helping to analyze it!) and @dlavrenuek (Our new/current medic, could you have an eye on this as well?)

Why should we do it?

The data that is rejected by Elasticsearch will not show up in Kibana to be analyzed by you and might distort your analysis of the usage data.

[MaxTru]

Quick update: we discussed and decided to improve the errorHandling in DeploymentTool. (See reasoning ). This would be released then next month (09.02.) and we should then

• only send strings to ET
• send other errors properly to our sentry

Please note that old Modelers would still have the existing behavior. So would it still be possible to allow for data types long and string likewise on ET side?

I will follow-up with a PR, but leave this issue open to track overall progress 👍

[MaxTru]

Thanks for highlighting and the first analysis. I looked into it from Modeler perspective and so far only have a status update (and no fix yet unfortunately) :

• This behavior was introduced with the 4.5.0 release (12th January).
• In particular this commit and line of code is relevant: f52063c. Note that we track error.code
  • Note that we introduced the tracking of data in the context of deployments here. So we did not change / introduce a bug into existing behavior, but this tracking was buggy (or at least based on buggy data) from the beginning.
• As @btepe explained, the “weird” thing is that sometimes an error 20 is sent, while mostly a String (DIAGRAM_PARSE_ERROR, FORBIDDEN, …) is sent.
  • First potential approach / point to discuss: one could argue, that an error code 20 is valid (yet we need to find what it means 😉 ). In this case, we might just ensure to stringify it / or E.T. might make sure to allow both types (as @btepe said). However, lets not discuss this before having understood why 20 is sent.
• The error.code is extracted from the deployment.error event, which is fired by the DeploymentTool
  • The DeploymentTool in the concerned cases (implicitly) expects a DeploymentError. However, looking at it, one will find that a real DeploymentError object will always set error.code to some string.
    • Ergo => IMO we should aim for fixing the errorHandling in the DeploymentTool. Currently we assume that all errors thrown, are deployment errors, but in fact they could come from e.g., saveProcessDefinition

I will sync with the team about this direction for the solution and follow-up.