"Key was already exists" on NEST indexing after upgrading from ES v6.8.1

NEST/Elasticsearch.Net version: 7.2.0/7.2.0 (and 7.1.0/7.1.0)

Elasticsearch version: 7.3.0

Description of the problem including expected versus actual behavior:

I upgraded my elasticsearch server from 6.8.1 to 7.3.0 and after this, the NEST indexing function (Client.IndexDocument(document)) throws with Key was already exists. Key:MultipleChoices. I can’t see any properties in my document corresponding to “MultipleChoices”, so I’m not sure where that is coming from. Furthermore, by changing what index it posts to, I still get the same error (even on a new index), and the (new) index is never created, so this error gets thrown before indexing happens.

If I change the document type from my custom class to just new-ing up a { Test = "testing" }, that indexes successfully without throwing an error.

Simply POSTing the document directly at the Elasticsearch API also works fine.

By rewriting my code to use the low-level client I made it work again:

Client.LowLevel.Index<MyType>(index, documentAsJson);

I searched a bit and found #3706, but that looks a lot more low-level than what I was doing, but maybe that error propagates up to the NEST index function? It is the case that my custom type was first created/indexed on ES v6.6.0, then used again on v6.8.1 and now in v7.3.0.

My custom class is as follows:

[Serializable] // using Elasticsearch.Net;
public class HealthCheckResponse : IElasticsearchResponse
{
    // HealthStatus, HealthCheckProtocol and HttpStatusCode are enums
    public string Host;
    public HealthStatus Status;
    public string Message;
    public string Name;
    public HealthCheckProtocol Protocol;
    public DateTime Timestamp;
    public HttpStatusCode HttpStatusCode;
    public long PingRoundTripTime;
}

And the JSON I now send “manually” through the low-level client API is as follows (manually mapping the enums from ints to strings)

{
  "host": "127.0.0.1",
  "status": "Success",
  "message": "Success",
  "name": "localhost",
  "protocol": "Ping",
  "timestamp": "2019-08-16T13:33:21.8036015+02:00",
  "httpStatusCode": 0,
  "pingRoundTripTime": 0
}

Steps to reproduce:

1. Use NEST.Client.IndexDocument to index a document against an ES v6.6.0 server
2. Upgrade the ES server to v7.3.0 (through 6.8.1)
3. Try to use IndexDocument the same way against the upgraded server, and get an exception thrown.

Do tell if you would like me to debug some more or test some other cases, I’ll do my best to help out debugging this 😃 And feel free to ask for more information if that’s relevant.

[sklirg]

A friend of mine noticed this pull request to the original Utf8Json library which targets an issue with the same exception message. https://github.com/neuecc/Utf8Json/issues/128

It looks related, but I’m not sure how to change the code to make a patch to work with my project. If you could give me some pointers on how to do this (link to a resource/guide would be plenty) I’d love to test it.

[russcam]

@sklirg right, the issue is that System.Net.HttpStatusCode maps multiple enum values to the same underlying int value:

https://github.com/dotnet/corefx/blob/master/src/System.Net.Primitives/src/System/Net/HttpStatusCode.cs

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

namespace System.Net
{
    // HTTP status codes as per RFC 2616.
    public enum HttpStatusCode
    {
        // Informational 1xx
        Continue = 100,
        SwitchingProtocols = 101,
        Processing = 102,
        EarlyHints = 103,

        // Successful 2xx
        OK = 200,
        Created = 201,
        Accepted = 202,
        NonAuthoritativeInformation = 203,
        NoContent = 204,
        ResetContent = 205,
        PartialContent = 206,
        MultiStatus = 207,
        AlreadyReported = 208,

        IMUsed = 226,

        // Redirection 3xx
        MultipleChoices = 300,
        Ambiguous = 300,
        MovedPermanently = 301,
        Moved = 301,
        Found = 302,
        Redirect = 302,
        SeeOther = 303,
        RedirectMethod = 303,
        NotModified = 304,
        UseProxy = 305,
        Unused = 306,
        TemporaryRedirect = 307,
        RedirectKeepVerb = 307,
        PermanentRedirect = 308,

        // Client Error 4xx
        BadRequest = 400,
        Unauthorized = 401,
        PaymentRequired = 402,
        Forbidden = 403,
        NotFound = 404,
        MethodNotAllowed = 405,
        NotAcceptable = 406,
        ProxyAuthenticationRequired = 407,
        RequestTimeout = 408,
        Conflict = 409,
        Gone = 410,
        LengthRequired = 411,
        PreconditionFailed = 412,
        RequestEntityTooLarge = 413,
        RequestUriTooLong = 414,
        UnsupportedMediaType = 415,
        RequestedRangeNotSatisfiable = 416,
        ExpectationFailed = 417,
        // From the discussion thread on #4382:
        // "It would be a mistake to add it to .NET now. See golang/go#21326,
        // nodejs/node#14644, requests/requests#4238 and aspnet/HttpAbstractions#915".
        // ImATeapot = 418

        MisdirectedRequest = 421,
        UnprocessableEntity = 422,
        Locked = 423,
        FailedDependency = 424,

        UpgradeRequired = 426,

        PreconditionRequired = 428,
        TooManyRequests = 429,

        RequestHeaderFieldsTooLarge = 431,

        UnavailableForLegalReasons = 451,

        // Server Error 5xx
        InternalServerError = 500,
        NotImplemented = 501,
        BadGateway = 502,
        ServiceUnavailable = 503,
        GatewayTimeout = 504,
        HttpVersionNotSupported = 505,
        VariantAlsoNegotiates = 506,
        InsufficientStorage = 507,
        LoopDetected = 508,

        NotExtended = 510,
        NetworkAuthenticationRequired = 511
    }
}

Utf8Json’s enum serialization does not currently handle this, but it’s a common enough case that it should be fixed in the client