Allow Document Deletion From Full Shards

Is your feature request related to a problem? Please describe. OpenSearch shards have a 2147483519 doc limit. This is driven by a hard Lucene limit on the number of documents that can exist in a Lucene index.

Once an OpenSearch shard has reached the maximum number of documents, the index begins rejecting document deletions:

GET /_cat/indices/<my index>?v
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   <my index> ne6hnbXiQSOdpxg6zsqPgw   1   1 2147483519            0    569.2gb        284.2gb

DELETE /<my index>/_doc/zpi-oH0Bo4VEviPH9aZC
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "number of documents in the index cannot exceed 2147483519"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "number of documents in the index cannot exceed 2147483519"
  },
  "status" : 400
}

Describe the solution you’d like The ability to delete some docs within the index to free up space. Not sure if there’s a Lucene-level limitation that prevents this behavior.

Describe alternatives you’ve considered N/A

Additional context N/A

[engechas]

I’m interested in contributing to core OpenSearch and this seems like a good starter issue so I can pick up the PR work when I get some time

[dblock]

I’m with @engechas. I vote for a circuit breaker that prevents more documents to be written before users get into the situation of not being able to delete documents in an index. Who cares to PR? 😃