Bulk Insert is failing silently

NEST/Elasticsearch.Net version: 6.3.0

Elasticsearch version: 6.2.3

Description of the problem including expected versus actual behavior: Bulk insert is failing silently. I have a BulkAllObserver with a listener on onError to log out any errors that occur during the batching but no errors are being output. This has been hiding data loading errors for weeks in production and we only just noticed. This seems to be true for all error conditions, but I will provide a single case that should be testable

Steps to reproduce:

1. Create a new, blank index and define a data schema
2. Create a bulk operation to insert a ton of documents into this index but have some of the object properties violate the schema (e.g. try to insert Foo : string when the schema requires Foo : int)
3. Subscribe a BulkAllObserver to output errors during the operation
4. Verify that no errors were logged out
5. Verify that the index contains no documents

Here is the code I am currently using:

var operationId = Guid.NewGuid();
Logger.Information("Beginning bulk insert operation to ElasticSearch. Operation ID: {operationId}", this, operationId);

var waitHandle = new CountdownEvent(1);
var bulkInsert = _client.BulkAll(vehicles, b => b
	.Index("inventory")
	.BackOffRetries(2)
	.BackOffTime("30s")
	.RefreshOnCompleted()
	.MaxDegreeOfParallelism(4)
	.Size(100));

bulkInsert.Subscribe(new BulkAllObserver(
	onError:     ex => Logger.Fatal("Bulk insert error {message} on operation {operationId}", this, ex, ex.Message, operationId),
	onCompleted: () =>
	{
		Logger.Information(
			"Successfully inserted {count} documents. Operation ID: {operationId}", 
			this, 
			vehicles.Count, 
			operationId);

		waitHandle.Signal();
	}));

waitHandle.Wait();

Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):

[Mpdreamz]

Hi @dasjestyr thank you for reporting this.

Agreed this is completely unexpected behaviour. When this was initially written the idea was to continue to try and succeed but nothing in the contract or docs hints at this. Working on a PR now that will halt the bulk and feed into OnError by default and provide an opt in flag to continue on dropped documents with a callback to feed these documents in to another system/index/DLQ.

[balukantu]

Hi Team,

Is there a way to insert a document if the document doesn’t exist or else update the document in Elasticsearch using BulkAll?

How can we handle not duplicating the records if we do the import twice?