Bulk updates and NPE

CrateDB version: 0.56.3

JVM version: 1.8.0_92-b14

OS version / environment description: Centos 7

Problem description: When sending bulk updates via the Crate HTTP API, the API would hang and not return any response. The following message shows up in the logs:

[WARN ][action.bulk ] ShardUpsert Item [someNumber] failed: Failure{id=UUID, message=NullPointerException[null], versionConflict=false}

Steps to reproduce: Updating 5 STRING columns and sending 1k bulk_args

POST /_sql
{
    "stmt": "UPDATE table SET col1 = ?, col2 = ?, col3 = ?, col4 = ?.... WHERE id = ?",
    "bulk_args": [ [someVal, anotherVal, .....], [...], [...] ]
}

Bulk Settings:

threadpool.bulk.type: fixed
threadpool.bulk.queue_size: 500
threadpool.bulk.processors: 8

[mfussenegger]

We’ve a fix for this (https://github.com/crate/crate/pull/4895) which will be in the next release (1.0.4).

Note that this fixes the getting stuck, the statement itself will still fail if the bulk_size is bigger than what can be handeld (thread_pool size/queue size). I’ve also added a card to our backlog to improve this in the future so that we can handle a bigger bulk-size without having to increase the threadpool queue size.

We’re still looking into https://github.com/crate/crate/issues/4885 which is in the same area.

[bputt]

6 Node Cluster (8cpu x 64 gb / node)

Table schema

CREATE TABLE table (
    a STRING PRIMARY KEY,
    b LONG PRIMARY KEY,
    c INTEGER,
    d INTEGER,
    e INTEGER,
    f STRING,
    g INTEGER,
    h INTEGER,
    i LONG,
    j INTEGER,
    k INTEGER,
    l LONG,
    m INTEGER,
    n INTEGER,
    o STRING,
    p STRING,
    q STRING,
    r STRING,
    s STRING,
    t STRING,
    u array(object as (
        a STRING,
        b STRING INDEX OFF
    )),
    v array(object as (
        a STRING,
        b STRING,
        c LONG,
        d LONG,
        e INTEGER,
        f INTEGER,
        g INTEGER
    ))
) CLUSTERED INTO 5 SHARDS
WITH (number_of_replicas = 3)