Rename via ALTER TABLE throws exception on some tables

CrateDB version: 2.0.6

JVM version: openjdk version “1.8.0_131” OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

OS version / environment description: Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-47-generic x86_64)

Problem description:

Renaming a table with ALTER TABLE query throws an exception on some tables (maybe related to shard count).

Steps to reproduce:

Table partitioned on 2 columns, 1434 (configured/started) shards, 1.1GB size, 6.3 million records, 0 replicas.

alter table "some_data" rename to "some_data_old";

SQLActionException: INTERNAL_SERVER_ERROR 5000 MultiException in io.crate.executor.transport.ChainableActions$Result.addResultAndError(ChainableActions.java:118)
	at io.crate.executor.transport.ChainableActions$Result.addResultAndError(ChainableActions.java:118)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
	at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
	at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
	at io.crate.executor.transport.ChainableActions.rollbackOnErrors(ChainableActions.java:90)
	at io.crate.executor.transport.ChainableActions.lambda$run$1(ChainableActions.java:59)
	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	at io.crate.action.FutureActionListener.onResponse(FutureActionListener.java:45)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:91)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:87)
	at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:224)
	at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:89)
	at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:222)
	at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:249)
	at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:235)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onResponse(TransportMasterNodeAction.java:152)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onResponse(TransportMasterNodeAction.java:149)
	at org.elasticsearch.action.admin.indices.open.TransportOpenIndexAction$1.onResponse(TransportOpenIndexAction.java:93)
	at org.elasticsearch.action.admin.indices.open.TransportOpenIndexAction$1.onResponse(TransportOpenIndexAction.java:89)
	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:64)
	at org.elasticsearch.cluster.service.ClusterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(ClusterService.java:874)
	at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:630)
	at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:915)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Use case:

Feature description:

[mikethebeer]

@rps-v @vmishra

• The root cause is already fixed in master, we’ll see if we can backport this into 2.1 as well
• 500 shards is still a lot. The root cause of the issue has to do with the number of shards. Would it be possible to reduce this further? You’re less likely to run into the issue with a more reasonable shard number.
• The OutOfMemory exception is actually hiding the root cause. We’ll fix that as well (also in 2.1)

[rps-v]

@mikethebeer We got around the rename/shard count issue by copying the table with a new schema (reduced number of shards) and everything is fine.