ReadSideProcessor ConfigurationException: Column family ID mismatch

Lagom Version (2.5.x / etc)

1.2.0-RC2

API (Scala / Java / Neither / Both)

both

Actual Behavior

1. lagomCassandraCleanOnStart in ThisBuild := true
2. sbt clean runAll is working
3. sbt runAll after that leads to following error.

2016-11-07 18:08:52,784 ERROR org.apache.cassandra.transport.Message - Unexpected error during query
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:513) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:343) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:313) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:85) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:93) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [cassandra-all-3.0.2.jar:3.0.2]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-3.0.2.jar:3.0.2]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_102]
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364) ~[cassandra-all-3.0.2.jar:3.0.2]
        ... 19 common frames omitted
Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:776) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:736) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.config.Schema.updateTable(Schema.java:626) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1349) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1305) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1255) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-3.0.2.jar:3.0.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102]
        ... 1 common frames omitted
2016-11-07 18:08:52,784 ERROR org.apache.cassandra.service.CassandraDaemon - Exception in thread Thread[MigrationStage:1,5,main]
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:776) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:736) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.config.Schema.updateTable(Schema.java:626) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1349) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1305) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1255) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-3.0.2.jar:3.0.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
2016-11-07 18:08:52,788 ERROR org.apache.cassandra.transport.messages.ErrorMessage - Unexpected exception during request
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:513) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:343) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:313) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:85) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:93) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [cassandra-all-3.0.2.jar:3.0.2]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-3.0.2.jar:3.0.2]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_102]
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364) ~[cassandra-all-3.0.2.jar:3.0.2]
        ... 19 common frames omitted
Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found dba62c30-a50c-11e6-a61d-afd668f101df; expected dba60521-a50c-11e6-a61d-afd668f101df)
        at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:776) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:736) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.config.Schema.updateTable(Schema.java:626) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1349) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1305) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1255) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531) ~[cassandra-all-3.0.2.jar:3.0.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-3.0.2.jar:3.0.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102]
        ... 1 common frames omitted

Here is my OrganizationEventProcessor.

class OrganizationEventProcessor @Inject()(
  session: CassandraSession,
  readSide: CassandraReadSide,
  organizationService: OrganizationService,
  userService: UserService,
  config: ConfigurationProvider
)(implicit ec: ExecutionContext) extends ReadSideProcessor[OrganizationEvent] {

  import CassandraReadSide._
  import converter.CompletionStageConverters._

  override def aggregateTags(): PSequence[AggregateEventTag[OrganizationEvent]] = OrganizationEvent.AllTags

  val buildHandler: ReadSideProcessor.ReadSideHandler[OrganizationEvent] = {
    readSide
      .builder("organizationeventprocessoroffset")
      .setGlobalPrepare(createTables)
      .build()
  }

  private[this] def createTables: CompletionStage[Done] = asCompletionStage {
    session.executeCreateTable(
      "CREATE TABLE IF NOT EXISTS organizations ("
        + "id text, org_type text, name text,"
        + "PRIMARY KEY (id))"
    ).flatMap(_ =>
      session.executeCreateTable(
        "CREATE INDEX IF NOT EXISTS organizations_type_idx ON organizations (org_type);"
      )
    ).flatMap(_ =>
      session.executeCreateTable(
        "CREATE TABLE IF NOT EXISTS subcontractors ("
          + "general_contractor text, subcontractor text,"
          + "PRIMARY KEY (general_contractor, subcontractor))"
      )
    )
  }
}

[ulsamern]

@kotdv did you ever find a solution to your issues? we are experiencing the same problem (timeouts during startup, C* failing with column family ID mismatch error, etc) when running tests using the embedded Cassandra instance for our project.

[TimMoore]

@kotdv it’s not best to discuss this in the comments on a closed issue. If you think there’s a reproducible bug in Lagom, please open a new issue. If you’re not sure, please post a thread in the Google Group.