Errors with cassandra storage after upgrading to 2.0.0

We just attempted an upgrade to 2.0.0 using the docker container running in kubernetes. We use cassandra storage backend, which was successful up until this upgrade. The pod appears to dump the following every 10 seconds. This happens across multiple restarts of the pod

WARN [2019-12-06 20:15:38,520] [main] i.c.s.CassandraStorage - Customization of cassandra's retry policy is not supported and will be overridden INFO [2019-12-06 20:15:38,520] [main] c.d.d.c.ClockFactory - Using native clock to generate timestamps. INFO [2019-12-06 20:15:38,563] [main] c.d.d.c.p.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor) INFO [2019-12-06 20:15:38,564] [main] c.d.d.c.Cluster - New Cassandra host XXXXXXX INFO [2019-12-06 20:15:38,564] [main] c.d.d.c.Cluster - New Cassandra host XXXXXXX INFO [2019-12-06 20:15:38,564] [main] c.d.d.c.Cluster - New Cassandra host XXXXXXX WARN [2019-12-06 20:15:38,567] [main] c.d.d.c.CodecRegistry - Ignoring codec DateTimeCodec [timestamp <-> org.joda.time.DateTime] because it collides with previously registered codec DateTimeCodec [timestamp <-> org.joda.time.DateTime] INFO [2019-12-06 20:15:38,592] [main] o.c.c.m.MigrationRepository - Found 8 migration scripts INFO [2019-12-06 20:15:38,592] [main] i.c.s.CassandraStorage - Keyspace reaper_db already at schema version 23 ERROR [2019-12-06 20:15:38,615] [main] i.c.ReaperApplication - Storage is not ready yet, trying again to connect shortly... com.datastax.driver.core.exceptions.InvalidQueryException: Undefined column name nodes at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:49) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35) at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:86) at io.cassandrareaper.storage.CassandraStorage.prepareStatements(CassandraStorage.java:434) at io.cassandrareaper.storage.CassandraStorage.<init>(CassandraStorage.java:228) at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:439) at io.cassandrareaper.ReaperApplication.tryInitializeStorage(ReaperApplication.java:310) at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:172) at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:91) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87) at io.dropwizard.cli.Cli.run(Cli.java:78) at io.dropwizard.Application.run(Application.java:93) at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:110) Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Undefined column name nodes at com.datastax.driver.core.Responses$Error.asException(Responses.java:181)

Is there a missing schema migration that we need to apply?

[adejanovski]

It just striked me that you may be upgrading from a 1.5 beta version of Reaper, where existed an old version of migration 23, which had a different schema for the diagnostic_event_subscription table. As you can see here, the include_nodes was renamed to nodes at some point. Your solution here will be to drop and recreate the table as suggested before. Let me know if that’s the case.

[adejanovski]

Thanks