Upgrading to 2.1 gives me "Storage is not ready yet, trying again to connect shortly..."

I’m using Docker version.

When I upgrade to tag ad53ff3 (because I need this fix), the message “Storage is not ready yet, trying again to connect shortly…” keeps appearing on the logs forever and the connection never really happens.

ERROR  [2020-11-07 13:11:16,885] [main] i.c.ReaperApplication - Storage is not ready yet, trying again to connect shortly...
java.lang.NullPointerException: null
	at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:469)
	at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:447)
	at io.cassandrareaper.storage.CassandraStorage.<init>(CassandraStorage.java:225)
	at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:477)
	at io.cassandrareaper.ReaperApplication.tryInitializeStorage(ReaperApplication.java:300)
	at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:179)
	at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:98)
	at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:93)
	at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:117)

At first I thought it could be a migration issue so I dropped and recreated the reaper_db keyspace. The issue is still there.

These are my envvars:

REAPER_CASS_CLUSTER_NAME: "myclustername"
REAPER_CASS_CONTACT_POINTS: "[bunch,of,ips]"
REAPER_CASS_LOCAL_DC: "datacenter1"
REAPER_STORAGE_TYPE: cassandra
REAPER_HANGING_REPAIR_TIMEOUT_MINS: "120"
REAPER_BLACKLIST_TWCS_TABLES: "0"
REAPER_REPAIR_MANAGER_SCHEDULING_INTERVAL_SECONDS: "10"
REAPER_JMX_CONNECTION_TIMEOUT_IN_SECONDS: "5"
REAPER_REPAIR_PARALELLISM: "DATACENTER_AWARE"

---
Upon rolling back to 2.0.5 (this might be useful, don't know) on an empty keyspace everything worked fine _up to migration 23 to 24_. That migration keeps giving the same error of "Storage is not ready yet, trying again to connect shortly..." 

Then it says: `java.lang.IllegalStateException: Database migration can not happen with other reaper instances running. Found  [[172.17.0.2]]`

But there's no other instance running. Checked `running_reapers` and it only shows 1 indeed.

reaper_instance_id | last_heartbeat | reaper_instance_host --------------------------------------±--------------------------------±--------------------- 59e4b43e-f4f2-4429-bfc0-03dac8f73355 | 2020-11-07 13:17:00.596000+0000 | 172.17.0.2

Anyways, truncating the table made the migration finish just fine. I'll keep using 2.0.5 for now.



┆Issue is synchronized with this [Jira Task](https://k8ssandra.atlassian.net/browse/K8SSAND-259) by [Unito](https://www.unito.io)
┆Issue Number: K8SSAND-259

[adejanovski]

I’m unable to reproduce the issue, using the 2.1.2 docker image on an Ubuntu 18.04 instance 🤔 So this is not a general problem but it affects some setups. According to this SO page, the unexpected redirection seems to be due to the shebang we use in configure-persistence.sh which is #!/bin/sh instead of #!/bin/bash. I guess the fix here would be to switch to a base docker image that includes bash as default.

[adejanovski]

ok, that’s a solid lead @Gurpartap, thanks!

Looking at this now.