"Message not delivered" probably causing flakiness in LN payments between eclair nodes

In simverse I have a test suite which exercises simple LN payment in these scenarios:

• bitcoind+2 eclair nodes (direct payment alice-bob)
• bitcoind+3 eclair nodes (indirect payment alice-charlie-bob)

The second scenario started to be flaky (failing most of the time). I noticed that when this happens there is logged a message like “Message … from Actor[…] to Actor[…] was not delivered…”

alice_1      | 2019-08-16 13:27:40,990 INFO  f.a.eclair.router.Router  - finding a route 027a9b56fc656011601cc1a64740dcc2a953e18ca998c3f473b113a5dcae8c1404->03b95bf9ad11766b9859e24ac8b7eb8f23a8d5c45c887e31718eaa0f564be9ca2d with assistedChannels= ignoreNodes= ignoreChannels= excludedChannels=
alice_1      | 2019-08-16 13:27:40,993 INFO  f.a.eclair.router.Router  - finding a route with randomize=true params=RouteParams(true,MilliSatoshi(21000),0.03,6,1008,Some(WeightRatios(0.15,0.35,0.5)))
alice_1      | 2019-08-16 13:27:41,075 INFO  a.a.DeadLetterActorRef  - Message [fr.acinq.eclair.payment.PaymentLifecycle$PaymentFailed] from Actor[akka://eclair-node/user/$m/payment-initiator/$a#528843153] to Actor[akka://eclair-node/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

This used to work and the breakage is quite recent (in last 10 days). I wanted to bisect for exact commit causing this but the situation got complicated a bit and I wasn’t able to identify it precisely due to other changes like 4929febbd11ebad6ae20caed4b161db44dfd263d).

Any hints what could be causing this and potential fixed would be appreciated. Thanks.

[darwin]

With git bisect I was able to pin-point the commit causing this to 4929febbd11ebad6ae20caed4b161db44dfd263d (this is expected because that was the only commit in last ~11 days).

The trick on my side was to make bisect work across that change and keep backward compatibility: https://github.com/darwin/simverse/commit/349b0cb75a82890f284aedcf8f6632bfd19aa4f2

[darwin]

Ok. My approach is fragile. But I don’t know a good way how to effectively wait for LN network to be ready to successfully execute my test payment.

Anyways. The test suite started to be passing again. At least I have new diagnostics code in place for case this happened again in the future. Also I bumped maxAttempts to 100.

Thanks for all your input. Closing.