Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Provider reconnect/recovery isn't working as expected

See original GitHub issue

Hi,

I’m running an Eventeum node that is listening to both Infura and Alchemy providers in HA manner (refs: https://github.com/ConsenSys/eventeum/issues/141#issuecomment-665352616). I’ve faced an issue where the Infura’s subscription goes down for some reason but the Eventeum wasn’t able to recover itself and reconnect to the provider. As you can see in the attached log, the latest attempt to reconnect was at 2020-09-25 11:27:33.004, after that the service continues to run for a few days without making no more attempts, the Infura connection was established again just after the Eventeum restart at 2020-09-28 07:50. eventeum-infura-recovery.log

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:9

Top GitHub Comments

1reaction

marcelomorgadocommented, Oct 20, 2020

Hi @craigwilliams84 Yes, the latest one was a few days ago:

After the last line, the Infura just back to work after restarting the Eventeum.

Thank you very much!

2020-10-16 19:50:19.491 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:19.492 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:19.496 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:20.343 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:23.469 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:23.469 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:23.469 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:24.304 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:27.425 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:27.426 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:27.426 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:28.272 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:31.398 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:31.399 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:31.399 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:32.236 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:35.375 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:35.379 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:35.380 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:36.181 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:39.315 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:39.315 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:39.315 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:40.158 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:43.279 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:43.279 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:43.279 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:44.119 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:47.251 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:47.252 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:47.252 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:48.089 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:51.211 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:51.211 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:51.212 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:52.003 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:55.133 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:55.134 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:55.134 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:55.972 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:59.095 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:50:59.095 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:50:59.095 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:50:59.933 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:03.066 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:03.066 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:51:03.071 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:51:03.911 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:07.040 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:07.041 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:51:07.041 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:51:07.956 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:11.086 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:11.086 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:51:11.086 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-10-16 19:51:14.203 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
2020-10-16 19:51:14.203 [eventeum-scheduler2] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-10-16 19:51:14.203 [eventeum-scheduler2] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure

0reactions

marcelomorgadocommented, Dec 12, 2020

Hi, update here: I’ve upgraded to 0.9.1-SNAPSHOT and it’s running without issues since a few days ago, I’ll close this issue and re-open it if back to see the same problem again. Thanks again!

Top Results From Across the Web

BitLocker recovery: known issues - Windows Client

This article describes common issues that may prevent BitLocker from behaving as expected when a drive is recovered, or that may cause BitLocker ......

Azure AD authentication troubleshooting: Known problems ...

Azure AD is not the UW Identity Provider, so this feature is not expected to work. Azure AD authentication tokens generally last indefinitely ......

MLN006903 – Medicare Secondary Payer - CMS

Generally, for MSP GHP situations, we recover improper payments. We can fine providers, physicians, and suppliers for knowingly, willfully, and repeatedly.

Recovery: It's time for the physician workforce to heal

Recover, don't revert. The goal is not to return to "normal" because normal wasn't working for physicians as it was. · Maintain the...

ProviderConnect - Providers - Login - Beacon Health Options

No information or resource provided through the Beacon Health Options site is intended to substitute for the professional judgment of a behavioral health ......