Might be Memory leak issue with latest release (v0.9.1 Snapshot)

I am using Eventeum on mainnet and tracking ERC20 transfers of USDCV2 contract (I am also tracking other events too). So I noticed that eventeum was keep getting crashed, see logs below.

Exception in thread "WebSocketConnectReadThread-118" java.lang.OutOfMemoryError: Java heap space
2020-12-28 12:55:40.090 [EVENT-INFURA-0] ERROR net.consensys.eventeum.chain.contract.DefaultContractEventProcessor: triggerListener - An error occurred when processing contractEvent with id 0xa2bcce5dbc2b35318dadf47f2e59c433db6d0cf8e2483ccabc2b1b2be387edd3-0x2178751b73d6c1b930711ffbee64748d685901a3c90c3b02a83ccad4bbf63bb1-428
net.consensys.eventeum.chain.service.BlockchainException: Unable to connect to the ethereum client
	at net.consensys.eventeum.chain.service.Web3jService.getTransactionReceipt(Web3jService.java:181) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.BroadcastAndInitialiseConfirmationListener.isTransactionStillInBlock(BroadcastAndInitialiseConfirmationListener.java:100) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.BroadcastAndInitialiseConfirmationListener.shouldInstantlyConfirm(BroadcastAndInitialiseConfirmationListener.java:95) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.BroadcastAndInitialiseConfirmationListener.onEvent(BroadcastAndInitialiseConfirmationListener.java:59) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.triggerListener(DefaultContractEventProcessor.java:93) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$triggerListeners$4(DefaultContractEventProcessor.java:88) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_275]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.triggerListeners(DefaultContractEventProcessor.java:87) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$processLogsForFilter$3(DefaultContractEventProcessor.java:71) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_275]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.processLogsForFilter(DefaultContractEventProcessor.java:69) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$null$0(DefaultContractEventProcessor.java:50) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_275]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$processLogsInBlock$1(DefaultContractEventProcessor.java:50) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
Caused by: java.io.IOException: Request with id 6916 timed out
	at org.web3j.protocol.websocket.WebSocketService.lambda$setRequestTimeout$3(WebSocketService.java:253) ~[core-4.5.18.jar!/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_275]
	... 3 common frames omitted
2020-12-28 12:55:40.093 [BLOCK-INFURA-0] ERROR net.consensys.eventeum.chain.service.strategy.AbstractBlockSubscriptionStrategy: onError - There was an error when processing a block, disposing blocksubscription (will be reinitialised)
java.util.concurrent.CompletionException: net.consensys.eventeum.chain.service.BlockchainException: Unable to connect to the ethereum client
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[na:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[na:1.8.0_275]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1643) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
Caused by: net.consensys.eventeum.chain.service.BlockchainException: Unable to connect to the ethereum client
	at net.consensys.eventeum.chain.service.Web3jService.getTransactionReceipt(Web3jService.java:181) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.BroadcastAndInitialiseConfirmationListener.isTransactionStillInBlock(BroadcastAndInitialiseConfirmationListener.java:100) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.BroadcastAndInitialiseConfirmationListener.shouldInstantlyConfirm(BroadcastAndInitialiseConfirmationListener.java:95) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.BroadcastAndInitialiseConfirmationListener.onEvent(BroadcastAndInitialiseConfirmationListener.java:59) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.triggerListener(DefaultContractEventProcessor.java:93) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$triggerListeners$4(DefaultContractEventProcessor.java:88) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_275]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.triggerListeners(DefaultContractEventProcessor.java:87) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$processLogsForFilter$3(DefaultContractEventProcessor.java:71) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_275]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.processLogsForFilter(DefaultContractEventProcessor.java:69) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$null$0(DefaultContractEventProcessor.java:50) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_275]
	at net.consensys.eventeum.chain.contract.DefaultContractEventProcessor.lambda$processLogsInBlock$1(DefaultContractEventProcessor.java:50) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[na:1.8.0_275]
	... 3 common frames omitted
Caused by: java.io.IOException: Request with id 6916 timed out
	at org.web3j.protocol.websocket.WebSocketService.lambda$setRequestTimeout$3(WebSocketService.java:253) ~[core-4.5.18.jar!/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_275]
	... 3 common frames omitted
2020-12-28 12:55:43.270 [eventeum-scheduler1] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: isNodeConnected - Get latest block failed with exception on node infura
net.consensys.eventeum.chain.service.BlockchainException: Error when obtaining the current block number
	at net.consensys.eventeum.chain.service.Web3jService.getCurrentBlockNumber(Web3jService.java:195) ~[eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.service.health.NodeHealthCheckService.isNodeConnected(NodeHealthCheckService.java:143) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.service.health.NodeHealthCheckService.checkHealth(NodeHealthCheckService.java:110) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at net.consensys.eventeum.chain.service.health.NodeHealthCheckService.lambda$onApplicationEvent$0(NodeHealthCheckService.java:101) [eventeum-core-0.9.1-SNAPSHOT.jar!/:0.9.1-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
Caused by: java.io.IOException: Request with id 6917 timed out
	at org.web3j.protocol.websocket.WebSocketService.lambda$setRequestTimeout$3(WebSocketService.java:253) ~[core-4.5.18.jar!/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_275]
	... 3 common frames omitted
2020-12-28 12:55:43.271 [eventeum-scheduler1] ERROR net.consensys.eventeum.chain.service.health.NodeHealthCheckService: checkHealth - Node infura is down or unsubscribed!!
2020-12-28 12:55:43.272 [eventeum-scheduler1] INFO  net.consensys.eventeum.chain.service.health.strategy.WebSocketResubscribeNodeFailureListener: reconnect - Reconnecting web socket because of infura node failure
2020-12-28 12:55:43.272 [eventeum-scheduler1] INFO  net.consensys.eventeum.chain.websocket.WebSocketReconnectionManager: reconnect - Attempting websocket reconnection...

I noticed that first error in logs was Exception in thread "WebSocketConnectReadThread-118" java.lang.OutOfMemoryError: Java heap space , I allocated 300-500 MB memory before, so I tried to increase it to 1gigs. It recoverd but it was still throwing error after 30-40 mins, the same error, sometimes it reconnects but most of the times it doesn’t.

I noticed that when I increased allocated RAM, crash was happening some time later than previous configuration. After that I allocated 4gigs RAM to the service and service ran fine for 3-4 hours, then it crashed. I also noticed that with time Eventeum service was consuming more memory. I restarted Eventeum service again, here is the memory usage over 1-2 hours.

[Ferparishuertas]

It’s more general. It’s a jvm tuning topic. If you have different values on xmx xms, you are forcing a continuous resize which in some cases, generates bottlenecks. Added to this, you have changed the new space, increasing it to have more transactional space, which is the main need of eventeum jvm behaviour.

Hope I have explain myself well.

Regards

El mié, 13 ene 2021 a las 9:34, Devashish Tomar ([e-mail]) escribió:

Its been 10-13 days since Eventeum was configured with XMX and XMS to the same value on startup 2500m to avoid a further memory leak. Eventeum start with usage of 1gigs of ram and then it kept using more until it reaches the saturation point of ~ 2.5 gigs, no more usage after 2.5 gigs.

So for now, I think the memory consumption issue is resolved for us, but @Ferparishuertas https://github.com/Ferparishuertas may I ask what was going wrong before? Is this problem exclusive to Eventeum or is much more general?

[image: image] https://user-images.githubusercontent.com/40403260/104426093-7a055000-55a7-11eb-9a29-f6d2f54ef1dd.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eventeum/eventeum/issues/158#issuecomment-759292906, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKXFDQHH7CKW24A7CAGTCLSZVLI5ANCNFSM4VMGNGJA .

– [image: logo] https://io.builders/ io.builders http://io.builders/ Blockchain venture builder Fernando París CTO • ioBUILDERS Mobile: [phone] Parque Científico y Tecnológico UPM Campus de Montegancedo, s/n. 28223 Pozuelo de Alarcón, MADRID

NOTA LEGAL: Este mensaje y cualquier archivo adjunto está destinado únicamente a quien se dirige y es confidencial. Si usted ha recibido este mensaje por error, comuníqueselo al remitente y bórrelo inmediatamente. La utilización, revelación y/o reproducción del mensaje puede constituir un delito.

PROTECCIÓN DE DATOS – Responsable: ioBUILDERS, S.L., Finalidad. Envío de información, respuesta a consultas y contactos genéricos, mientras dure nuestra relación y tengamos su consentimiento. Destinatarios. No se cederán datos a terceros salvo obligación legal. Derechos. Puede ejercer los derechos de acceso, rectificación, supresión y oposición, limitar el tratamiento de sus datos, o directamente oponerse al tratamiento, o ejercer el derecho a la portabilidad de los mismos. Todo ello, mediante escrito, acompañado de copia de documento oficial que le identifique, dirigido a Parque Científico y Tecnológico UPM Campus de Montegancedo, s/n. 28223 Pozuelo de Alarcón, MADRID (España). En caso de disconformidad con el tratamiento, también tiene derecho a presentar una reclamación ante la Agencia Española de Protección de Datos. También podrá oponerse a nuestros envíos de comunicaciones comerciales (Art.21.2 de la LSSI) a través de la siguiente dirección de correo electrónico: [e-mail]

[dev1644]

Thanks for the explanation @Ferparishuertas , for now, I am closing this issue since I am not encountering it after the fix.