Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Local and remote state disagreement
See original GitHub issueEnough stable system handles ~5 thousand WS requests per day and the 1%(~50 connections) connections are closing on an error with the next exception: during the resume process, we have a disagreement during the frames removing.
Did someone face this issue? Any idea what’s going on?
15:34:05.998 [reactor-http-epoll-3] ERROR reactor.core.publisher.Operators - Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalStateException: Local and remote state disagreement: need to remove additional 194 bytes, but cache is empty
Caused by: java.lang.IllegalStateException: Local and remote state disagreement: need to remove additional 194 bytes, but cache is empty
at io.rsocket.resume.InMemoryResumableFramesStore.releaseFrames(InMemoryResumableFramesStore.java:120)
at io.rsocket.resume.ServerRSocketSession.doResume(ServerRSocketSession.java:153)
at io.rsocket.resume.ServerRSocketSession.lambda$resumeWith$2(ServerRSocketSession.java:109)
at io.rsocket.resume.ServerRSocketSession.resumeWith(ServerRSocketSession.java:117)
at io.rsocket.core.ServerSetup$ResumableServerSetup.acceptRSocketResume(ServerSetup.java:137)
at io.rsocket.core.RSocketServer.acceptResume(RSocketServer.java:356)
at io.rsocket.core.RSocketServer.accept(RSocketServer.java:368)
at io.rsocket.core.RSocketServer.lambda$acceptor$0(RSocketServer.java:350)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
at com.company.mdc.config.MdcContextLifter.onNext(MdcConfig.kt:40)
at reactor.core.publisher.FluxFirstWithSignal$FirstEmittingSubscriber.onNext(FluxFirstWithSignal.java:329)
at com.company.mdc.config.MdcContextLifter.onNext(MdcConfig.kt:40)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:160)
at io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:114)
at io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19)
at com.company.mdc.config.MdcContextLifter.onNext(MdcConfig.kt:40)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:265)
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:371)
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:381)
at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:544)
at reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:161)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
@Bean
fun resumeCustomizer(): RSocketServerCustomizer {
return RSocketServerCustomizer {
val resume = Resume().cleanupStoreOnKeepAlive()
it.resume(resume)
}
}
Libs
java 11
kotlin 1.4.10
org.springframework.boot:spring-boot-starter-rsocket:2.4.0
-> io.rsocket:rsocket-core:1.1.0
-> io.rsocket:rsocket-transport-netty:1.1.0
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:12 (11 by maintainers)
Top Results From Across the Web
RSocket: Local and remote state disagreement - Stack Overflow
IllegalStateException: Local and remote state disagreement: need to remove additional 194 bytes, but cache is empty Caused by: java.lang.
Read more >Local and Remote State - DevOps with Terraform - CloudCasts
Local and Remote State. We start by inspecting the state file. This is how Terraform knows what state it is managing. It's not...
Read more >Local and remote state conflict on no changes #4154 - GitHub
Hi, I have stumbled upon issue with remote states. We use these for different TF deployments. Whereas usually it works, in one simple...
Read more >Backend Configuration - Terraform - HashiCorp Developer
The local backend configuration is different and entirely separate from the terraform.tfstate file that contains state data about your real-world infrastruture.
Read more >How to manage Terraform state - Gruntwork Blog
The default backend, which you've been using this entire time, is the local backend, which stores the state file on your local disk....
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@emax19 can you please enable debug logs for the following classes:
io.rsocket.resume.InMemoryResumableFramesStoreio.rsocket.resume.ClientRSocketSessionio.rsocket.resume.InMemoryResumableFramesStoreio.rsocket.resume.ResumableDuplexConnectionio.rsocket.resume.ServerRSocketSessionand share them with us (if possible) to see what is going on when the resumption fails.
Thanks, Oleh