Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Sending logs to Grafana Cloud via JavaHttpClient fails with "Received RST_STREAM: Protocol error" on JDK 11.0.14

See original GitHub issue

On JDK 11.0.13 everything worked fine, but after switching to JDK 11.0.14, the CI started to fail on Grafana Cloud integration test:

22:39:09,475 |-ERROR in com.github.loki4j.client.pipeline.DefaultPipeline@502efa63 - Error while sending Batch #32fa6944ba (60,813 bytes) to Loki (https://logs-prod-us-central1.grafana.net/loki/api/v1/push) java.io.IOException: Received RST_STREAM: Protocol error
    	at java.io.IOException: Received RST_STREAM: Protocol error
    	at 	at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:565)
    	at 	at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    	at 	at com.github.loki4j.client.http.JavaHttpClient.send(JavaHttpClient.java:66)
    	at 	at com.github.loki4j.client.pipeline.DefaultPipeline.sendBatch(DefaultPipeline.java:263)
    	at 	at com.github.loki4j.client.pipeline.DefaultPipeline.sendStep(DefaultPipeline.java:249)
    	at 	at com.github.loki4j.client.pipeline.DefaultPipeline.runSendLoop(DefaultPipeline.java:183)
    	at 	at com.github.loki4j.client.pipeline.DefaultPipeline.lambda$start$2(DefaultPipeline.java:108)
    	at 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    	at 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    	at 	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: java.io.IOException: Received RST_STREAM: Protocol error
    	at 	at java.net.http/jdk.internal.net.http.Stream.handleReset(Stream.java:504)
    	at 	at java.net.http/jdk.internal.net.http.Stream.schedule(Stream.java:178)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    	at 	at java.net.http/jdk.internal.net.http.Stream.receiveResetFrame(Stream.java:336)
    	at 	at java.net.http/jdk.internal.net.http.Stream.incoming_reset(Stream.java:482)
    	at 	at java.net.http/jdk.internal.net.http.Stream.otherFrame(Stream.java:418)
    	at 	at java.net.http/jdk.internal.net.http.Stream.incoming(Stream.java:408)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection.processFrame(Http2Connection.java:801)
    	at 	at java.net.http/jdk.internal.net.http.frame.FramesDecoder.decode(FramesDecoder.java:155)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection$FramesController.processReceivedData(Http2Connection.java:232)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection.asyncReceive(Http2Connection.java:663)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.processQueue(Http2Connection.java:1292)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.runOrSchedule(Http2Connection.java:1310)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1336)
    	at 	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1270)
    	at 	at java.net.http/jdk.internal.net.http.common.SSLTube$DelegateWrapper.onNext(SSLTube.java:202)
    	at 	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:484)
    	at 	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:287)
    	at 	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:318)
    	at 	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:261)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    	at 	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:234)
    	at 	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:200)
    	at 	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:403)
    	at 	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:264)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    	at 	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    	at 	... 3 common frames omitted

I guess the same behavior is likely to reproduce on JDK 17.0.2 (not available on CI yet).

Apart from that, log query code that is used for tests only started to fail with timeouts.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
nehaevcommented, Feb 12, 2022

Hi @abouquet! I’ve published the fix in v1.3.2. Please upgrade.

0reactions
abouquetcommented, Feb 14, 2022

Hi @abouquet! I’ve published the fix in v1.3.2. Please upgrade.

Thanks ! Will try it asap 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting Grafana Agent | Grafana Cloud documentation
If the agent is running but Check Connection is still failing, try retrieving logs for your agent by running kubectl -ndefault logs daemonset/grafana-agent...
Read more >
Unable to find traces | Grafana Tempo documentation
Tracing records not getting picked up to send to Tempo by the internal sampler. Application is running inside docker and sending traces to...
Read more >
Troubleshooting | Grafana Loki documentation
This error can appear in Grafana when Loki is added as a datasource, indicating that although Grafana has connected to Loki, Loki hasn't...
Read more >
Troubleshooting | Grafana documentation
This page lists some tools and advice to help troubleshoot common Grafana issues. Troubleshoot with logs. If you encounter an error or problem,...
Read more >
Troubleshooting Promtail - Grafana Loki documentation
Troubleshooting Promtail This document describes known failure modes of ... Promtail supports piping data for sending logs to Loki (via the flag --stdin...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Size (ratio) of image preview is incorrect

Next page

High idle CPU usage