Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

HTTP driver cannot resume work after server returns HTTP GO_AWAY

See original GitHub issue

Describe the bug While nosqlbench runs with the HTTP driver against Datastax Astra, the astra cluster will periodically issue a GO_AWAY, to tell the client to stop using its current connection and open up a new one. However, nosqlbench interprets the GO_AWAY signal as an error that should stop the scenario from running to completion, and it stops abruptly instead with an error:

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: /192.168.0.131:64152: GOAWAY received
	at io.nosqlbench.engine.api.activityapi.errorhandling.modular.handlers.StopErrorHandler.handleError(StopErrorHandler.java:16)
	at io.nosqlbench.engine.api.activityapi.errorhandling.modular.NBErrorHandler.handleError(NBErrorHandler.java:52)
	at io.nosqlbench.activitytype.http.HttpAction.runCycle(HttpAction.java:130)
	at io.nosqlbench.engine.api.activityimpl.motor.CoreMotor.run(CoreMotor.java:405)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: /192.168.0.131:64152: GOAWAY received
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2086)
	at io.nosqlbench.activitytype.http.HttpAction.runCycle(HttpAction.java:94)
	... 4 more
Caused by: java.io.IOException: /192.168.0.131:64152: GOAWAY received
	at java.net.http/jdk.internal.net.http.Http2Connection.handleGoAway(Http2Connection.java:999)
	at java.net.http/jdk.internal.net.http.Http2Connection.handleConnectionFrame(Http2Connection.java:867)
	at java.net.http/jdk.internal.net.http.Http2Connection.processFrame(Http2Connection.java:738)
	at java.net.http/jdk.internal.net.http.frame.FramesDecoder.decode(FramesDecoder.java:155)
	at java.net.http/jdk.internal.net.http.Http2Connection$FramesController.processReceivedData(Http2Connection.java:232)
	at java.net.http/jdk.internal.net.http.Http2Connection.asyncReceive(Http2Connection.java:663)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.processQueue(Http2Connection.java:1289)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.runOrSchedule(Http2Connection.java:1307)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1333)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1267)
	at java.net.http/jdk.internal.net.http.common.SSLTube$DelegateWrapper.onNext(SSLTube.java:210)
	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:492)
	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:295)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:316)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:259)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:232)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:198)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:444)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:268)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	... 3 more
 401294 WARN  [scenarios:001] SCENARIO     Error in scenario, shutting down.
 401304 ERROR [main] ERRORHANDLER Error from driver or included library: java.lang.RuntimeException: javax.script.ScriptException: java.lang.RuntimeException: Error in activity thread activities/documents-api/http-docsapi-crud-basic.yaml:001
java.util.concurrent.ExecutionException: java.lang.RuntimeException: javax.script.ScriptException: java.lang.RuntimeException: Error in activity thread activities/documents-api/http-docsapi-crud-basic.yaml:001
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at io.nosqlbench.engine.core.script.ScenariosExecutor.getAsyncResultStatus(ScenariosExecutor.java:150)
	at io.nosqlbench.engine.core.script.ScenariosExecutor.awaitAllResults(ScenariosExecutor.java:115)
	at io.nosqlbench.engine.core.script.ScenariosExecutor.awaitAllResults(ScenariosExecutor.java:73)
	at io.nosqlbench.engine.cli.NBCLI.run(NBCLI.java:406)
	at io.nosqlbench.engine.cli.NBCLI.main(NBCLI.java:70)
Caused by: java.lang.RuntimeException: javax.script.ScriptException: java.lang.RuntimeException: Error in activity thread activities/documents-api/http-docsapi-crud-basic.yaml:001
	at io.nosqlbench.engine.core.script.Scenario.runScenario(Scenario.java:319)
	at io.nosqlbench.engine.core.script.Scenario.call(Scenario.java:374)
	at io.nosqlbench.engine.core.script.Scenario.call(Scenario.java:56)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: javax.script.ScriptException: java.lang.RuntimeException: Error in activity thread activities/documents-api/http-docsapi-crud-basic.yaml:001
	at io.nosqlbench.engine.core.lifecycle.ActivityExecutor.notifyException(ActivityExecutor.java:526)
	at io.nosqlbench.engine.core.lifecycle.ActivityExceptionHandler.uncaughtException(ActivityExceptionHandler.java:30)
	at java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:1993)
Caused by: java.lang.RuntimeException: Error in activity thread activities/documents-api/http-docsapi-crud-basic.yaml:001
	... 3 more
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: /192.168.0.131:64152: GOAWAY received
	at io.nosqlbench.engine.api.activityapi.errorhandling.modular.handlers.StopErrorHandler.handleError(StopErrorHandler.java:16)
	at io.nosqlbench.engine.api.activityapi.errorhandling.modular.NBErrorHandler.handleError(NBErrorHandler.java:52)
	at io.nosqlbench.activitytype.http.HttpAction.runCycle(HttpAction.java:130)
	at io.nosqlbench.engine.api.activityimpl.motor.CoreMotor.run(CoreMotor.java:405)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: /192.168.0.131:64152: GOAWAY received
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2086)
	at io.nosqlbench.activitytype.http.HttpAction.runCycle(HttpAction.java:94)
	... 4 more
Caused by: java.io.IOException: /192.168.0.131:64152: GOAWAY received
	at java.net.http/jdk.internal.net.http.Http2Connection.handleGoAway(Http2Connection.java:999)
	at java.net.http/jdk.internal.net.http.Http2Connection.handleConnectionFrame(Http2Connection.java:867)
	at java.net.http/jdk.internal.net.http.Http2Connection.processFrame(Http2Connection.java:738)
	at java.net.http/jdk.internal.net.http.frame.FramesDecoder.decode(FramesDecoder.java:155)
	at java.net.http/jdk.internal.net.http.Http2Connection$FramesController.processReceivedData(Http2Connection.java:232)
	at java.net.http/jdk.internal.net.http.Http2Connection.asyncReceive(Http2Connection.java:663)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.processQueue(Http2Connection.java:1289)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.runOrSchedule(Http2Connection.java:1307)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1333)
	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Http2Connection.java:1267)
	at java.net.http/jdk.internal.net.http.common.SSLTube$DelegateWrapper.onNext(SSLTube.java:210)
	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:492)
	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(SSLTube.java:295)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:316)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:259)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:232)
	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:198)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:444)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:268)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	... 3 more

To Reproduce Steps to reproduce the behavior:

  1. Run java -jar nb.jar run driver=http yaml=documents-api/http-docsapi-crud-basic.yaml protocol=https path_prefix='/api/rest' stargate_port=443 stargate_host='<valid astra host>' auth_token='<valid auth token>' tags=phase:main,type:write cycles=20000 threads=1

What was Expected Workload runs to completion

Additional context Add any other context about the problem here. OS: Windows, Linux (distribution), macOS … environment: k8s, docker, … version info (./nb --version or java -jar nb.jar --version)

Screenshots, if applicable If applicable, add screenshots to help explain your problem.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
jshookcommented, Jan 12, 2022

@EricBorczuk It also occurs to me that the retry error handler might not be sufficient. In that case we may have to add an active reconnect capability to the HTTP driver, or at least something to invalidate the current client so that it will reinit as needed.

0reactions
mpenickcommented, Jun 8, 2022

@jshook In practice, GOAWAY is common (in fact many HTTP/2.0 servers send this every N requests for a persistent connection see http://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests and http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests). The problem here is that java.net.http.HttpClient isn’t a very good HTTP/2.0 client. A good client like OkHttp (https://square.github.io/okhttp/) will just restart a new connection and send the any unprocessed requests on it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to handle HTTP/2 GOAWAY with HttpClient?
I tried to reproduce this problem with a local nginx server with the default HTTP/2 configuration: server { listen 443 http2 ssl; ...
Read more >
Chrome is not opening another HTTP2 connection when it ...
Example URL: Steps to reproduce the problem: This is an intermittent problem. It reproduces one out of five times. 1. Access some server...
Read more >
RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2)
HTTP /2 adds a new interaction mode whereby a server can push responses to a ... After sending the GOAWAY frame for an...
Read more >
Changelog
... to work with "data" and "data2"; tests/server/sws.c: change the HTTP ... Curl_send: return error when pre_receive_plain can't malloc ...
Read more >
RHSA-2022:5069 - Security Advisory - Red Hat 고객 포털
For OpenShift Container Platform 4.11 see the following documentation, ... has a failure with "server returned HTTP status 502 Bad Gateway" ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Problem with binding when variable is in a solr query.

Next page

v4.15.44 not compatible with C* 4.0 RC1