Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Thread locked after stress test

See original GitHub issue

Describe the bug (Describe the problem clearly and concisely.)

I did a simple stress test，then the program keeps giving alarm log.

21:59:37 WARNING [io.ve.co.im.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-9,5,main]=Thread[vert.x-worker-thread-9,5,main] has been blocked for 3382564 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at io.quarkus.resteasy.runtime.standalone.VertxBlockingOutput.awaitWriteable(VertxBlockingOutput.java:111)
        at io.quarkus.resteasy.runtime.standalone.VertxBlockingOutput.write(VertxBlockingOutput.java:71)
        at io.quarkus.resteasy.runtime.standalone.VertxHttpResponse.writeBlocking(VertxHttpResponse.java:179)
        at io.quarkus.resteasy.runtime.standalone.VertxOutputStream.write(VertxOutputStream.java:69)
        at io.quarkus.resteasy.runtime.standalone.VertxOutputStream.write(VertxOutputStream.java:42)
        at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:64)
        at org.jboss.resteasy.util.DelegatingOutputStream.write(DelegatingOutputStream.java:42)
        at org.jboss.resteasy.plugins.providers.jsonb.JsonBindingProvider.writeTo(JsonBindingProvider.java:137)
        at org.jboss.resteasy.core.interception.jaxrs.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:193)
        at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:64)
        at org.jboss.resteasy.core.interception.jaxrs.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:155)
        at org.jboss.resteasy.core.ServerResponseWriter.lambda$writeNomapResponse$2(ServerResponseWriter.java:156)
        at org.jboss.resteasy.core.ServerResponseWriter$$Lambda$417/1780592217.run(Unknown Source)
        at org.jboss.resteasy.core.interception.jaxrs.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:404)
        at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:232)
        at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:97)
        at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:70)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:578)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:508)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:252)
        at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$332/1061176646.run(Unknown Source)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
        at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$338/1804746610.get(Unknown Source)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:156)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:238)
        at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:109)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatchRequestContext(VertxRequestHandler.java:84)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.lambda$handle$0(VertxRequestHandler.java:71)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$$Lambda$313/2064123549.handle(Unknown Source)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316)
        at io.vertx.core.impl.ContextImpl$$Lambda$327/865803395.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

After the test ,All requests are unresponsive . jstack log:

"mysql-cj-abandoned-connection-cleanup" #54 daemon prio=5 os_prio=0 tid=0x00007ff55c042800 nid=0xfaa in Object.wait() [0x00007ff60e3e8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x00000006ca58bd08> (a java.lang.ref.ReferenceQueue$Lock)
	at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:85)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x00000006ca58b8f8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"vert.x-worker-thread-12" #53 prio=5 os_prio=0 tid=0x00007ff58c007000 nid=0xfa9 in Object.wait() [0x00007ff60e8e8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at io.quarkus.resteasy.runtime.standalone.VertxBlockingOutput.awaitWriteable(VertxBlockingOutput.java:111)
	at io.quarkus.resteasy.runtime.standalone.VertxBlockingOutput.write(VertxBlockingOutput.java:71)
	- locked <0x00000006c8739c58> (a io.vertx.core.http.impl.Http1xServerConnection)
	at io.quarkus.resteasy.runtime.standalone.VertxHttpResponse.writeBlocking(VertxHttpResponse.java:179)
	at io.quarkus.resteasy.runtime.standalone.VertxOutputStream.write(VertxOutputStream.java:69)
	at io.quarkus.resteasy.runtime.standalone.VertxOutputStream.write(VertxOutputStream.java:42)
	at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:64)
	at org.jboss.resteasy.util.DelegatingOutputStream.write(DelegatingOutputStream.java:42)
	at org.jboss.resteasy.plugins.providers.jsonb.JsonBindingProvider.writeTo(JsonBindingProvider.java:137)
	at org.jboss.resteasy.core.interception.jaxrs.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:193)
	at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:64)
	at org.jboss.resteasy.core.interception.jaxrs.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:155)
	at org.jboss.resteasy.core.ServerResponseWriter.lambda$writeNomapResponse$2(ServerResponseWriter.java:156)
	at org.jboss.resteasy.core.ServerResponseWriter$$Lambda$417/1780592217.run(Unknown Source)
	at org.jboss.resteasy.core.interception.jaxrs.ContainerResponseContextImpl.filter(ContainerResponseContextImpl.java:404)
	- locked <0x00000007bea82098> (a org.jboss.resteasy.core.interception.jaxrs.ContainerResponseContextImpl)
	at org.jboss.resteasy.core.ServerResponseWriter.executeFilters(ServerResponseWriter.java:232)
	at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:97)
	at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:70)
	at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:578)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:508)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:252)
	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$332/1061176646.run(Unknown Source)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
	at org.jboss.resteasy.core.SynchronousDispatcher$$Lambda$338/1804746610.get(Unknown Source)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
	- locked <0x00000007bea82130> (a org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext)
	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:156)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:238)
	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:109)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatchRequestContext(VertxRequestHandler.java:84)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.lambda$handle$0(VertxRequestHandler.java:71)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$$Lambda$313/2064123549.handle(Unknown Source)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316)
	at io.vertx.core.impl.ContextImpl$$Lambda$327/865803395.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x00000006c8863b08> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Expected behavior (Describe the expected behavior clearly and concisely.)

Actual behavior (Describe the actual behavior clearly and concisely.)

To Reproduce Steps to reproduce the behavior:

  1. My program was modified from hibernate-orm-panache-resteasy in quickstart. I add a restful service returns all records of the table，there are 800 records in the table.
    @GET
    public List<Student> findAll() {
        List<Student> list = Student.findAll().list();
        if (list == null) {
            throw new WebApplicationException("Fruit  does not exist.", 404);
        }
        return list;
    }
  1. Then run in JVM mode.
  2. Did a simple stress test .
wrk -t12 -c2000 -d30s  http://10.32.1.***:8888/rest/students/

Configuration

# Add your application.properties here, if applicable.
quarkus.datasource.url=jdbc:mysql://10.32.1.***:3306/xyz
quarkus.datasource.driver=com.mysql.cj.jdbc.Driver

    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-jdbc-mysql</artifactId>
    </dependency>

Screenshots (If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

  • Output of uname -a or ver:
    Linux ***** 3.10.0-862.3.3.el7.x86_64 #1 SMP Fri Jun 15 04:15:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • Output of java -version: java version “1.8.0_201” Java™ SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot™ 64-Bit Server VM (build 25.201-b09, mixed mode)

  • GraalVM version (if different from Java):

  • Quarkus version or git rev: 1.0.0.CR1

Additional context

The full jstack dump info jstack-l.txt

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:15 (13 by maintainers)

github_iconTop GitHub Comments

1reaction
johnaoharacommented, Nov 13, 2019

I looks like there is a race condition in request.response().drainHandler(). I have opened a pr #5451.

1reaction
geoandcommented, Nov 13, 2019

I just ran it my reproducer against master and it worked just fine.

Read more comments on GitHub >

github_iconTop Results From Across the Web

WorKManager STUCK Thread Making Server Health To ...
Symptoms. When attempting a stress test more than 400 threads go to Hogging and after a few minutes to STUCK state. Also there...
Read more >
HnD | Threading issue - stress test, Page 1 - LLBLGen Pro
I've just written a small test app that generates 10 threads, ... rows and after the first 50 SQL server decides to escalate...
Read more >
Vert.x threads block after brief load testing #4173 - GitHub
A request is trying to load a file, the static handler isn't blocking anymore but the file resolver is locking trying to find...
Read more >
Close Those Loopholes: Stress-Test those Stored Procedures
A few suggestions: 1. Make sure read committed snapshot is not enabled. 2. Try running on a box with more CPUs. My development...
Read more >
Multi-threading locks test - Stack Overflow
It's a C# code derived from a C++/CLI Code which has locks in the C++ level... · @Yochai, so you want to stress...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Native image build error on Win10

Next page

quarkus.http.cors.origins=* should be supported