question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BlockHound integration causes blocking on netty FastThreadLocalRunnable

See original GitHub issue

Expected behavior

Other applications with BlockHound integration shouldn’t fail with internal netty blocking on FastThreadLocalRunnable.

Actual behavior

BlockHound integration added the following https://github.com/netty/netty/pull/9687/files#diff-2652ecd7677e7e42648c274b05543377R75 marking FastThreadLocalRunnable as blocking. Spring Cloud Gateway also has BlockHound integration and it causes the following error

java.lang.Error: sun.misc.Unsafe#park
	at reactor.blockhound.integration.CustomBlockHoundIntegration.lambda$applyTo$0(CustomBlockHoundIntegration.java:29)
	at reactor.blockhound.BlockHound$Builder.lambda$install$6(BlockHound.java:318)
	at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:46)
	at sun.misc.Unsafe.park(Unsafe.java)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:95)
	at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:239)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

I had to add an exception to pass

 builder.allowBlockingCallsInside(
		"io.netty.util.concurrent.FastThreadLocalRunnable", "run");

builder.allowBlockingCallsInside(“io.netty.util.concurrent.GlobalEventExecutor”, “takeTask”); didn’t work because of NPE exceptions that resulted.

i.n.u.c.SingleThreadEventExecutor : Unexpected exception from an event executor: java.lang.IllegalMonitorStateException: null
 at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151) ~[na:na]
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261) ~[na:na]
 at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457) ~[na:na]
 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:474) ~[na:na]
 at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:256) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
 at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:64) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.45.Final.jar:4.1.45.Final]
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.45.Final.jar:4.1.45.Final]
 at java.lang.Thread.run(Thread.java:748) 

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

It’s not minimal, but https://github.com/spring-cloud/spring-cloud-gateway/tree/2.1.x

./mvnw clean install.

Netty version

4.1.44.final

JVM version (e.g. java -version)

openjdk version “1.8.0_232” OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)

OS version (e.g. uname -a)

Linux sgibb-dell 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

/cc @bsideup @violetagg

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:2
  • Comments:6 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
violetaggcommented, Feb 11, 2020

@spencergibb I tried spring-cloud-gateway build with https://github.com/netty/netty/pull/10020/files#diff-2652ecd7677e7e42648c274b05543377R78-R85 and I think it is OK now. Can you check with this configuration?

@normanmaurer @bsideup Can you check PR #10020

0reactions
normanmaurercommented, Feb 7, 2020

@bsideup cool… I think @spencergibb said that whitelist GlobalEventExecutor#takeTask did not work in his report.

Read more comments on GitHub >

github_iconTop Results From Across the Web

reactor/BlockHound - Gitter
Hi guys - should all Jackson ObjectMapper work be classed as blocking (i.e. off the Netty thread)?. Scott ...
Read more >
Blocking call caught by BlockHound on reactive MongoDB ...
Issue comes from netty concurrent method with fast access to FastThreadLocal variables. You need to allow blocking method.
Read more >
Having blocking operations in Reactive application using ...
I am using blockhound to detect blocking calls. This happens only when concurrency above 100. Any suggestions on how to resolve this error?...
Read more >
BlockHound: detect blocking calls in Reactive code before it's ...
ServiceLoader to load all implementations of the BlockHoundIntegration interface that are listed inside the META-INF/services/reactor.blockhound.integration.
Read more >
Why do we see 1 blocked thread? - yCrash Answers
What is causing one blocked thread in this case? and ther are many Timed_Waiting threads, how do we anlayse on what are they...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found