Blocking call detected in MonoRateLimiter

Resilience4j version: 1.5.0 Java version: 13

Test project: reactive-client.zip

While evaluating resilience4j rate limiter for our reactive WebClient we observed a blocking call coming from RateLimiter.reservePermission

reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
	at jdk.internal.misc.Unsafe.park(Unsafe.java) ~[?:?]
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:359) ~[?:?]
	at io.github.resilience4j.ratelimiter.internal.AtomicRateLimiter.compareAndSet(AtomicRateLimiter.java:185) ~[resilience4j-ratelimiter-1.5.0.jar:1.5.0]
	at io.github.resilience4j.ratelimiter.internal.AtomicRateLimiter.updateStateWithBackOff(AtomicRateLimiter.java:163) ~[resilience4j-ratelimiter-1.5.0.jar:1.5.0]
	at io.github.resilience4j.ratelimiter.internal.AtomicRateLimiter.reservePermission(AtomicRateLimiter.java:127) ~[resilience4j-ratelimiter-1.5.0.jar:1.5.0]
	at io.github.resilience4j.ratelimiter.RateLimiter.reservePermission(RateLimiter.java:651) ~[resilience4j-ratelimiter-1.5.0.jar:1.5.0]
	at io.github.resilience4j.reactor.ratelimiter.operator.MonoRateLimiter.subscribe(MonoRateLimiter.java:39) ~[resilience4j-reactor-1.5.0.jar:1.5.0]
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.resubscribe(FluxRetryWhen.java:204) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onNext(FluxRetryWhen.java:250) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:147) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]

Here are our use cases. There are 2 basic patterns we are testing. In both cases we assume that application logic is trying to execute much more than it’s allowed by rate limiter.

• sequential task - continuously fetch data from the same external source using multiple independent polling tasks that share the same RateLimiter.

        private Flux<String> sequentialTask(TestWebClient client, int ctx) {
		return client.request(ctx)
				.expand(res -> client.request(ctx));
	}

• parallel task - continuously fetch data from multiple external sources in parallel using. For example, there are multiple users and we want to fetch some portion of data for every user running multiple ‘tasks’ in parallel and then repeat the process.

	private Flux<String> parallelTask(TestWebClient client) {
		return Flux.defer(ReactiveApplication::getUsers)
				.flatMapDelayError(client::request, 10, 1)
				.repeat();
	}

	private Flux<Integer> getUsers() {
		return Flux.range(1, 100);
	}

Web client

	public Mono<String> request(int context) {
		return webClient.get()
				.uri(uriBuilder -> uriBuilder
						.path("/test")
						.queryParam("ctx", context)
						.build()
				)
				.exchange()
				.transformDeferred(RateLimiterOperator.of(rateLimiter))
				.flatMap(clientResponse -> {
					if (clientResponse.statusCode() == HttpStatus.OK) {
						return clientResponse.bodyToMono(String.class);
					} else {
						return Mono.error(new HttpClientErrorException(clientResponse.statusCode(), clientResponse.toString()));
					}
				})
				.retryWhen(Retry
						.fixedDelay(Long.MAX_VALUE, Duration.ofMillis(5000))
						.doBeforeRetry(retrySignal -> log.warn(retrySignal.toString(), retrySignal.failure()))
				);
	}

	private WebClient webClient(String baseUrl) {
		return WebClient.builder()
				.baseUrl(baseUrl)
				.filter(logRequest())
				.filter(logResponse())
				.build();
	}

	private RateLimiter rateLimiter() {
		RateLimiterConfig config = RateLimiterConfig.custom()
				.limitRefreshPeriod(Duration.ofSeconds(1))
				.limitForPeriod(1)
				.timeoutDuration(Duration.ofSeconds(1))
				.build();

		RateLimiter rateLimiter = RateLimiterRegistry
				.of(config)
				.rateLimiter("web-client");

		return rateLimiter;
	}

[storozhukBM]

@amrynsky @RobWin Yes, this backoff is here to resolve some atomic swap under big contention issues. Our tests show that is increases throughput under contention up to 2 times. On my machine, it goes from 8 permission acquires per microsecond to 14, on machines with more cores gains are even more significant, but this is only necessary for Java 8, starting from version 9 it is already fixed in the JVM itself. This parkNanos call will not actually “block” your thread, but razer gives OS scheduler a hint to run some other thread else instead and return to it as soon as possible. I think it is totally fine to add our AtomicRateLimiter.compareAndSet to BlockHound allow-list. If you take a look at https://github.com/reactor/BlockHound/blob/master/docs/tips.md#how-to-select-what-to-whitelist you can see that our method totally qualifies with this documentation. It does not call any user-provided code, and we know how it will behave and that it will not block worker thread for any prolonged time but would just reschedule itself, and we can safely whitelist it.

[amrynsky]

Added compareAndSet to whitelist. Here is my configuration for reference

BlockHound.install(builder -> builder
		.allowBlockingCallsInside(io.github.resilience4j.ratelimiter.internal.AtomicRateLimiter.class.getName(), "compareAndSet")
);

@storozhukBM @RobWin thanks for your help