Armeria Client no respond encountered

Armeria Client no respond encountered, would you please help look into this issue. Armeria version is 0.81.0

Background:

Service Provider: Thrift Based Service

service registration as below:

public static ThriftServiceRegistrationBean serviceRegistrationBean(Object serviceIfaceBean) {
    String path = null;
    Class<?> parent = serviceIfaceBean.getClass();
    for (Class<?> iface : parent.getInterfaces()) {
        if (iface.getName().endsWith("Iface")) {
            path = iface.getName();
            break;
        }
    }

    requireNonNull(path, "Thrift Service must implement Iface or AsyncIface");

    return new ThriftServiceRegistrationBean()
            .setServiceName(serviceIfaceBean.getClass().getName())
            .setService(THttpService.of(serviceIfaceBean))
            .setPath("/" + path)
            .setDecorators(LoggingService.newDecorator(), ServerAuthCheckDecorator.newDecorator());
}

Client:

client code as below:

public static <T> T generateClient(String endpoint, Class<T> clazz) {
    endpoint = String.format("tbinary+h2c://%s/%s", endpoint, clazz.getName());
    return build(endpoint, clazz);
}

private static <T> T build(String endpoint, Class<T> clazz) {
    ClientBuilder builder = new ClientBuilder(endpoint);
    builder.decorator(ClientAuthCheckDecorator.newDecorator());
    return builder.build(clazz);
}

The Problem

when service A call service B, by the help of the logging, we found that, the request didn’t reach up to service B, and the thread was hung up, client is waiting forever. here is the thread dump for the hung up thread.

"armeria-common-worker-epoll-2-2" #79 daemon prio=5 os_prio=0 tid=0x00007fad0c006000 nid=0x5f waiting on condition [0x00007facfa6ea000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000f252edf0> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at com.linecorp.armeria.client.thrift.THttpClientInvocationHandler.invokeClientMethod(THttpClientInvocationHandler.java:135)
	at com.linecorp.armeria.client.thrift.THttpClientInvocationHandler.invoke(THttpClientInvocationHandler.java:85)
	at com.sun.proxy.$Proxy68.getOrgThirdPartResConfig(Unknown Source)
	at com.hnair.phoenix.mgmt.services.project.domain.service.impl.ApplicationServiceImpl.sendCreatApplicationEvent(ApplicationServiceImpl.java:129)
	at com.hnair.phoenix.mgmt.services.project.domain.service.impl.ApplicationServiceImpl.createApplication(ApplicationServiceImpl.java:110)
	at com.hnair.phoenix.mgmt.services.project.domain.service.impl.ApplicationServiceImpl$$FastClassBySpringCGLIB$$94c5215e.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
	at com.hnair.phoenix.mgmt.services.project.domain.service.impl.ApplicationServiceImpl$$EnhancerBySpringCGLIB$$49a6a3f4.createApplication(<generated>)
	at com.hnair.phoenix.mgmt.services.project.interfaces.facade.impl.ApplicationServiceFacadeImpl.createApplication(ApplicationServiceFacadeImpl.java:54)
	at com.hnair.phoenix.mgmt.services.project.api.ApplicationServiceAsyncApi.createApplication(ApplicationServiceAsyncApi.java:68)
	at com.hnair.phoenix.services.project.v1.ApplicationService$AsyncProcessor$createApplication.start(ApplicationService.java:1342)
	at com.hnair.phoenix.services.project.v1.ApplicationService$AsyncProcessor$createApplication.start(ApplicationService.java:1280)
	at com.linecorp.armeria.server.thrift.ThriftCallService.invokeAsynchronously(ThriftCallService.java:162)
	at com.linecorp.armeria.server.thrift.ThriftCallService.invoke(ThriftCallService.java:145)
	at com.linecorp.armeria.server.thrift.ThriftCallService.serve(ThriftCallService.java:129)
	at com.linecorp.armeria.server.thrift.ThriftCallService.serve(ThriftCallService.java:52)
	at com.linecorp.armeria.server.thrift.THttpService.invoke(THttpService.java:605)
	at com.linecorp.armeria.server.thrift.THttpService.decodeAndInvoke(THttpService.java:580)
	at com.linecorp.armeria.server.thrift.THttpService.lambda$doPost$3(THttpService.java:434)
	at com.linecorp.armeria.server.thrift.THttpService$$Lambda$631/883285835.apply(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	at com.linecorp.armeria.common.HttpMessageAggregator.apply(HttpMessageAggregator.java:151)
	at com.linecorp.armeria.common.HttpMessageAggregator.apply(HttpMessageAggregator.java:36)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

The current workaround

At the moment, we setup the timeout for the client as following. It’s curious that why the request didn’t send out, and no exception occured, but the thread is still waiting for the response.

private static <T> T build(String endpoint, Class<T> clazz) {
    ClientBuilder builder = new ClientBuilder(endpoint);
    builder.defaultWriteTimeoutMillis(1000L)
        .defaultResponseTimeoutMillis(5000)
        .decorator(ClientAuthCheckDecorator.newDecorator());
    return builder.build(clazz);
}       

[eonezhang]

after discussion with @hyangtack , we infer the root cause is, the blocking api call used up the armeria worker threads, so there is no thread to send request to service provider, then this issue happened. thanks @hyangtack for your kindness support.

[eonezhang]

Spring Bean Configuration

@Bean
public ProjectService.AsyncIface asyncProjectService() {
    return generateGroupClient(zkAddress, projectServiceName, ProjectService.AsyncIface.class);
}


public static <T> T generateGroupClient(String zkAddress, String serviceName, Class<T> clazz) {
    String groupName = UUID.randomUUID().toString().replaceAll("-", "");
    ZooKeeperEndpointGroup group = new ZooKeeperEndpointGroup(zkAddress, serviceName, 10000);
    EndpointGroupRegistry.register(groupName, group, EndpointSelectionStrategy.WEIGHTED_ROUND_ROBIN);

    String endpoint = String.format("tbinary+http://group:%s/%s", groupName, clazz.getName());
    return build(endpoint, clazz);
}

private static <T> T build(String endpoint, Class<T> clazz) {
    ClientBuilder builder = new ClientBuilder(endpoint);
    builder.decorator(ClientAuthCheckDecorator.newDecorator());
    return builder.build(clazz);
}

Spring MVC Controller to use the ProjectService.AsyncIface

after call asyncProjectSrv.isNameAvailable, the service provider didn’t receive the request(here is the problem)

@Autowired
private ProjectService.AsyncIface asyncProjectSrv;

@RequestMapping(value = "/isnameavailable", method = RequestMethod.GET)
@ResponseBody
public Future<ProjectNameIsAvResponse> isNameAvailable(
        @RequestParam(value = "name", required = true) String name,
        HttpServletRequest req)
        throws TException {
    logger.info("enter isNameAvailable");
    CompletableFuture<ProjectNameIsAvResponse> retFuture = new CompletableFuture<>();
    ProjectNameIsAvResponse res = new ProjectNameIsAvResponse();
    res.setResultCode(BaseResponse.RESULT_CODE_SUCCESS);
    
    ThriftCompletableFuture<Boolean> isNameAvailableFuture = new ThriftCompletableFuture<>();
    asyncProjectSrv.isNameAvailable(SessionUtils.getOrganization(req), name, isNameAvailableFuture);
    
    isNameAvailableFuture.thenAccept(isAvailable -> {
        res.setIsAvailable(isAvailable);
        if (!isAvailable) {
            res.setDes("项目名称重复");
        }
        res.setResultMessage("isNameAvailable success");
        retFuture.complete(res);
    }).exceptionally(cause -> {
        res.setResultCode(BaseResponse.RESULT_CODE_ERROR);
        res.setResultMessage("isNameAvailable error:" + cause.getMessage());
        retFuture.complete(res);
        return null;
    });
    return retFuture;
}

A client decorator to fill context info, and pass it to the service provider.

@Slf4j
public class ClientAuthCheckDecorator extends SimpleDecoratingClient<HttpRequest, HttpResponse> {

    /**
     *  Creates a new auth {@link Client} decorator using the specified {@link ClientAuthCheckDecorator} instance.
     * @return
     */
    public static Function<Client<HttpRequest, HttpResponse>, ClientAuthCheckDecorator> newDecorator() {
        return client -> new ClientAuthCheckDecorator((client));
    }

    /**
     * Creates a new instance that decorates the specified {@link Client}.
     * @param delegate client
     */
    protected ClientAuthCheckDecorator(Client<HttpRequest, HttpResponse> delegate) {
        super(delegate);
    }

    @Override
    public HttpResponse execute(ClientRequestContext ctx, HttpRequest req) throws Exception {
        try {
            auditRequest(req);
        } catch (Throwable th) {
            log.error("client sent context error", th);
        }
        return delegate().execute(ctx, req);
    }

    private void auditRequest(HttpRequest req) {
        Context ctx = Context.current();
        req.headers()
           .set(string("actor"), ctx == null ? "SYSTEM" : ctx.getActorId())
           .set(string("clientIp"), ctx == null ? "none" : ctx.getClientIp())
           .set(string("ip"), ServerUtil.hostIP());
    }
}