"Reply channel not set up yet" when 2 HTTP Citrus actions happen in parallel

Citrus Version 2.7.8

Expected behavior Test passes

Actual behavior Test fails

Test case description 2 logically identical async blocks with different test data, pseudo code:

async().actions(
  insert data 1 into DB,
  behaviour with HTTP receive() and respond() with HTTP 200 OK,
  DB check that a record with status "SUCCESS" exists for the test data 1
);

async().actions(
  insert data 2 into DB,
  behaviour with HTTP receive() and respond() with HTTP 200 OK,
  DB check that a record with status "SUCCESS" exists for the test data 2
);

SUT behavior:

• SUT has a background job which is triggered every 5 sec (on TEST env)
• The background job scans DB for new entries and starts processing them
• SUT then does HTTP POST with this new data as payload to an external system (Citrus HTTP mock server)
• The external system responds with some additional data (the mock replies with HTTP 200 OK)
• When SUT receives this data, the processing is finished and SUT then stores a record with status SUCCESS into DB (Citrus checks that DB has this entry at the end of the async blocks)

Note: one async block alone works but 2 blocks cause a failure:

12:57:06,068 DEBUG on.DefaultCorrelationManager| Finding correlated object for 'citrus_message_id = '9a369286-4824-4d6c-9eac-0bb12308d19a''
12:57:06,068 DEBUG           citrus.RetryLogger| Reply channel not set up yet - retrying in 500ms

It seems that Citrus cannot find the right HTTP channel for sending a response.

[progaddict]

Hi @svettwer !

I had some spare time to implement a super-duper-small example SUT and a couple of Citrus tests which reproduce this bug: https://github.com/progaddict/citrus-multiple-async-blocks

I hope this will help you guys to fixie-wixie the bug super-duper-fast 😆

[svettwer]

Thx to @progaddict and his SUT project, I was able to shrink the context of the issue a little bit further.

TL;DR It seems that multiple async container disturb each other in some way. It might be a race condition or a message overwriting in the message channel depending on the order the messages arrive.

Details I’ve created the following test cases:

public class TodoListIT extends TestNGCitrusTestRunner {

    @Autowired
    private HttpClient httpClient;

    @Autowired
    private HttpServer httpServer;

    @Test
    @CitrusTest
    public void testFailing() {

        async().actions(
                http(b -> b.client(httpClient)
                        .send()
                        .post("/foo")
                        .payload(createPayload("foo"))),

                http(b -> b.client(httpClient).
                        receive()
                        .response(HttpStatus.OK)),

                simulateBackend("foo")

        ).addErrorAction(echo("foo failed"));

        async().actions(
                http(b -> b.client(httpClient)
                        .send()
                        .post("/boo")
                        .payload(createPayload("boo"))),

                http(b -> b.client(httpClient)
                        .receive()
                        .response(HttpStatus.OK)),

                simulateBackend("boo")
        ).addErrorAction(echo("boo failed"));

    }

    @Test
    @CitrusTest
    public void testPassingAsyncParallelSequential() {

        parallel().actions(
                sequential().actions(
                        http(b -> b.client(httpClient)
                                .send()
                                .post("/foo")
                                .payload(createPayload("foo"))),

                        http(b -> b.client(httpClient)
                                .receive()
                                .response(HttpStatus.OK)),

                        simulateBackend("foo")
                )
        );

        async().actions(
                http(b -> b.client(httpClient)
                        .send()
                        .post("/boo")
                        .payload(createPayload("boo"))),

                http(b -> b.client(httpClient)
                        .receive()
                        .response(HttpStatus.OK)),

                simulateBackend("boo")
        );

    }

    @Test
    @CitrusTest
    public void testPassingParallelSequentialParallelSequential() {

        parallel().actions(
                sequential().actions(
                        http(b -> b.client(httpClient)
                                .send()
                                .post("/foo")
                                .payload(createPayload("foo"))),

                        http(b -> b.client(httpClient)
                                .receive()
                                .response(HttpStatus.OK)),
                        simulateBackend("foo")
                )
        );

        parallel().actions(
                sequential().actions(
                        http(b -> b.client(httpClient)
                                .send()
                                .post("/boo")
                                .payload(createPayload("boo"))),

                        http(b -> b.client(httpClient)
                                .receive()
                                .response(HttpStatus.OK)),
                        simulateBackend("boo")
                )
        );
    }

    private String createPayload(final String boo) {
        return String.format("{\"name\": \"%s\" }", boo);
    }

    private ApplyTestBehaviorAction simulateBackend(final String payload) {
        return applyBehavior(new AbstractTestBehavior(){

            @Override
            public void apply() {
                http(b -> b.server(httpServer)
                        .receive()
                        .post("/" + payload)
                        .selector(Collections.singletonMap("jsonPath:$.name", payload)));

                http(b -> b.server(httpServer)
                        .respond()
                        .status(HttpStatus.OK));
            }
        });
    }
}

with the config:

@Configuration
public class EndpointConfig {

    @Bean
    public HttpClient todoClient() {
        return CitrusEndpoints.http()
                .client()
                .requestUrl("http://localhost:8888")
                .build();
    }

    @Bean
    public HttpServer todoServer() {
        return CitrusEndpoints.http()
                .server()
                .port(8888)
                .autoStart(true)
                .build();
    }
}

The sample project configured the endpoints differently, with a higher timeout than the async container comes with. That’s why the error message is misleading and the endpoint is polling much longer as the container runs. This is especially important for #548.

I’ve reduced the high timeouts which leads to a different, more test action specific error message.

At the end, it seems like the multiple async container disturb each other. If you execute the failing test multiple times, you’ll notice that the issue occurs sometimes for foo and sometimes for boo. This could indicate a race condition or a message overwriting in the message channel depending on the order the messages arrive.

BR, Sven