Poor Consumer Test Performance on Windows

Consumer tests with the new Rust FFI have very poor performance when calling the mock server via HttpClient on Windows. This is quite a large regression on the Ruby backend performance, and one of the key benefits of the Rust FFI backend is supposed to be increased performance.

Demonstration of performance

It typically takes around 2 seconds for the HttpResponse to come back, which means every consumer test takes a minimum of 2 seconds. It’s not uncommon to have dozens of consumer tests per solution, and testing libraries typically won’t run tests within the same file/collection in parallel, and so it would result in the consumer test run taking minutes to complete.

Here’s an example:

And with some logging added to show where the problem happen:

Only on Windows

It’s worth noting that this only seems to happen on Windows. On Linux the consumer tests run extremely quickly. Check recent CI runs to confirm, but typically the Ubuntu run finishes in a fraction of the time of the Windows run. I have confirmed locally that the tests are extremely slow on my Windows machine as well, so it’s not some issue with GitHub Actions.

Replace the Pact Mock Server with an echo server

To confirm that it’s not caused by HttpClient itself, I ran a local Mock Server, like this:

docker run --rm -it -p 1080:1080 mockserver/mockserver

I set up the following hard-coded expectation (which is one of the responses expected by one of the example consumer tests) and hard-coded the same example consumer test to use the echo server on http://localhost:1080 instead of the Pact Mock Server:

curl -v -X PUT "http://localhost:1080/mockserver/expectation" -d '{
    "httpRequest": {
        "path": "/events"
    },
    "httpResponse": {
        "statusCode": 200,
        "headers": {
            "content-type": ["application/json"]
        },
        "body": [{
            "eventId": "45d80d13-d5a2-48d7-8353-cbb4c0eaabf5",
            "eventType": "SearchView",
            "timestamp": "2014-06-30T01:37:41.0660548"
        }, {
            "eventId": "83f9262f-28f1-4703-ab1a-8cfd9e8249c9",
            "eventType": "DetailsView",
            "timestamp": "2014-06-30T01:37:52.2618864"
        }, {
            "eventId": "3e83a96b-2a0c-49b1-9959-26df23f83aeb",
            "eventType": "SearchView",
            "timestamp": "2014-06-30T01:38:00.8518952"
        }]
    }
}'

The test executed in 131ms as I’d expect, with only 52ms of that being the HTTP request, so that proves that it’s caused by the Pact Mock Server:

19:28:43.812 - Finished setup
19:28:43.824 - Starting verify
19:28:43.835 - Starting get events
19:28:43.836 -     Enter GetAllEvents
19:28:43.837 -     Sending request...
19:28:43.889 -     Performed request
19:28:43.889 -     Response is OK
19:28:43.890 -     Read content
19:28:43.914 -     Deserialised response
19:28:43.915 -     Started Dispose
19:28:43.915 -     Finished Dispose
19:28:43.915 - Starting assert
19:28:43.943 - Finished verify

[bethesque]

I had to laugh at “This is quite a large regression on the Ruby backend performance”, because this is a sentence nobody has ever said before 😆

[adamrodger]

This issue can be closed now since the linked PR fixes it (referencing 127.0.0.1 explicitly instead of localhost), but I’ll leave the related upstream issue open because the Pact mock server should probably also listen on the IPV6 loopback.