Rest client lambda crash

Describe the bug

I have a strange issue with lambda - its crashing, I found that it happens when I’m using the rest client for long requests. 40-60 sec.

Expected behavior

Lambda should not crash.

Actual behavior

`00:31:04.650 INFO (Quarkus Main Thread) - [io.quarkus] - Installed features: [amazon-dynamodb, amazon-lambda, amazon-s3, amazon-sqs, cdi, hibernate-validator, rest-client, rest-client-jackson, resteasy, resteasy-jackson, resteasy-mutiny, smallrye-context-propagation, vertx] 00:31:25.574 INFO (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.Exporter] - Start 00:31:26.589 INFO (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.ht.HttpService] - Fast all ok: [[com.github.storytime.lambda.backup.http.UserItem@67a4f028]] 00:32:31.111 INFO (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.ht.HttpService] - Slow all ok: [[com.github.storytime.lambda.backup.http.UserItem@681c16ea]] 00:32:31.111 INFO (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.Exporter] - Finish 00:33:12.350 ERROR (executor-thread-0) - [io.qu.ve.ht.ru.QuarkusErrorHandler] - HTTP Request to /lambda/2018-06-01/runtime/invocation/4291ca0a-07a8-4187-ac8a-42092a6091a2/response failed, error id: 565c7cd0-1439-4849-bff0-4ed311e55bed-1: javax.ws.rs.BadRequestException: RESTEASY003520: Malformed quality value. at org.jboss.resteasy.core.request.QualityValue.parseAsInteger(QualityValue.java:113) at org.jboss.resteasy.core.request.QualityValue.valueOf(QualityValue.java:40) at org.jboss.resteasy.core.request.AcceptHeaders.evaluateAcceptParameters(AcceptHeaders.java:292) at org.jboss.resteasy.core.request.AcceptHeaders.getMediaTypeQualityValues(AcceptHeaders.java:170) at org.jboss.resteasy.core.request.ServerDrivenNegotiation.setAcceptHeaders(ServerDrivenNegotiation.java:41) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.selectVariant(NotFoundExceptionMapper.java:386) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.respond(NotFoundExceptionMapper.java:244) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:225) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:53) at org.jboss.resteasy.core.ExceptionHandler.executeExactExceptionMapper(ExceptionHandler.java:64) at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:304) at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:258) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364) at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247) at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73) at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:151) at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:91) at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:564) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478) at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29) at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833)

00:33:12.359 ERROR (Lambda Thread (DEVELOPMENT)) - [io.qu.am.la.ru.AbstractLambdaPollLoop] - Failed to run lambda (DEVELOPMENT): java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/_lambda_/2018-06-01/runtime/invocation/4291ca0a-07a8-4187-ac8a-42092a6091a2/response at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1997) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop.postResponse(AbstractLambdaPollLoop.java:251) at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop$1.run(AbstractLambdaPollLoop.java:138) at java.base/java.lang.Thread.run(Thread.java:833)

00:33:13.431 ERROR (executor-thread-0) - [io.qu.ve.ht.ru.QuarkusErrorHandler] - HTTP Request to /lambda/2018-06-01/runtime/invocation/4291ca0a-07a8-4187-ac8a-42092a6091a2/error failed, error id: 565c7cd0-1439-4849-bff0-4ed311e55bed-2: javax.ws.rs.BadRequestException: RESTEASY003520: Malformed quality value. at org.jboss.resteasy.core.request.QualityValue.parseAsInteger(QualityValue.java:113) at org.jboss.resteasy.core.request.QualityValue.valueOf(QualityValue.java:40) at org.jboss.resteasy.core.request.AcceptHeaders.evaluateAcceptParameters(AcceptHeaders.java:292) at org.jboss.resteasy.core.request.AcceptHeaders.getMediaTypeQualityValues(AcceptHeaders.java:170) at org.jboss.resteasy.core.request.ServerDrivenNegotiation.setAcceptHeaders(ServerDrivenNegotiation.java:41) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.selectVariant(NotFoundExceptionMapper.java:386) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.respond(NotFoundExceptionMapper.java:244) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:225) at io.quarkus.resteasy.runtime.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:53) at org.jboss.resteasy.core.ExceptionHandler.executeExactExceptionMapper(ExceptionHandler.java:64) at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:304) at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:258) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364) at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247) at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73) at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:151) at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:91) at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:564) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478) at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29) at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833)

00:33:13.433 ERROR (Lambda Thread (DEVELOPMENT)) - [io.qu.am.la.ru.AbstractLambdaPollLoop] - Error running lambda (DEVELOPMENT): java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/_lambda_/2018-06-01/runtime/invocation/4291ca0a-07a8-4187-ac8a-42092a6091a2/error at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1997) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop.postError(AbstractLambdaPollLoop.java:274) at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop$1.run(AbstractLambdaPollLoop.java:149) at java.base/java.lang.Thread.run(Thread.java:833)

00:33:13.435 ERROR (Lambda Thread (DEVELOPMENT)) - [io.qu.ru.StartupContext] - Running a shutdown task failed [Error Occurred After Shutdown]: java.lang.IllegalStateException: Unable to unregister all message consumer methods at io.quarkus.vertx.runtime.VertxRecorder.unregisterMessageConsumers(VertxRecorder.java:198) at io.quarkus.vertx.runtime.VertxRecorder$1.run(VertxRecorder.java:57) at io.quarkus.runtime.StartupContext.runAllInReverseOrder(StartupContext.java:84) at io.quarkus.runtime.StartupContext.close(StartupContext.java:73) at io.quarkus.runner.ApplicationImpl.doStop(Unknown Source) at io.quarkus.runtime.Application.stop(Application.java:208) at io.quarkus.runtime.Application.stop(Application.java:155) at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop$1.run(AbstractLambdaPollLoop.java:160) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1048) at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230) Caused by: java.lang.InterruptedException

at io.quarkus.vertx.runtime.VertxRecorder.unregisterMessageConsumers(VertxRecorder.java:195)
... 8 more

00:33:13.451 INFO (Lambda Thread (DEVELOPMENT)) - [io.quarkus] - bug-report stopped in 0.017s `

How to Reproduce?

user my demo code: https://github.com/storytime/bug-report-rest

./gradlew quarkusDev --stacktrace

printf '{ "Records": [ { "messageId": "111", "receiptHandle": "MessageReceiptHandle", "body": "1111", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1523232000000", "SenderId": "123456789012", "ApproximateFirstReceiveTimestamp": "1523232000001" }, "messageAttributes": {}, "md5OfBody": "1111", "eventSource": "aws:sqs", "eventSourceARN": "arn:{partition}:sqs:{region}:1111:MyQueue", "awsRegion": "{region}" } ] }'| http --follow POST 'http://localhost:8080' \ Content-Type:'application/json'

Output of uname -a or ver

Darwin mac16.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64

Output of java -version

openjdk 17.0.4 2022-07-19 LTS OpenJDK Runtime Environment Corretto-17.0.4.8.1 (build 17.0.4+8-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.4.8.1 (build 17.0.4+8-LTS, mixed mode, sharing)

GraalVM version (if different from Java)

GraalVM Updater 22.2.0

Quarkus version or git rev

2.13.1.Final

Build tool (ie. output of mvnw --version or gradlew --version)

gradew 7.4

Additional information

Demo with issue: https://share.cleanshot.com/x0jSWl Demo not issue: https://share.cleanshot.com/fHJo6U

[geoand]

I am sure someone else will have a look 😃

[michalvavrik]

Hello @storytime , reproducer has both amazon-lambda and quarkus-resteasy-mutiny/jackson that both tries to use 8080, when I remove

implementation 'io.quarkus:quarkus-resteasy-jackson:2.13.1.Final'
implementation 'io.quarkus:quarkus-resteasy-mutiny:2.13.1.Final'

app starts fine (Linux rules!) and curl --verbose -L -X POST 'http://localhost:8080' -H "Content-Type:application/json" --data '{ "Records": [ { "messageId": "111", "receiptHandle": "MessageReceiptHandle", "body": "1111", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1523232000000", "SenderId": "123456789012", "ApproximateFirstReceiveTimestamp": "1523232000001" }, "messageAttributes": {}, "md5OfBody": "1111", "eventSource": "aws:sqs", "eventSourceARN": "arn:{partition}:sqs:{region}:1111:MyQueue", "awsRegion": "{region}" } ] }' leads sometimes to

<====18:47:37.695 INFO  (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.ht.HttpService] - Fast all ok: [[com.github.storytime.lambda.backup.http.UserItem@13e88342]]G [1m 41s]
> :qu18:48:38.321 ERROR (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.ht.HttpService] - fatal error: : javax.ws.rs.WebApplicationException: Unknown error, status code 504s]
        at org.jboss.resteasy.microprofile.client.DefaultResponseExceptionMapper.toThrowable(DefaultResponseExceptionMapper.java:21)
        at org.jboss.resteasy.microprofile.client.ExceptionMapping$HandlerException.mapException(ExceptionMapping.java:41)
        at org.jboss.resteasy.microprofile.client.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)

and sometimes to

fatal error: : javax.ws.rs.WebApplicationException: Unknown error, status code 502s]
        at org.jboss.resteasy.microprofile.client.DefaultResponseExceptionMapper.toThrowable(DefaultResponseExceptionMapper.java:21)
        at org.jboss.resteasy.microprofile.client.ExceptionMapping$HandlerException.mapException(ExceptionMapping.java:41)

(I adjusted logging to see the error) Fast response is ok, slow response is out of my reach. Once you resolve gateway issues or find reproducer that’s does not depend on external calls, please kindly let me know and I’ll have a look. Thank you

P.S. wild guess based on the stack trace you provided: Rest client receives response as you can see

00:32:31.111 INFO (Lambda Thread (DEVELOPMENT)) - [co.gi.st.la.ba.ht.HttpService] - Slow all ok: [[com.github.storytime.lambda.backup.http.UserItem@681c16ea]]

and exception goes through io.quarkus.resteasy.runtime.NotFoundExceptionMapper that is not in the class path when I removed the 2 deps above (quarkus-resteasy-jackson/mutiny).

Way I understand docs https://quarkus.io/guides/amazon-lambda-http you should use amazon-lambda-http rather than combination of amazon-lambda + resteasy. Could you give it a try, please?