question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

After the first async request, Tomcat will shutdown immediately when an async request is in-flight

See original GitHub issue

What happened: Graceful shutdown does not work with keepalive for embedded Tomcat server. Requests are not successful. Error 500 is returned.

What you expected to happen: Graceful shutdown works, just like for embedded Jetty, embedded Undertow.

How to reproduce it: Spring boot with the following endpoints:

  • /url1 with a immediate result.
  • /url2 that returns DeferredResult<ResponseEntity<?>> immediately. Some result is set after 20 seconds by another thread.

application.properties content:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=60s

I attach the appropriate code. spring-boot-gracefulshutdown-issue23437-2020-09-21-220000.zip

[n] means “terminal number n”

[1] $ git clone --depth 1 https://github.com/180254/spring-boot-gracefulshutdown-issue-23437.git [1] $ cd spring-boot-gracefulshutdown-issue-23437/demo-tomcat/ [1] $ mvn package -DskipTests [1] $ java -jar target/demo-tomcat-0.0.1-SNAPSHOT.jar [2] $ curl -v --keepalive --keepalive-time 60 “http://localhost:8080/url1” “http://localhost:8080/url2” [3] $ kill $(ps aux | grep [d]emo-tomcat-0.0.1-SNAPSHOT.jar | awk ‘{print $2}’)

The request is not successful. Error 500 is returned.

Logs: app

2020-09-21 21:35:04.839  INFO 208159 --- [extShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete
2020-09-21 21:35:04.847  INFO 208159 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete
2020-09-21 21:35:04.896  INFO 208159 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

curl

* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /url2 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 455
< Date: Mon, 21 Sep 2020 19:35:04 GMT
< Connection: close
< 
* Closing connection 0
<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1></body></html>

Anything else we need to know?:

If you replace an embedded Tomcat server with an embedded Jetty or embedded undertow, the problem does not exist. I attach examples in the above mentioned repository.

Logs when used Jetty: app

2020-09-21 21:40:18.109  INFO 209508 --- [extShutdownHook] o.s.b.web.embedded.jetty.JettyWebServer  : Commencing graceful shutdown. Waiting for active requests to complete
2020-09-21 21:40:35.531  INFO 209508 --- [ jetty-shutdown] o.s.b.web.embedded.jetty.JettyWebServer  : Graceful shutdown complete
2020-09-21 21:40:35.535  INFO 209508 --- [extShutdownHook] o.e.jetty.server.AbstractConnector       : Stopped ServerConnector@7b420819{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2020-09-21 21:40:35.536  INFO 209508 --- [extShutdownHook] org.eclipse.jetty.server.session         : node0 Stopped scavenging
2020-09-21 21:40:35.538  INFO 209508 --- [extShutdownHook] o.e.j.s.h.ContextHandler.application     : Destroying Spring FrameworkServlet 'dispatcherServlet'
2020-09-21 21:40:35.539  INFO 209508 --- [extShutdownHook] o.e.jetty.server.handler.ContextHandler  : Stopped o.s.b.w.e.j.JettyEmbeddedWebAppContext@65045a87{application,/,[file:///tmp/jetty-docbase.4659637512004705994.8080/],UNAVAILABLE}
2020-09-21 21:40:35.541  INFO 209508 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

curl

* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /url2 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 21 Sep 2020 19:40:15 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 7
< Connection: close
< 
* Closing connection 0
OK/url2

Environment: Spring Boot 2.3.4.RELEASE Apache Tomcat/9.0.38 Apache Maven 3.6.3 Java version: 11.0.8, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/adoptopenjdk-11-hotspot-amd64 Ubuntu 20.04.1 LTS (Focal Fossa), 5.4.0-47-generic, amd64

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
markt-asfcommented, Sep 22, 2020

Fixed for 10.0.x, 9.0.x and 8.5.x and will be in the next release of each (early next month - Oct '20).

1reaction
markt-asfcommented, Sep 22, 2020

Agreed. This is a Tomcat bug, Working on it…

Read more comments on GitHub >

github_iconTop Results From Across the Web

Tomcat7 is not waiting for async requests to complete their ...
I am heavily using asynchronous API in my application, which is running on Tomcat 7. One of the requirements that I have --...
Read more >
63003 – Tomcat is closing async connection prematurely ...
I am heavily using asynchronous API in my application, which is running on ... then immediately I request a shutdown of a tomcat...
Read more >
Allow the embedded web server to be shut down gracefully
When an in-flight request completes successfully after graceful shutdown has begun, both respond with a Connection: keep-alive header. At this ...
Read more >
Changelog - Apache Tomcat 8 (8.5.66)
Correct some double counting in the code that tracks the number of in-flight asynchronous requests. The tracking enables Tomcat to shutdown gracefully when ......
Read more >
17.12 Asynchronous Processing - Java Platform, Enterprise ...
The following code shows a basic servlet that does not use asynchronous ... The service method of AsyncServlet returns immediately, and the request...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found