How to configure the Gunicorn application server for graceful restarts, and what is the difference between regular, hot, and phased restarts?
Application restarts are necessary when things go wrong or if we need to push a new application version. A regular restart isn’t usually anything more than stopping and starting the server again. To keep clients connected or even keep serving requests, we need a better strategy.
Graceful restarts
Application servers like Gunicorn can be usually stopped and restarted in various ways:
- regular restarts: connections are lost, and new ones are established after booting the new application process
- hot restarts: connections are not lost but wait for the new application server workers to boot
- phased restarts: current connections finish with old workers and new workers handle new connections
Hot and phased restarts is what we call graceful restarts, because they try to gracefully finish the already started requests first.
Gunicorn supports graceful restarts with the HUP signal, which reloads
the configuration, starts the new worker processes, and gracefully shut-
downs older workers. If the application is not preloaded, Gunicorn will
also load the new version of it.
To send a signal, we use the famous kill program:
$ kill -SIGHUP 25197
If we need to replace the Gunicorn executable, we can issue the
SIGUSR2 signal to the current master process after we replaced its
executable with the new one. A new master process with new workers
and a new PID will be started (.2 will be appended to the PID file).
You can try it out with:
$ kill -SIGUSR2 25197
Usually, we achieve zero downtime configuration change by leveraging systemd socket activation, swapping containers, or load balancing processes and servers. But as you can see, we can depend on SIGUSR2 as well.
To shut down the old master process, send the WINCH signal:
$ gunicorn -c config/gunicorn.conf app:app
$ #... some changes to Gunicorn like an update
$ sudo kill -SIGUSR2 $PID
$ sudo kill -WINCH $PID
Finally, Gunicorn configuration has a directive to set the maximum timeout for graceful shutdowns (in seconds):
$ config/gunicorn.conf
...
graceful_timeout = 30
