In this post i will describe how i use Gunicorn inside Docker. We will specifically see how to configure Gunicorn and how to configure the logger to work nicely with Docker.
We will use the following Python WSGI app in our examples as myapp.py:
import logging
logger = logging.getLogger(__name__)
def app(environ, start_response):
path = environ.get('PATH_INFO', '')
if path == '/exception':
raise Exception('My exception!')
data = "Request on %s \n" % path
logger.info(data, extra={'tags': ['role:web', 'env:prod']})
start_response("200 OK", [
("Content-Type", "text/plain"),
("Content-Length", str(len(data)))
])
return iter([data])
The complete source code is available here.
Configuring Gunicorn
The best way to configure a Docker Container is using environment variables, Gunicorn does not natively support this. Gunicorn supports a configuration file that can contain Python code, we will use this feature to be able to pass environment variables to configure it.
We create a file named gunicorn.conf with the following content:
import os
for k,v in os.environ.items():
if k.startswith("GUNICORN_"):
key = k.split('_', 1)[1].lower()
locals()[key] = v
This code will iterate over all environment variables and find those starting by GUNICORN_ and set a local variable with the remaining part, lowercased: GUNICORN_MY_PARAMETER=42 will create a variable named my_parameter with ‘42’ as the value.
We are now able to pass any parameter from this list by prefixing them with GUNICORN_ and uppercasing the parameter name.
Examples:
- The number of Gunicorn workers: GUNICORN_WORKERS=5
- The backlog: GUNICORN_BACKLOG=4096
We can test it by starting Gunicorn like this:
$ export GUNICORN_WORKERS=2
$ export GUNICORN_BACKLOG=4096
$ export GUNICORN_BIND=0.0.0.0:8080
$ gunicorn --config gunicorn.conf myapp:app
[2015-10-29 11:39:34 +0800] [27229] [INFO] Starting gunicorn 19.3.0
[2015-10-29 11:39:34 +0800] [27229] [INFO] Listening at: http://0.0.0.0:8080 (27229)
[2015-10-29 11:39:34 +0800] [27229] [INFO] Using worker: sync
[2015-10-29 11:39:34 +0800] [27232] [INFO] Booting worker with pid: 27232
[2015-10-29 11:39:34 +0800] [27233] [INFO] Booting worker with pid: 27233
Setting up logging
A common way to setup logging in a Docker base architecture is to output the logs on the standard output and configure the Docker Daemon to forward this to Syslog.
As Python logs can be multilines, this is especially true for exceptions, it is recommended to use a JSON formatter.
We will use the json-logging-py (GitHub) library as our logging formatter.
Gunicorn can use a configuration file using the Python logging fileConfig format.
The following logging.conf file configures the root logger and Gunicorn error logger to output the logs on the standard output using a JSON formatter:
[loggers]
keys=root, gunicorn.error
[handlers]
keys=console
[formatters]
keys=json
[logger_root]
level=INFO
handlers=console
[logger_gunicorn.error]
level=ERROR
handlers=console
propagate=0
qualname=gunicorn.error
[handler_console]
class=StreamHandler
formatter=json
args=(sys.stdout, )
[formatter_json]
class=jsonlogging.JSONFormatter
We can test it by starting Gunicorn like this:
$ gunicorn --log-config logging.conf myapp:app
Note: I will ‘pretty print’ the JSON output for readability but the real output is on a single line.
If we do a request on http://127.0.0.1:8000/path we have the following output:
{
"host": "silicon.local",
"level": "INFO",
"logger": "myapp",
"message": "Request on /path \n",
"path": "/Users/sebest/work/blog-demo/gunicorn-with-docker/myapp.py",
"tags": [
"role:web",
"env:prod"
],
"timestamp": "2015-10-29T03:41:41.589027Z"
}
Now let’s try to request http://127.0.0.1:8000/exception to generate an Exception:
{
"exc_info": "Traceback (most recent call last):\n File \"/Users/sebest/.virtualenvs/pxlapi/lib/python2.7/site-packages/gunicorn/workers/sync.py\", line 130, in handle\n self.handle_request(listener, req, client, addr)\n File \"/Users/sebest/.virtualenvs/pxlapi/lib/python2.7/site-packages/gunicorn/workers/sync.py\", line 171, in handle_request\n respiter = self.wsgi(environ, resp.start_response)\n File \"/Users/sebest/work/blog-demo/gunicorn-with-docker/myapp.py\", line 10, in app\n raise Exception('My exception!')\nException: My exception!\n",
"filename": "glogging.py",
"host": "silicon.local",
"level": "ERROR",
"lineno": 228,
"logger": "gunicorn.error",
"message": "Error handling request",
"path": "/Users/sebest/.virtualenvs/pxlapi/lib/python2.7/site-packages/gunicorn/glogging.py",
"tags": [],
"timestamp": "2015-10-29T03:46:02.349887Z"
}
We can see the Python traceback in the exec_info key of the JSON output:
Traceback (most recent call last):
File "/Users/sebest/.virtualenvs/pxlapi/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
self.handle_request(listener, req, client, addr)
File "/Users/sebest/.virtualenvs/pxlapi/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/Users/sebest/work/blog-demo/gunicorn-with-docker/myapp.py", line 10, in app
raise Exception('My exception!')
Exception: My exception!
Wrapping all of this together in a Docker image
We use the following Dockerfile:
FROM python:2.7-slim
RUN pip install gunicorn json-logging-py
COPY logging.conf /logging.conf
COPY gunicorn.conf /gunicorn.conf
COPY myapp.py /
EXPOSE 8000
ENTRYPOINT ["/usr/local/bin/gunicorn", "--config", "/gunicorn.conf", "--log-config", "/logging.conf", "-b", ":8000", "myapp:app"]
Let’s build it
$ docker build -t sebestblog/gunicorn-with-docker .
We can now run the Docker image with some Gunicorn parameters
$ docker run -e GUNICORN_WORKERS=4 -e GUNICORN_ACCESSLOG=- -p 8000:8000 sebestblog/gunicorn-with-docker
The complete source code is available here.