Docker Storage Caveats

Over the last year, we’ve been using Docker in production over at RADIOactive. Docker has been pretty great for us. From an operational perspective, it gives us the advantage of being able to deploy and monitor apps in a generic manner. From a developer’s perspective, application packaging is greatly simplified as well, as any dependencies can just be put into the container no matter what language or framework we are using, with any further configuration being added via environment variables.

It hasn’t been all smooth-sailing though, with most of the problems coming from Docker’s storage drivers. The choice of storage driver impacts how images are stored and how container filesystems work.

Device mapper with loopback storage

When Docker is first installed and run, by default it will select either the AUFS or device mapper driver based on available kernel support. Since AUFS requires additional kernel packages to be installed, it’s quite easy to end up using the device mapper driver. The problem is that by default device mapper runs in loopback mode, which is not recommended for production, as warned when running the docker info command:

Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

Even switching over to the direct-lvm mode of device mapper seems to bring it’s own share of problems, so we decided to skip devicemapper and move to AUFS instead.

AUFS

When we switched from device mapper to AUFS, we started experiencing performance issues on some of our servers. Our graphite server started running into sustained periods of 100% CPU use, and graph queries became extremely slow. It also took minutes to delete a stopped container on the system, even after restarting the docker daemon or rebooting.

Worse still, one of the production application servers started locking up entirely. Docker commands on that system would not complete at all.

OverlayFS

The third storage driver we tried was OverlayFS, which is only available on newer versions of the Linux kernel. In order to use overlay we decided to switch our base OS to Ubuntu 16.04 LTS (which should also support the upcoming overlay2 driver).

After the switch from AUFS to Overlay, Docker has been performing reasonably well, with none of the lockup issues from AUFS seen so far. However, we started hitting problems with inode exhaustion recently. Our deployments would fail because the filesystem was at 100% inode usage, meaning that no new files could be created even if there was still free space left. To mitigate the problem we added a line to our deployment script that would clear any unused docker images after starting up the containers.

The next version of Overlay specifically addresses this problem, and it will be coming out with Docker 1.12, so hopefully that gives us a trouble-free driver to use once and for all.

docker