CentOS 7 and Docker Logging Drivers

CentOS 7 and Docker Logging Drivers

Background

I was testing log shipping configurations with FluentD as a container that reads Docker’s logs from the file system with their ‘default’ Logging Driver: json-file.

It didn’t appear that anything was logging. When I’d execute a tree /var/lib/docker/containers, no *.log file was being created. Why?

I have installed Docker all over the place, but didn’t realize some versions of the package make changes to many defaults. In this case, the default logging driver was journald which you can ‘tail’ with journalctl -xefu docker.

[root@centos-vm fluentd-docker]# docker info | grep 'Logging Driver'

Logging Driver: journald

Investigation

Here’s info about the default docker package:

[root@centos-vm fluentd-docker]# yum info docker

# redacted
Name        : docker
Arch        : x86_64
Epoch       : 2
Version     : 1.13.1
Release     : 75.git8633870.el7.centos

The first thing I checked was process arguments to see if --log-driver is passed explicitly:

[root@centos-vm fluentd-docker]# ps -ef | grep log-driver

root     12005     1  4 15:53 ?        00:00:00 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2

Yup, there it is: --log-driver=journald. The next question is How does it get there? The SystemD Service Unit should guide the way:

[root@centos-vm fluentd-docker]# cat /usr/lib/systemd/system/docker.service

# redacted
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
          --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
          --default-runtime=docker-runc \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --init-path=/usr/libexec/docker/docker-init-current \
          --seccomp-profile=/etc/docker/seccomp.json \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
	  $REGISTRIES
# redacted

There’s no mention of the argument --log-driver here. So it must be further downstream.

The Fix

One of the Environment Files that is parsed before starting Docker is /etc/sysconfig/docker.

The very first line in it is this:

OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'

You can either remove --log-driver=journald from that line, or change the value to json-file to explicitly set the default. Then restart Docker:

systemctl daemon-reload && systemctl restart docker

Quick Note

If you try to edit /etc/docker/daemon.json to include this:

{
  "log-driver": "json-file"
}

Then do:

systemctl daemon-reload && systemctl restart docker

…That’ll conflict with the --log-driver argument passed to Docker by the docker.service SystemD Service Unit File, and startup will fail.

References