Docker’s power and versatility have cemented its place in developers’ and administrators’ toolkits. Along with this widespread adoption comes the critical need for effective logging in Docker containers. However, once you scale beyond a single container on a single machine, effectively capturing and working with logs from Docker presents a challenge. The native docker logs command quickly becomes inadequate, and you’ll need a more scalable solution.
In this article, we’ll address this challenge by using the Docker Syslog logging driver to export and aggregate your container logs to Papertrail™ for continuous live monitoring. Let’s first dive into Docker logging basics to build a strong foundation for what’s to come.
Docker Logging Basics
Every Docker container generates logs as it runs, providing a chronological account of events and interactions that occur within the container. These logs are essential for debugging purposes and performance analysis. You can access these logs simply by using the docker logs command followed by the name or ID of the container.
As an example, we have a Docker container that runs a simple Nginx server. The Nginx configuration file (
default.conf) looks like this:
Dockerfile for our container looks like this:
When we run our container, we immediately see our Docker container logs:
However, if you start a Docker container in daemon mode (running in the background), you won’t automatically see the logs as you would when running a container interactively.
With the process detached, you will only see the container ID, but none of the container’s logs will be displayed.
The docker logs command will output all the logs for the container. If you add the
--follow (or -f) option, you can see all the previous logs and tail the logs, allowing you to monitor its ongoing activity.
Notice that the
docker logs command can take the container ID or the container name if you assign a name to the container.
Limit the amount of output to a specified number of lines
To control the amount of output, you can use the
--tail option, which allows you to limit the output to a specific number of lines. For example:
Filter the logs to fit within a specified timeframe
--until options are useful for filtering logs based on a specific timeframe. These arguments expect a timestamp formatted as
YYYY-MM-DDTHH:MMZ. For example, if we want to see all logs since a certain time, we would do the following:
Or perhaps we want to see any logs until a certain point in time:
Understanding these basics of Docker logging is the first step in effectively managing and monitoring your Docker logs.
Exploring the Docker Syslog Driver
Docker provides several logging drivers, each designed to send container logs to a different place. For example, the default
json-file driver writes logs formatted as JSON. The Docker Syslog logging driver sends logs to a Syslog daemon running on the container’s host or on a remote server.
Within the context of exporting Docker container logs, the Syslog logging driver proves particularly useful. The logs sent to the Syslog daemon can be further processed downstream and forwarded to other systems for storage and analysis. Therefore, the Syslog logging driver is an ideal solution when aggregating logs from multiple containers to a single location.
If you want to send your container’s logs to the local syslog, you simply need to set
A quick check of our local Syslog shows our container logs showing up there.
Now that we’re getting comfortable with the Syslog logging driver, let’s explore how to send our container logs to a remote Syslog daemon for centralized log management and live monitoring.
Papertrail is a cloud-based log management service used by organizations to aggregate logs across their entire tech stack—applications, services, devices, and more. With its live monitoring capabilities, fast search, and webhooks, Papertrail enables engineering teams to analyze and troubleshoot incidents with efficiency.
For distributed applications—such as containerized microservices running on different host machines—log aggregation to the Papertrail Syslog server is an excellent solution that is simple to implement.
Sign up for a free Papertrail account to get started with this solution.
After logging in, navigate to your Dashboard and then click Add Systems.
At the top of the system setup page, you’ll see a message that looks similar to this:
Your URL will be unique, but it is the one you will use as your remote Syslog server address. Copy it down.
Run Docker container with Syslog address specified
In our example Docker container, we simply need to add a
--log-opt option, setting our
syslog-address to use UDP and reference the URL and port that points to Papertail. We start up our container like this:
Notice that we provide an additional
--log-opt option, tagging our logs for this container with
Next, we navigate to the Events page at Papertrail. This is what we see:
Our logs are arriving, and you can see that they are tagged with my-nginx-container. Imagine if you are aggregating logs from multiple containers to Papertrail. By tagging each one (for example, with
payment-processing-service), then you’ll have all your logs in one place but can still take advantage of filtering (by tag) to narrow down your view to a specific container.
One of the best features of Papertrail is its ability to live tail your incoming logs. With this feature, we have continuous monitoring of our Docker logs. As activity happens in your Docker containers, you see the log events propagate to Papertrail almost immediately.
In a sense, the live tailing in Papertrail is similar to the output you would get by using
docker logs --follow. However, consider how cumbersome it would be to use
docker logs --follow for every one of your Docker containers, across all the host machines on which they run. With Papertrail, you get the best of both worlds: log aggregation to a centralized location and live tailing.
Couple this with the alerts in Papertrail—which can send you notifications through Slack, PagerDuty, or other means whenever incoming logs meet certain search criteria—and efficient search and analysis capabilities, and you have a robust solution for managing your Docker logs that is quick to implement and easy to use.
As software organizations increasingly pursue the building of distributed microservice-based applications, you’re likely depending on Docker more and more for your containerized microservices. With this comes a need to be adept at working with Docker logs.
Following the log for a single Docker container is trivial. However, distributed applications often comprise dozens (or even hundreds) of containers. For this, you need aggregation to a centralized log management platform like Papertrail. Fortunately, with Docker’s Syslog logging driver, shipping your container logs to Papertrail is simple and straightforward. From there, you can take advantage of live tailing, blazing-fast search, and alerts. You’ll be all set for continuous monitoring of your Docker container logs, ready to respond to and handle any incidents that come your way.