Help Contact @papertrailapp Legal Documents California Privacy Rights Software Services Agreement Privacy Notice GDPR Resource Center Email Preference Center COVID-19 Resource Center
© 2020 SolarWinds Worldwide, LLC. All rights reserved.
At Papertrail we are software engineers who are passionate about programming, debugging, logging and pretty much everything about building and running applications. We enjoy keeping our coding skills sharp and playing with new technologies. Below are some of things we have picked up along the way.
Fully Functional for 14 Days
Docker changed the way developers build software. It solved many issues, but bugs can still occur. When this happens, the first step in the debugging process is usually to read logs. However, when using Docker, this isn’t as straightforward as you may think. You can simply execute docker logs [container_id], but it’s not always possible to use this command, and it’s not an ideal solution for bigger applications. In this post, you’ll learn the pros and cons of the different logging options and what to consider when choosing a logging strategy in Docker.
Docker provides a few different ways to approach logging. It has some so-called “logging drivers” built in, which allow you to customize logging for different purposes. At the same time, however, you need to choose which logging driver will be the best for you.
If you choose the wrong one, it can impact your application performance, or you may periodically lose some logs. This is why it’s important to understand the purpose of all the methods and know which one suits you best. Additionally, when using Docker, we usually work with more than one container. And the more containers you need to manage, the more important it becomes centralize log management.
When it comes to optimizing logging for debugging, two configuration options play the biggest role: logging driver and delivery method. You can configure both. They are, in theory, independent from each other, but you should think about both when changing one or the other. We’ll explain why later, but first, let’s walk through what these two parameters do:
It’s best to think about both of these together because some logging drivers work better with one delivery method, and others work better with another. We’ll explain this in more detail in the next section.
No matter which logging driver you end up with, you can configure it in blocking or non-blocking delivery mode. The delivery method defines what’s more important: delivering logs or running the application.
Docker defaults to the blocking delivery mode. Blocking delivery guarantees all log messages are saved to the destination because every time a container needs to write a log entry, Docker blocks an application execution until the log delivery is completed. In most cases, it’s not as bad as it sounds—you likely won’t notice any interruptions. But in some high-usage cases, the interruption may be noticeable.
As you may have guessed, non-blocking delivery works in the opposite way. This delivery method won’t block the application from executing to deliver logs. Instead of waiting until logs are saved to their destination, the container will write logs into a buffer in the container’s memory. From there, logs will be delivered to wherever they need to be saved only when a logging driver is ready to process them. Though the non-blocking delivery method sounds like a better approach, it also creates a risk of some log entries being lost. The memory buffer the logs are stored in has a limited capacity, so it can fill up. Also, if a container crashes, logs might simply get lost before being released from the buffer.
Choosing between these two delivery methods depends on what’s more important to you: performance or reliability.
For most use cases, the blocking mode is recommended, unless your logging drivers send logs over the network to a remote server. A network can be slow or even periodically unavailable. This can cause issues with the application, as it would be blocked until logs are saved.
On the other hand, logging drivers designed to save logs to local files are usually fast. Therefore, in this use case, it’s safe to use the blocking mode.
If your application produces many logs or is sensitive to disruptions (or if for any other reason performance is the top priority), then you should use non-blocking mode. You just need to make sure the in-memory buffer size is adequate for your needs to avoid filling up the buffer.
You can set up the desired delivery method per container by passing the mode option to the –log-opt parameter when running a container. Here’s an example of this:
docker run --log-opt mode=non-blocking busyboxYou can also change the global Docker delivery method by adding the following to the daemon.json file:
{
"log-opts": {
"mode": "non-blocking"
}
}By default, Docker uses the json-file driver, which simply writes logs in JSON format to a local file. It’s fast and efficient, and it works well in most cases. In small environments, it’s best to either keep the default json-file driver or use the syslog or journald driver. Using the docker logs command is only possible with local, json-file, and journald drivers (unless you’re using Docker Enterprise, in which case the docker logs command can be used for any logging driver).
On a bigger scale, however, it’s more efficient to centralize logs. Some logging drivers, such as fluentd, awslogs, gcplogs, and gelf, can do this for you by gathering logs from all containers and shipping them to a common destination. The more containers you have, the more difficult it becomes to find the root cause of the problem. By centralizing logs, this becomes much easier.
You can set the logging driver by passing the –log-driver parameter to either the docker run command or to a daemon.json file, similar to what we saw with delivery method.
If you decide to centralize logs from all containers, it’s important to assign useful tags and labels to log messages. Otherwise, you’ll end up with a difficult-to-understand stream of messages. Tags and labels can help you easily find and filter the messages you’re looking for. It’s wise to tag logs with information like the environment and zone. Using well-defined tags can help you answer questions like “Are there any errors from my back-end services in a production environment?”
Another important thing to remember is to secure log shipping when using syslog or other remote destination drivers. Logs can contain sensitive data, so shipping them in plain text shouldn’t be an option. You should configure a logging driver to use a TLS/SSL connection to the logs’ destinations.
When using a log driver other than json-file or local, you have to deal with other components/services in the whole log delivery pipeline. It could either be syslog or Journald on the host machine or it could be a remote destination such as Fluentd, Logstash, or one of the other remote destinations Docker supports.
No matter which one you choose, it’s important to keep its availability, too. For example, if you choose Fluentd, you can install it on each host where Docker is running, on a separate dedicated machine, or only on one host per X machines/clusters. However, this simple decision can affect your application.
If you decide to install one Fluentd host for multiple servers, it can become overloaded. If you choose the blocking delivery method at the same time, it can cause issues for your application. The same applies if you use logging drivers designed to ship logs directly to a remote server, such as Amazon CloudWatch Logs or Cloud Logging from Google Cloud. To deliver these logs, Docker must establish a connection to different networks. Therefore, it’s crucial to either use non-blocking mode or to create an extra network link other services can’t saturate.
In other words, when you decide to use a logging driver, it’s important to not only change Docker’s configuration but to make sure the other side of the link is available.
In this post, we’ve run through five tips for selecting the optimal logging strategy for Docker container logs. Unless you’re running a small stack, centralized log management is the best option. It’s easier to find the root cause of any issue when you can see sorted and aggregated messages from all containers in one place. But centralizing logs is only half the battle. If you don’t tag your containers properly, you’ll end up with a centralized mess.
Tools like SolarWinds® Papertrail™ can simplify debugging with fast search and features like a log velocity graph to visualize error volumes. On top of this, you don’t have to worry about scalability, security, and reliability. And if you’re used to livestreaming logs (e.g., via tail -f or docker logs –follow), you can still do this with the tail feature in Papertrail. Creating alerts is also possible, so you get a debugging tool capable of removing all the complexity involved with Docker logging.
This post was written by Dawid Ziolkowski. Dawid has 10 years of experience as a network/system engineer, has experience with DevOps, and has recently worked as a cloud-native engineer. He’s worked for an IT outsourcing company, a research institute, telco, a hosting company, and a consultancy company, so he’s gathered a lot of knowledge from different perspectives. Nowadays, he’s helping companies move to cloud and/or redesign their infrastructure for a more cloud-native approach.
Today’s applications and services support the core business activities organizations rely on. Service interruptions and downtime time are no longer just inconvenient, they’re directly tied to both lost revenue and customers. It’s more important than ever to actively monitor the health of your applications and services to detect issues before there’s an outage. Being able to quickly detect and resolve issues is key to your organization’s financial and competitive position, as well as your career success.
Monitoring and alerting can help with problem detection, but it’s not enough to understand what happened or to pinpoint the cause. Let’s say a user accesses one of your applications. After performing a couple of actions, an error pops up. Through your monitoring tool, you know something went wrong, but debugging the actual problem is much harder. You don’t have any data that can help you diagnose it.
Logging is the answer, but getting the most out of your logs depends on how you implement logging. Logging as a service (LaaS), especially in today’s hybrid and distributed environments, can transform your log management program.
Below are the five reasons my team and I implemented logging as a service for our applications.
Due to the distributed nature of a microservices architecture, each service generates its own logs, stored in different locations. This means you have to search multiple systems and locations for relevant logs when trying to debug a problem. This can consume precious time when you’re under pressure to get an application back online.
Imagine a web shop where the shopping cart, user information handling, and payment handling all are separate services outputting logs. If something goes wrong during the payment handling step, you’d need to look for logs in each of the distinct services the user has accessed and somehow correlate them.
Using logging as a service, logs from multiple services are aggregated and searchable from a centralized location. This means you no longer spend time looking for logs or manually correlating them. You can quickly retrieve a full list of logs across all services.
Core benefit: a centralized interface for searching logs from multiple different log streams.
LaaS simplifies log management. Who wants to spend time creating an elaborate logging infrastructure for aggregation and analysis? This type of project requires a large initial investment in time and resources and long-term support and maintenance. It distracts your team from delivering business-impacting features and can quickly increase development costs.
Some DevOps teams use an open-source stack such as the ELK stack (ElasticSearch, Logstash, and Kibana) for their logging infrastructure. Open-source stacks are free to use, but they still require resources from your team to learn, install, and maintain.
So why would you invest time and resources building your own logging infrastructure when you can access world-class logging solutions for a fraction of the price? To be blunt, it makes more sense to outsource your logging infrastructure to a logging as a service solution, so your team can focus on creating business value.
Core benefit: you invest less time and resources than you would when building and maintaining your own logging infrastructure.
Many logging as a service solutions are dynamic and can handle a spike in log volume. This is particularly useful both when your application experiences an increase in traffic resulting in an increase in logs, and in a crash scenario when multiple services and applications are down. Frequently, home-grown logging infrastructures aren’t equipped to handle the sudden increase in data and will crash. This is the worst-case scenario. You’ll lose log data at the exact time you need it most to diagnose what’s going on.
A LaaS solution operates in the cloud and can easily scale when the amount of data increases, so you can be confident logs will be accessible when you need them.
Core benefit: logging as a service allows for easy scaling of your logging infrastructure in the cloud.
Mean time to repair (MTTR) is an important metric for many development teams. It indicates how quickly your team can resolve problems and get systems back online. Obviously, you want to reduce the total downtime, since this is linked to financial impact of an outage.
Several characteristics of logging as a service can reduce MTTR. First, logs are aggregated via your logging tool, which cuts down on the time and difficulty in locating the relevant log data. Second, you can access and search through logs via a single interface. Additionally, most LaaS also interleave or automatically interrelate events, so you can see all the log data surrounding an issue and quickly identify the root cause. The combination of the log aggregation, centralized search, and showing events in context significantly reduces the time it takes to understand and resolve issues.
Core benefit: mean time to repair can be reduced via a LaaS solution, reducing the financial impact of availability and performance issues.
Last, a LaaS solution can help your team become more proactive and identify issues before there’s a service impact. LaaS solutions are optimized to process large amounts of data and quickly process queries. They can also use this data to populate visualizations and generate alerts on changes in logging trends.
Understanding if an application or service is suddenly generating more events can easily be the key to fixing an issue before users are impacted. This data can also help you see exactly when and where the problem started, so you can get the root cause of issues faster. Using these insights has allowed my team to develop “early warning” alerts for potential service degradations and catch issues before they went critical.
Core benefit: quickly run queries over large amounts of data to find insights.
The answer is simple: no! You can have a logging as a service tool in place but still write vague or ambiguous logs. The value of your logging tool depends entirely on the way you handle logging. Keep in mind the following best practices if you want to get the most value out of your logging tool:
As a general rule of thumb, your logs should tell a meaningful story, so no matter which team ends up troubleshooting the issue they have the information they need to replicate what happened. Based on the story created in the logs, it should be easier to find and debug issues. .
A lot of applications, services, and devices generate logs: component logs, node logs, service logs, router logs, firewall logs, database logs, and many more. These logs reproduce the detailed path of all events leading up to a problem. Logs provide valuable pieces of information you can use to quickly debug a problem. However, scattered logs can be a real nuisance for developers when they need to find pertinent information. And they can sometimes prevent developers from finding the root cause of a problem.
Logging as a service aggregates logs to make them easily searchable from a single interface and makes log data more useful.
When logging has been implemented correctly, it adds immense value to your organization.
For example, SolarWinds® Papertrail™, a logging as a service solution, allows you to quickly search through multiple, combined log streams from a central interface and helps you reduce the mean time to repair. It also provides contextual search and log visualizations to help your team be more proactive in catching issues early to reduce the potential downtime.
This post was written by Michiel Mulders. Michiel is a passionate blockchain developer who loves writing technical content. Besides that, he loves learning about marketing, UX psychology, and entrepreneurship. When he’s not writing, he’s probably enjoying a Belgian beer!
Logs play a crucial role in any service as they provide tons of information about the wellbeing of your service. For example, logs can contribute important data to metrics, such as the incident rate, retry rate, latency rate, or even the number of issues a user experiences. Logs are also useful for monitoring the health of your service. For example, a high error rate indicates you need to improve the quality of your service to make it more reliable for users.
However, to get the most value from your logs, you need to set up and manage them correctly. Incorrect log management makes it harder to retrieve valuable information from your application logs. It can even provide incomplete or inaccurate information that complicates troubleshooting.
To avoid this, there are six key principals to keep mind when designing your log management program :
The log management process starts with knowing what to monitor. Avoid starting in a rush without having an actual plan. It’s important to first have a good understanding of what data you can monitor. When you know what data can be monitored, you can then decide what to monitor based on the importance of the log data.
Don’t try to monitor all data at once. Too much data is hard to manage, so try keeping things simple. To start, I recommend you implement monitoring for your critical services. This data can give you insights into the performance and reliability of those services. It also helps with identifying other dependent or interrelated services to include in your logging program.
It’s a fine balance, however. You want to avoid logging too little information. If you lack data, it can create blind spots, making troubleshooting a complicated task. If there’s data missing, you’ll have to guess what happened with your code instead of making an informed decision based on log data.
Key learning: Log management is about finding the logging sweet spot. Too much logging creates extra overhead, while too little logging creates blind spots.
Log aggregation is an important concept when dealing with multiple connected services. Especially for a microservices architecture, log aggregation is a must. Every service in your architecture produces logs. However, to debug a request that passed multiple services, it’s hard to find all relevant logs from different services.
Log aggregation helps you collect those logs and make them easily searchable. The ability to quickly search for relevant logs is a major benefit for your development and DevOps team. This will likely contribute to faster bug resolution and less time spent debugging.
Another trick is attaching a random ID to each request. The ID will be passed along the different services, and you can use it for logging important actions. This way, you can easily track the flow of a request by looking up all logs for a specific ID.
Key learning: Log aggregation helps make logs easily searchable, reducing troubleshooting time.
Building a log management system requires a lot of resources and time: it increases project costs and can pull resources away from more critical systems. As the old saying goes, don’t try to reinvent the wheel.
There are many powerful and affordable log management solutions already available. In the long run, the cost compared with building a system yourself will be drastically less.
Log management tools, like SolarWinds® Papertrail™, were designed by engineers just to do one thing really well—logging. They have spent years optimizing log ingestion, parsing, and storage and provide fast searching over mega volumes of log data.
Key learning: Avoid building your own log management system. Implement an off-the-shelf solution to keep costs down and ensure optimum performance.
Many organizations forget about the security of their logs. They treat logs as just another data source and often forget logs can contain tons of important or sensitive information.
Log security matters, and you need to manage who has access to your logs to prevent security issues. Role-based access control (RBAC) helps organizations create an audit trail of who has access to which logs, and which users can edit, delete, view, or share log data. This information can be useful to quickly identify how an attacker got access to data in the event of a data breach.
Key learning: Treat your logs as a valuable piece of information that deserves appropriate security measures such as role-based access control.
Integrations with teaming tools is a common innovation theme in many developer forums. Getting information quickly into the right hands increases responsiveness and allows you to shorten communication delays.
Before we had fancy tools, developers were notified about problems via email. This paradigm has changed completely. Nowadays, almost every service integrates with teaming tools like Slack, so developers can see live status updates about their services and log management. The major benefit of this approach is developers can access alerts from multiple different services all from one tool.
In other words, it allows developers to seamlessly integrate event monitoring into their everyday workflow. This means they don’t have to access other tools for receiving alerts or messages, and it allows the entire team to be aware of issues and troubleshoot them collaboratively. This leads to streamlined communication and faster resolution time. Furthermore, it also ensures your team doesn’t miss important alerts.
Key learning: Teaming tool integrations allow team members to access event logs and alerts in tools they’re already using and encourages a collaborative approach to troubleshooting issues.
Lastly, logs take up space. The growing need for more storage space can become costly, especially with on-premises servers. To avoid the complexity of managing a large number of logs, a cloud-based log management solution can scale up quickly, reduce management challenges, and help you contain costs.
Storing logs in a cloud-based log management solution is much cheaper, and eliminates the hassle of maintaining on-premises servers. For example, you need to have a backup generator in place to provide your servers with power in case of an outage, manage service updates, create offline backups, and other resource-intensive tasks. These little things add up and can exponentially increase the cost of hosting logs on your local servers.
Key learning: Save your organization time and effort by outsourcing log management to the cloud.
A few simple tricks can get your log management off to a good start. Don’t try to build a log management solution yourself or manage logs through on-premises servers. You can easily avoid those costs by choosing an affordable cloud-based log management solution. In the long run, you’ll save time and effort, which you can dedicate to more important tasks such as creating exceptional applications.
If you’re looking for a simple cloud-based log management solution, give Papertrail a try. Built by engineers for engineers, SolarWinds Papertrail provides an intuitive search syntax for combing through your logs from a central interface. If you want to see it for yourself, sign up for a trial or request a demo.
This post was written by Michiel Mulders. Michiel is a passionate blockchain developer who loves writing technical content. Besides that, he loves learning about marketing, UX psychology, and entrepreneurship. When he’s not writing, he’s probably enjoying a Belgian beer!
Every container produces logs with valuable information. A log is basically data written by the container to STDOUT or STDERR. However, if you run a container in detached mode, you can’t see the logs in your console because detached mode refers to running a container in the background of your terminal. Therefore, you won’t see any logging or other output from your Docker container.
Read moreLogging is vital for any modern software team, and it’s essential to get it right. A poorly-implemented logging strategy could cause you more headaches than it solves. In this article, we’ll discuss PHP logging and the main best practices you should be aware of and adopt.
Read moreLet’s imagine you’re looking for a flight ticket booking app for your Android smartphone. When you search for such an app in the Play Store, you’ll find multiple options. How do you decide which one to install? Easy! You try out a few of them. Find out which one is fastest, serves your purpose, and has a nice user interface.
Read moreLogs are often the foundation of metrics and observability infrastructure because they contain business-level statistics that help you and your team make decisions. Without them, it’s impossible to know how often users are hitting errors or how the latencies in your services are varying over time.
Read moreIf you’re responsible for keeping web servers running, you already know easy access to log messages is critical when troubleshooting issues. Apache provides comprehensive support for logging, and its highly customizable configuration allows you to tailor its logging to your exact needs. You can gain visibility into your web servers by logging everything from the initial request through to the URL mapping process and connection termination. And if this wasn’t enough, third-party modules provide additional logging capabilities such as support for application runtimes including PHP, Java, and CGI programs.
Read moreThe JSON format is ubiquitous and used everywhere from logging in web apps to message passing for microcontrollers. Thanks to its compact nature and it being easily readable by humans, JSON has become the de facto standard for sharing structured data.
Read moreModern apps and services generate a constant stream of log data capturing the inner workings of the code and how users interact with it externally. Buried inside those logs are golden nuggets of information critical for troubleshooting problems in development and in production.
Read moreHelp Contact @papertrailapp Legal Documents California Privacy Rights Software Services Agreement Privacy Notice GDPR Resource Center Email Preference Center COVID-19 Resource Center
© 2020 SolarWinds Worldwide, LLC. All rights reserved.