Server logs are important and can never be left out of web development. There’s no such thing as a perfect website—even one owned by a big tech company is likely to have errors in production. Using web server logs, you can easily know where the problem is coming from and solve it on time. Logs are automatically created by the server and consist of files containing information such as errors, requests made to the server, and other information worth looking at.
Even though these logs can be created automatically by the server, you can also create custom log management functionalities. These can be done within the web app. Today, almost all programming languages have log management modules or at least a framework. With good configurations, you can utilize the language logging system and get the same log messages from the server.
Logs contain different types of files, such as images. In most cases, you’ll only want to see a certain type of information from the server. In this post, I’m going to give some of the best tips on how you can monitor and filter web server logs and get the most value out of them.
Why Do You Need Logging on Your Website?
Before we look at how to monitor your logs, let’s talk about why you should. Money comes from speed, and a website with good speed and without lag is what every user wants. You’ll need logs from the server to know if everything is working fine on your website. Sometimes, you don’t even notice your website is throwing errors until you have access to the log files. It’s not easy to know if the database is complaining about some bad queries you’ve written. You only know when you’re able to analyze the logs from the server. If you make use of web server logs, you’ll be able to serve your clients (users) better and generate more revenue.
How Can I Monitor My Web Server Logs?
Knowing how to monitor your web server logs is important. To monitor web server logs, you’ll need to use some tools to aid you, and these tools will depend on the type of server you’re using. Most servers today run on the Unix/Linux operating system (OS), and most common tools out there are Linux tools.
Before you can even start monitoring your website, you need to know what server your website is hosted on. There several servers out there, and the tools you’re going to use are mostly similar. Once you have an idea of what your web server is, you’ll be able to know when the logs are sitting on the server. There are four types of web servers: Apache, NGINX, LiteSpeed, and IIS.
Apache is the most popular web server. Most web servers use Apache because it supports almost all the operating systems we use today, including Mac OS X, Windows, Linux, Unix, and more. It’s easy to monitor web server logs on an apache server, but the first thing to know is where the logs are located. You can find logs on the Apache server by navigating to these directories on a Linux/Unix OS:
- /etc/httpd/log/access_log (on MacOS X)
Using the Tail Command to View Log Messages
To monitor error logs from the Apache server, you can use the Linux tail command to view all the errors as they occur in real time. Here’s an example of how you can log the errors on the console using the tail Linux command.
-$ sudo tail -f /var/log/apache2/error.log
When you run the above command, you’ll be able to view all the errors in the terminal console. You can now go ahead and use the error message to troubleshoot the problem you’re having. The -f flag in the command helps make sure all the log messages are echoed to the console. Seeing errors as they happen in real time can help you solve the problem before it affects the user experience, saving you money. If a revenue-generating website is down for hours or even just minutes, you’ll lose revenue, so being able to fix errors in real time is crucial.
Tail and egrep Commands
When you combine these two commands, the power is now in your hands. With these two commands, you can specify the types of files you don’t want to see displayed in the console. When these commands are combined to filter a log file, you get rid of the unwanted files, allowing you to experience greater speed as you’re processing the files you need. Here’s an example using the tail and egrep commands to filter unwanted files:
tail -f /var/log/apache2/error.log | egrep -v "(.gif|.jpg|.png|.swf|.ico)"
Using the grep Command to View Log Messages
Global regular expression print, or grep, is a Linux command you can use to read a file and echo the output on the console or write the data in another file. The grep command can be used to filter data in the file. If you want to display only a specific IP address, here is how you can use the grep command to filter that particular IP address (e.g., when searching for an IP address like 127.0.0.1):
$ grep "^127\.0\.0\.1" /var/log/apache2/access.log
In the same way, you can search for a specific keyword in any file via error.log. You just have to pass the keyword as the first option before the file you’re looking into and you’ll get the desired results (e.g., when you’re searching for an error message like “syntax error”):
$ grep "syntax error" /var/log/apache2/error.log
Logging on the NGINX Server
NGINX is also a commonly used server. You can access the log messages from the NGINX server by navigating to the following directories:
The first step is knowing where your log files are. This is important because it’ll bring you closer to actually solving the issue at hand. You can use the cat command to open the file and read its content, and the command is as simple as the one below:
$ cat ./var/log/nginx/error.log
You can also filter the error.log file to get specific information, and you can still use the grep command just like you can when accessing logs via the Apache server. Here’s an example of how you can filter a log file using a grep command:
$ grep "syntax error" /var/log/nginx/error.log
Any syntactical error message will be filtered for you. You can use this to solve syntax errors you have in your code. As you can see, grep is a powerful command. If you can, utilize its power and do even more advanced regular expressions to filter logs.
Writing all these commands isn’t easy, and neither is viewing data on the console. Reading data on the console is difficult because it’s usually not well aligned. It’s even more difficult if you’re reading a larger file. Even though these commands will do the job, you have to be patient enough to avoid making errors as you write them. If writing commands isn’t something you enjoy, there are logging tools designed to help you log your web apps. SolarWinds® Papertrail™, for example, is a cloud-based logging tool you can easily set up and start logging with. You’ll have a better feel and also be able to search syntax and filter logs in real time, an important bonus.
Logging is an important technology in website hosting, as it helps you troubleshoot server issues and optimize websites. You can also use server logs to know where most of the traffic is coming from, and you can look for things such as what files are accessed most by the website and its users. If you’re a web developer, you’ll need logs from the server. These can be used either to fix errors or to optimize the website.
Server logs can help you with almost anything regarding website issues and performance. But analyzing logs on the console won’t be easy, so you can use a cloud-based logging tool like Papertrail. With Papertrail, you can search logs and filter based on what you need (e.g., filter an IP address).