Routers play a key role in any network. If you’re like most network engineers, you usually spend most of the time working with routers (as opposed to other network devices). In smaller networks, routers may even be the only devices managing traffic. Router logs can tell you a lot about your network. In this post, we’ll walk through how to analyze router logs efficiently.
Routers: What Are They Anyway?
Put simply, a router is a network device that manages network traffic. It can forward packets from one computer to another and from one network to another.
While it sounds relatively simple, routers do much of the work in a network and sometimes can even do the job of other devices (for example, switch or firewall). So, if you have any issues with your router, you’ll probably completely lose connectivity between the devices in your network and to the internet. It’s crucial to identify and fix router-related issues as soon as possible.
Even if the issue is related to another device because routers play such a central role in the network, analyzing router logs can help you spot the problem. And because any connection will eventually have to pass the router, managing router logs also helps with forensic security analysis. But with all this traffic, the volume of events can be overwhelming and hard to manage manually. Spotting anomalies in the volumes of the ordinary traffic is difficult.
What Router Logs Can Tell You
Whenever you’re facing network issues, you probably can find the cause in router logs. In most cases, you’ll find a simple configuration error causing the problem. Let’s cover a few examples:
- Allow/deny lists: Allow/deny lists are popular, firewall-like mechanisms built in on most routers that allow you to simply allow or deny traffic based on the source or destination of the packet. For example, maybe you forgot to add a new IP address to the list. Unfortunately, spotting this on an affected computer can be difficult (because there can be many reasons for a connection not working). Looking at the router logs, however, can help you see if a particular computer was blocked because its IP address isn’t listed in the router’s allow list.
- Protocol-based filtering/rules: Many routers also filter traffic or apply special rules to it based on the traffic’s protocol. For example, a common security practice is to block ICMP traffic. But it’s also a common practice to use a ping tool to check the connections. If a router blocks ICMP traffic, a ping command will fail, which may mislead you in to thinking the system you’re trying to test isn’t available.
- ARP protection: The address resolution protocol (ARP), which is responsible for mapping hardware addresses to IP addresses, is crucial for any network to work properly. Unfortunately, because ARP allows a spontaneous reply, ARP spoofing attacks exist. Some routers implement various mechanisms to prevent this form of attack. Sometimes, those mechanisms also block legitimate requests. Again, your best bet is to analyze logs to see if there’s a message indicating the ARP protection mechanism has been triggered
- Connection rejected: We covered a few examples above, but there are many other reasons connections can be rejected. Some routers have more default security mechanisms built in than others. Whenever your connection isn’t correctly routed, you should see the reason in the router logs.
Logs of the traffic flowing through the router aren’t the only logs you should be monitoring. You should also look at the access activity of the router itself, especially if you suspect a security problem. Antivirus software and firewalls can protect you from malware and phishing, but if someone gains access to your router, then the damage could be company- or network-wide.
For starters, you should be monitoring successful and failed login attempts. Why should you monitor successful attempts as well? Because if someone logs in at 3 a.m. when you’re sure you and your team are offline, it might mean an unauthorized user got your credentials. And you should monitor failed attempts for obvious reasons. One or two failed attempts can mean someone simply forgot their password or made a typo. But if you are seeing more failed attempts over a longer period of time, then it could indicate someone is trying to guess (or is performing a brute force attack to bypass) the credentials.
You should also monitor access attempts related to other services. The more feature-rich the router is, the more extra services it exposes. Some routers may allow SSH, Telnet, or VPN access. If you don’t use these extra services, it can be easy to forget about them. Analyzing all possible access logs gives you a clear understanding of whether your router is safe or has been compromised.
Frustration-free log management. (It’s a thing.)Aggregate, organize, and manage your logs with Papertrail
How to Analyze Router Logs Efficiently
We covered the importance of analyzing router logs. Now let’s discuss how to do it efficiently.
Even in small environments, routers can produce a huge number of event messages. About 80% of the event messages, however, are created with ordinary traffic. For example, making a new connection from one machine to another can produce more logs in a single second than you can follow. So, collecting and centralizing log messages is key.
In small networks, there may be only one router. But in most cases, for high availability and scalability reasons, there are multiple routers, which means there will be multiple sets of logs to find and analyze. To make it even more challenging, some routers require special software or are only accessible from a specific network segment. All this complexity can complicate troubleshooting. The best way to get around this is to aggregate your logs, so you can easily analyze and search logs when you need them. One possibility is to use syslog for sending all the logs from routers and other devices to a centralized log management tool. This solves the challenges of locating scattered across your environment. A cloud-based log aggregation tool is also highly scalable and can adapt to the sudden spikes in log volumes that frequently arise when there’s an issue. This approach also opens new possibilities, which we’ll cover next.
Searching Through Centralized Logs
Aggregating logs from your routers into one place allows you to have a much better understanding of what’s happening across your network. Centralization on its own, however, is only part of the solution. You still need to know what to search for. Here are a few tips for making the process more efficient:
- Key events: Imagine if you forgot to add an IP address to a router’s allow list. From a router perspective, it will automatically block traffic from this IP. You can’t just search for errors. You need to focus on key events and search for these specific events. For example, if there’s a problem with ICMP traffic, you’ll need to first filter logs for ICMP traffic and then pick a time span matching when the issue started. Or if there’s an issue on one specific host, you can filter the logs by the host IP or MAC address.
- Repetitive searches: If you notice you’re executing the same search queries multiple times, you can further streamline your troubleshooting by saving the searches. This will let you create a list of your most common searches, name them accordingly, and use these custom searches to speed up the whole debugging process.
- Proactive alerts: To build on the idea of creating a saved searches list, if you frequently search for the same types of events, you might want to consider turning the search into an alert. This saves you the hassle of executing the same search each time and help you catch issues quickly. Too many failed login attempts? Alert! Too many ARP updates? Alert!
Pick the Right Log Management Tool
Centralizing logging is a great way to analyze router logs and cut down on the time you spend troubleshooting. You can also speed up troubleshooting by saving searches and setting up proactive alerts. These capabilities, however, vary widely across log management tools. This makes it important, to also pick a good log management tool. How can you tell which tool is best?
First and foremost, a good log management tool should allow you to aggregate all of your logs—not just network logs, but also logs from applications, services, and platforms. SolarWinds® Papertrail™, for example, aggregates logs from most sources.
Log aggregation saves you time by eliminating the need to hunt for logs in different systems and lets you search logs centrally to see events in context across devices, applications, and services. A good log tool, like Papertrail, will also offer a live tail function to let you view event messages as they’re written to the log. This capability will let you do real-time troubleshooting and help you pinpoint issues more quickly.
Interactive visualization of event volume can be useful when trying to identify when and where a problem occurred. Using the types of charts, you can quickly see when the event volume started increasing and spot trends. It’s great for quickly focusing your troubleshooting efforts and understanding the scope of the systems impacted.
The importance of analyzing router logs can’t be neglected, and neither can be the importance of a good tool for that purpose. Hopefully, you now know how to deal with router logs and how to pick the best tool for the job.
This post was written by Dawid Ziolkowski. David has 10 years of experience as a Network/System Engineer at the beginning, DevOps in between, Cloud Native Engineer recently. He’s worked for an IT outsourcing company, a research institute, telco, a hosting company, and a consultancy company, so he’s gathered a lot of knowledge from different perspectives. Nowadays he’s helping companies move to cloud and/or redesign their infrastructure for a more Cloud Native approach.