Make Your Logs Work for You

The days of logging in to servers and manually viewing log files are over. SolarWinds® Papertrail™ aggregates logs from applications, devices, and platforms to a central location.

View Technology Info


Troubleshoot Fast and Enjoy It

SolarWinds® Papertrail™ provides cloud-based log management that seamlessly aggregates logs from applications, servers, network devices, services, platforms, and much more.

View Capabilities Info


Aggregate and Search Any Log

SolarWinds® Papertrail™ provides lightning-fast search, live tail, flexible system groups, team-wide access, and integration with popular communications platforms like PagerDuty and Slack to help you quickly track down customer problems, debug app requests, or troubleshoot slow database queries.

View Languages Info


TBD - APM Integration Title

TBD - APM Integration Description

TBD Link

APM Integration Feature List

TBD - Built for Collaboration Title

TBD - Built for Collaboration Description

TBD Link

Built for Collaboration Feature List

Blog > Aggregating Application Logs From EKS on Fargate

Aggregating Application Logs From EKS on Fargate

Posted by By Jillian Rowe on March 22, 2021

Today we’re going to talk about logging with Kubernetes on AWS using CloudWatch and SolarWinds® Papertrail. We’ll cover setting up Papertrail, installing and configuring the rKubeLog package, viewing the logs in the Papertrail event viewer, and cross-checking those logs with the ones we see with kubectl. From there, we’ll set up a few different alerts.

Papertrail supports most log types and is easy to configure with multiple services, applications, and cloud providers. It can aggregate from other logging services such as CloudWatch as well. It looks a lot like Log4j, so it should feel familiar to  software engineers. This is great for people who have multi-cloud setups who want to aggregate logs from different CloudWatch groups. It’s also very lightweight, and the solution we’re using today doesn’t require any sidecars, ELK stacks, or the like. Once it’s all configured, it will monitor logs from all your pods across all namespaces.

One simple kubectl apply, and we have logging!

Since all we require is kubectl, this is also a good choice for groups who prefer to do their dev work on minikube and deploy to AWS or another cloud provider, but want the same setup for dev and prod.

What’s Required?

You’ll need to sign up for an account with Papertrail. It’s free to get started, and no credit card is required. If you decide to pull logs from more applications or your event volume increases,  you can check out their plans. Everything in this tutorial is under the free plan.

Once you’ve created an account login, you’ll see an interface showing you how to set up some of the connections. For this tutorial, we won’t be using the standard logging setups; instead we’ll be using this Github repo. You’ll only need to get your Papertrail credentials, and from there you’ll be using the kubectl tool.

This tutorial assumes you already have an EKS or Fargate cluster setup on AWS, and we don’t go over that setup. If you’re just trying this out, the easiest method to get a cluster is to log in to the AWS Console, select EKS service, and create a new cluster from the console. My preferred method is to use Terraform to deploy a Rancher instance, and then use the Terraform Rancher Provider. If you’re looking to try out Papertrail, you can also use minikube.

Create a Logging Destination

To create a log destination in Papertrail and get the credentials you need to run rKubeLog, go to Settings -> Log Destinations.

Get the credentials you need to run rKubeLog to stream logs from EKS on Fargate

Keep everything as defaults, scroll down, and press Create. If all goes according to plan, you should see a green message at the top with your credentials.

If successful, a green message at the top of the screen will appear with your credentials.

Now, head on over to the console with your kubectl pointing at the cluster you’d like to use to log with Papertrail.

Install the Papertrail Deployment

To install rKubeLog, go to the GitHub Repo Releases page and grab the latest release. At the time of this writing, that’s 17.

Create Kubernetes Secrets

You can create your secrets in whatever way you prefer, including Terraform. For the sake of simplicity, I’m going to modify the kustomization.yaml supplied and then apply it all at once. Make sure to delete the secretGenerator key if you’re creating your secrets using a different method.

# ...
- name: logging-secret
  type: Opaque

Then apply it.

kubectl apply -k .

I’m only using this as a demo, and I’m deleting it as soon as this tutorial is finished. If you aren’t behind a firewall, you should encode your variables.


If you get an error, make sure you’ve removed the secretGenerator key from the kustomizations.yaml file, or else your secrets will get clobbered with demo variables.

Pay attention to your input types. The Papertrail default is TLS, but the default protocol in the kustomizations.yaml is TCP. If you don’t know which type you chose, go to the Papertrail Settings -> Log Destinations menu. You’ll see what is currently set, and if need be select the Settings option to the right to make changes.

To see which input type is selected go to the Papertrail Settings -> Log Destinations menu.

Check the Pods

kubectl get pods -n kube-system | grep rkube

Grab your pod name to check out its logs in the next steps.

Do Something Cool

Now that everything is set up, it’s time to generate events. You can dust off your “Hello World” code, install NGINX, or access an existing service. This cluster is a demo JupyterHub cluster, which is perfect for testing and seeing events.

kubectl logs -n kube-system rkubelog-5bd78cf697-d2wqv

You should see some events based on your input activity.

You should see events from EKS on Fargate  based on your activity in the kubectl logs.

Check Out Your Logs on Papertrail

Now that events are flowing, let’s head back to Papertrail and check out the logs in the event viewer. Once in Papertrail , go to Settings -> Log Destinations -> Your Log Destination Settings. You should see systems that correspond to your K8s pod names.

Validating log destination settings in Papertrail.

Now, go to Events to access the event viewer. You should see the event messages tagged with the Kubernetes pod names, in my case hub-78cd597d48-hxsin.

Event messages from EKS on Fargate via rKubeLog tagged with the Kubernetes pod names in the Papertrail event viewer.

Papertrail Systems

Papertrail automatically tags the events with the different system names, which correspond to the Kubernetes pod names. These are formatted differently (darker blue), and you can click on them to filter by them. For example, in my JupyterHub, you can see a proxy, login hub, and some user pods. The ability to filter and search event by pod name is a great time saver when troubleshooting.

The ability to search and filter by pod name is a great time saver when troubleshooting

If you want to double-check the Papertrail Systems, run:

kubectl get pods -n my-namespace

Of course, you won’t see any events unless the pod creates some output.

Optional: Install NGINX

If you don’t have anything installed on your AWS EKS cluster, you can always install NGINX.

helm repo add bitnami
helm update
helm install my-nginx bitnami/nginx

Once it’s installed, grab the Load Balancer address, put it into a browser, and press Refresh a couple of times to get some logs going.

kubectl get svc | grep nginx

Check out the pod logs.

kubectl get pods
# Get the pod name

Now, open the Papertrail event viewer. You should see event messages tagged with a system ID corresponding to your NGINX pod. If you don’t, go back to the secrets step and make sure you have the correct input type. If you’re still having trouble, you can reach out to the Papertrail support team directly through the Live Chat option under the Support menu.

Getting and Fine-Tuning Alerts

What good is logging if you don’t get a warning when something terrible happens or when users are doing something potentially deadly? Papertrail lets you setup alerts two ways. First, Papertrail is set up with a default alert, which will send you a daily email that aggregates messages from any logs with the word “error” in it. You can increase the frequency and add other types of alerts by opening the Alerts menu off the main navigation.

Setting up alerts a default alert.

Papertrail also lets you create alerts right in the event viewer, as you’re searching. When you open the event viewer, you’ll see a search bar at the bottom of the screen. To search the event data, type in your search criteria and save the search. The search is easy to use and works a bit like the Google search bar. To create an alert, save the search. In the save search box, you’ll have the option to create an alert

Papertrail lets you create alerts as you are searching.

Creating alerts from the search bar lets you create alerts while you’re troubleshooting, which is a great way to create an on-the-fly runbook and catch issues early.

Papertrail alerts integrates with many different notification and communication services. Not only do you get an email integration, but you also get Slack, other chat programs, webhooks, Zapier, and AWS SNS, to name a few.

You could potentially do some nifty things with this. Say you have a user who’s trying to do something that requires more memory than they have. You could use AWS SNS to push a topic to give you a different EC2 instance as the underlying node group, or you could ping the user on Slack and say, “Hey, stop doing that.” Bonus points if you make the message come from a user account and not the Slack bot.

Papertrail alerts integrate with many different notification and communication services.

Setting up alerts is a much broader topic than I have the room to go into in this post. But if you’d like to learn more, check out the official Papertrail Alert docs.

Clean Up

If you’d like to clean up your demo environment, you can delete your Papertrail-associated K8s resources. Navigate back to the rKubelog folder and run:

kubectl delete -k .


If you’re getting an error about the kubectl apply -k, you’re on an old version of kubectl. Install the newest version with:

curl -LO "$(curl -s"
mv kubectl /usr/local/bin/kubectl
chmod 777 /usr/local/bin/kubectl

When I see something odd happening, and I can’t figure out what it is, I usually jump straight to describing the pods.

kubectl get pods -n kube-system | grep rkube
# get the pod name
kubectl describe pod -n kube-system rkube-some-numbers

If the pod’s running, you can also get the logs. But this won’t work if the pod is in a failed state, not running, not scheduled, and so on.

kubectl logs -n kube-system rkube-some-numbers

Papertrail should be forwarding the logs directly from the pods. If you’re not getting logs, you can manually investigate them with the kubectl CLI. The best test is always NGINX because if NGINX doesn’t work, then you know the problem isn’t Papertrail.

helm repo add bitnami
helm update
helm install my-release bitnami/nginx

Wrap up

By now you should have rKubelog installed, configured, and forwarding your logs on over to the Papertrail. From here, you can start to think about what kinds of saved searches and dashboard views you need from your logs, and what triggers should be in place in case danger is on the way.

Jillian Rowe

Jillian is a DevOps engineer who specializes in elastic solutions for genomics and the life sciences. Her favorite programming languages these days are Python and JavaScript, and when she’s not writing in these languages, she’s deploying infrastructure to high-performance computing environments or AWS, or she’s hanging out at the beach with her family.