After a log sender permanently stops generating logs (like because a system was deprovisioned), Papertrail can automatically remove it for you.

Until today, Papertrail automatically decided whether or not to remove senders based on log destination and sender settings. We’ve made the behavior more configurable and more explicit. We hope this eliminates what might have been a manual (or simply overlooked) process.

How it worked

Papertrail relied on a basic heuristic to infer whether long-idle senders (which no longer represent any active logs) could be removed. The heuristic would only delete senders which had not been manually modified, and as a result if the sender began logging again, the new automatically-created sender would match the deleted one.

This isn’t enough information to reach the right decision, though, because 2 environments with otherwise-identical settings might prefer different behavior. Modifications aren’t a reliable indicator of preference.

Also, the heuristic was generally too conservative, so inactive senders would linger until they were manually removed. If you’ve ever thought “Hey, my Papertrail Dashboard seems to be accumulating old senders. I should remove them,” this was why.

Finally, as a byproduct of not being explicitly configurable, its behavior wasn’t obvious enough.

What changed?

We removed the magic and replaced it with a simple yes-or-no choice.

Product design means balancing automatic behavior against explicit choice. In this case, savvy people have different preferences, the heuristic can’t fully consider them, and an explicit decision isn’t hard to make. This problem is better solved by a “dumb” yet explicit implementation.

There’s now a simple “Yes, expire” or “No, don’t” option for each log destination. When enabled, idle senders are removed 2 days after their most recent log message is no longer searchable, but never less than 1 week after they’ve quit sending.

Transition

We needed to gracefully transition existing senders, so that no temporarily-idle systems are removed yet no one needs to manually remove hundreds of senders.

To that end, as part of this transition, senders with no searchable logs which haven’t been active for 1 month will be removed. 1 month was chosen as a very conservative threshold of “inactive.” Remember that no logs are being removed; the sender is simply a container for a tiny bit of metadata, and adding it again is simple.

Senders created after today will honor the policy set for its log destination. By default, it is enabled, and for most environments, auto-deletion is appropriate. To prevent idle senders from being deleted, change the log destination setting. Existing senders can also be changed via API.

If we can remove a large number of existing senders for you, please ask.

Live now

We’ve tested this with a few customers and it’s made their Papertrail Dashboard a more accurate view of their environment with no effort. In this case, a simple decision does means less work.

As usual, please send any comments or suggestions. Enjoy!