Make Your Logs Work for You

The days of logging in to servers and manually viewing log files are over. SolarWinds® Papertrail™ aggregates logs from applications, devices, and platforms to a central location.

View Technology Info

FEATURED TECHNOLOGY

Troubleshoot Fast and Enjoy It

SolarWinds® Papertrail™ provides cloud-based log management that seamlessly aggregates logs from applications, servers, network devices, services, platforms, and much more.

View Capabilities Info

FEATURED CAPABILITIES

Aggregate and Search Any Log

SolarWinds® Papertrail™ provides lightning-fast search, live tail, flexible system groups, team-wide access, and integration with popular communications platforms like PagerDuty and Slack to help you quickly track down customer problems, debug app requests, or troubleshoot slow database queries.

View Languages Info

FEATURED LANGUAGES

TBD - APM Integration Title

TBD - APM Integration Description

TBD Link

APM Integration Feature List

TBD - Built for Collaboration Title

TBD - Built for Collaboration Description

TBD Link

Built for Collaboration Feature List

Tips from the Team

HAProxy Logging- How to Tune Timeouts for Performance

Start Free Trial

Fully Functional for 14 Days

HAProxy (high availability proxy) is a critical part of modern systems infrastructure. It’s ideally the first point of contact for users who access your application. When configured correctly, HAProxy improves your app’s performance significantly. Through load balancing, HAProxy makes sure each service your application depends on is accessible to users, especially under load conditions otherwise impacting application performance.

I’m going to take you through the process of tuning timeouts with the intent to boost application performance. You’ll see how robust HAProxy logging can help you with troubleshooting timeout issues, and improve the performance of your application. I’ll quickly go through some of the HAProxy timeout configurations to lay a foundation.

Before we dive into the overview, let’s go over a few reasons why we need HAProxy and the logic behind it. This should help us visualize the “how” part later and understand why it’s worth going through the tuning processes.

Why You’ll Need HAProxy

The image above shows the basic design of how users access web. This method works fine if the application doesn’t get much traffic. Once the application gains traction and the number of users increases, you can see application performance begin to decline. When numerous users access the application at the same time, requests can back up and even overwhelm the application.

A good analogy is a single-lane road going from point A to point B filling up when there are too many cars. Adding HAProxy as a load balancer, is like adding lanes to the road. More lanes mean more vehicles can now travel the road. Additional lanes won’t increase speed but cars travel down the road without waiting for the car ahead to advance. If you add enough vehicles, however, even these additional lanes will become congested. This is where the concept of timeouts is essential to avoid jams.

Timeouts terminate connections after a client waits for a predetermined amount of time to access the server. This frees up connections, so active users can access the application. It’s as if the stalled cars are taken off the road, so other cars can move freely. Let’s quickly cover the various types of timeouts before we get to the tuning part.

The Three Basic HAProxy Timeouts

Just to show you what timeout configurations look like, here’s a sample including the three basic timeouts.

##based on Mesosphere Marathon’s servicerouter.py haproxy config

global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
tune.ssl.default-dh-param 2048

defaults
log global
retries 3
maxconn 2000
timeout connect 5s
timeout client 50s
timeout server 50s

Source: GitHub

1. Timeout Client

The <timeout client> setting defines the maximum time a client can be inactive when connected to the server. A common value for this timeout is five minutes. You can go with shorter timeouts, even as little as thirty seconds, if you’re attempting to maximize security or the total number of active connections. As the name suggests, this timeout is handled on the client side.

2. Timeout Connect

The <timeout connect> works like a grace period. It sets the maximum time the client has to connect to a server. The time it takes a client to connect to a server varies widely, depending on network complexity. The more complex a topology, the longer it can take the client to connect. The timeout connect allows the client to try to connect again if the initial attempt fails. In addition to the connection time, you’ll need to set the numbers of retries. The default is three, but you can adjust it to fit your environment.

3. Timeout Server

When a client sends a request to the server, it expects a response. If the server doesn’t respond in the configured time duration, a <timeout server> is invoked. This is akin to the <timeout client>, only in reverse. You can see in the list of HTTP responses, if a <timeout serve> is invoked, you’ll get a 504 Gateway Timeout response from HAProxy.

HAProxy Tuning for Good Performance

Just by configuring these three timeout values in your haproxy.cfg file, you can achieve a basic level of performance. If you want to take it up a notch, there are other timeout settings you can set to enhance performance. While the values you set will vary depending on your traffic load and environment, I’ve listed the most common configurations below. To use these, append the following to your configuration file.

timeout http-request 10s
timeout http-keep-alive 2s
timeout queue 5s
timeout tunnel 2m
timeout client-fin 1s
timeout server-fin 1s

Timeout HTTP-Request

The <timeout http-request> variable limits the total amount of time each request can persist. Aside from optimizing request performance, it can also defend against denial of service attacks by limiting the the amount of time a single request can last. Usually ten seconds is a good limit. Some php.ini files may have this setting as well, but since the proxy server is the first point of contact with an application, php.ini settings are overridden. This is true unless the application server level (php.ini) setting is shorter than the proxy determined variable.

Timeout HTTP-Keep-Alive

As the name suggests, this is a timeout designed to keep a single connection between the client and the server “alive” for a desired amount of time. While the connection is alive, all data packets can pass without needing to ask for a connection again. This essentially makes it faster to get responses from the server to the client. Keep in mind the <timeout http-request> regulates how long a single request can persist, so these two settings work hand in hand. If the <timeout http-keep-alive> isn’t set or has a value less than the <timeout http-request>, the latter is used to determine the connection status.

Timeout Queue

The <timeout queue> limits the number of concurrent connections, which can also impact performance. Setting the queue timeout shortens wait times by limiting connections and allowing clients try connecting again if the queue is full. This is similar to <timeout connect>, except <timeout queue> limits the number of connections. If you don’t set the <timeout queue>, HAProxy will default to the <timeout connect> settings to manage the queue.

Timeout Tunnel

The <timeout tunnel> variable only applies when you’re working with WebSockets. Essentially it’s <timeout keep-alive> on steroids, often with durations exceeding minutes. It may seem counterproductive and a potential security risk to keep a connection open for that long. However, used with other timeout configurations, it’s possible to maintain a safe yet high-performing connection. For instance, the <timeout http-request> variable would prevent an attack even if the <timeout tunnel> is set at several minutes. Keep in mind, though, the virtual tunnel you create by implementing this timeout requires for you to terminate it at some point using the <timeout client-fin> parameter.

Timeout Client-Fin

Say a connection drops in the middle of a client request, if you look at the HAProxy logs you’re likely to see the lost connection is a result of client-side network issues. To handle these types of situations, HAProxy creates a list of dropped client side connections. The <timeout client-fin> limits the amount of time a client request will be maintained on this list. This parameter starts ticking when a connection joins the list. Without it, the server will maintain a “maybe they’ll return” sort of connection while others are denied service. To optimize performance, the time values set for this timeout are usually short.

Timeout Server-Fin

Much like the <timeout client-fin> concept, abrupt disconnections can occur on the server side of the application, as well. An optimal setup would include redundant servers for load-balancing. When a server has too many requests, redundancy would allow you to reroute overflow requests to less busy servers and speed up response times. The <timeout server-fin> limits the time client waits for a server response before an alternate server is queried.

HAProxy Logging: How to Determine Perfect Timeouts

By now you probably can see how environment variables and traffic can help you determine the time allocations for the timeouts above. Finding the perfect balance between peak performance and optimal security is a matter of trial and error. Keeping a close eye on HAProxy logs can help you see the interactions between these different configurations and find the “sweet spot” for your application and environment. You can use HAProxy logs to understand :

  • Timestamped metrics about traffic (timing data, connections counters, traffic size)
  • Detailed event logs of HAProxy operations (content switching, filtering, persistence)
  • Request and responses (headers, status codes, payloads)
  • Session terminations and tracking where failures are occurring

Knowing when a timeout event occurs and monitoring the events preceding it are the first steps for successfully troubleshooting and tuning timeout settings. If you’re looking for an easy cloud-based log management tool, check out SolarWinds® Papertrail. Built by engineers for engineers, Papertrail can automatically parse HAProxy logs and help you quickly troubleshoot timeout issues.

It offers a simple search syntax allowing you to search all your logs from a central interface, see events in context, and pinpoint issues. The live tail feature is particularly helpful for real-time troubleshooting. If you want to start HAProxy logging with better results, sign up for a trial or request a demo.

This post was written by Taurai Mutimutema. Taurai is a systems analyst with a knack for writing, which was probably sparked by the need to document technical processes during code and implementation sessions. He enjoys learning new technology and talks about tech even more than he writes.

Aggregate, organize, and manage your logs

  • Collect real-time log data from your applications, servers, cloud services, and more
  • Search log messages to analyze and troubleshoot incidents, identify trends, and set alerts
  • Create comprehensive per-user access control policies, automated backups, and archives of up to a year of historical data
START FREE TRIAL

Fully Functional for 30 Days

Let's talk it over

Contact our team, anytime.