Latency and Throughput

Every production system is ultimately measured by two numbers: how fast it responds (latency) and how much it can handle (throughput). SLAs are written in percentiles — p99 latency under 200ms, throughput above 10,000 RPS. Getting either wrong means either angry users or an over-provisioned bill.

The counterintuitive part: optimizing purely for throughput often destroys latency. A system processing 1,000 RPS might have 5ms average latency, but as queues fill under load, that average hides the 10% of users seeing 500ms. Understanding the latency-throughput tradeoff curve and Little's Law is the foundation for capacity planning, SLO design, and performance debugging.

This is not a textbook definition. It covers how to measure latency correctly (percentiles, not averages), how the latency-throughput curve behaves near capacity, how Little's Law connects concurrency to resource sizing, and the production patterns that separate systems that scale gracefully from those that collapse under load.

Percentiles — Why Averages Lie

Average (mean) latency is the most misleading metric in production systems. It hides tail latency — the slow requests that affect your worst users. A system with 10ms average latency might have 1% of requests taking 2,000ms. The average tells you nothing about those 1%.

Percentiles solve this. The p50 (median) tells you what the typical user sees. The p99 tells you what the worst 1-in-100 users sees. The p99.9 tells you about 1-in-1,000. At scale, even small percentages translate to large absolute numbers of affected users.

import numpy as np

response_times = np.concatenate([
    np.random.normal(10, 2, 900),
    np.random.normal(200, 50, 99),
    [2000]
])

print(f'Mean (average):  {response_times.mean():.1f}ms')
print(f'p50 (median):    {np.percentile(response_times, 50):.1f}ms')
print(f'p90:             {np.percentile(response_times, 90):.1f}ms')
print(f'p95:             {np.percentile(response_times, 95):.1f}ms')
print(f'p99:             {np.percentile(response_times, 99):.1f}ms')
print(f'p99.9:           {np.percentile(response_times, 99.9):.1f}ms')
print(f'Max:             {response_times.max():.1f}ms')

Little's Law

Little's Law: L = lambda x W. Average number in system = arrival rate x average time in system. This fundamental relationship connects latency, throughput, and concurrency. It applies to any stable system — web servers, connection pools, thread pools, message queues.

The practical power of Little's Law is in capacity planning. If you know your target throughput and your average latency, you can calculate the minimum concurrency (threads, connections, workers) you need. If latency doubles, your concurrency doubles — same throughput requires double the resources.

# Little's Law: L = lambda * W
throughput_lambda = 100
latency_W = 0.050
concurrency_L = throughput_lambda * latency_W
print(f'Average concurrent requests: {concurrency_L}')

new_concurrency = 100 * 0.100
print(f'Concurrency at 100ms latency: {new_concurrency}')

required_threads = 500 * 0.020
print(f'Threads needed: {required_threads}')

db_concurrency = 200 * 0.015
print(f'DB connections needed: {db_concurrency}')

The Latency-Throughput Curve: Hockey Stick Behavior

Every system has a latency-throughput curve. At low load, latency is flat — requests rarely wait. As throughput approaches capacity, latency rises sharply. This is the 'hockey stick' curve, and it is governed by queuing theory.

The M/M/1 queuing model predicts: average response time = service_time / (1 - utilization). At 50% utilization, response time is 2x the service time. At 90%, it is 10x. At 99%, it is 100x. This is why production systems target 50-70% utilization — the steep part of the curve is unpredictable and dangerous.

service_time_ms = 10
for utilization_pct in [10, 30, 50, 70, 80, 90, 95, 99]:
    rho = utilization_pct / 100.0
    response_time = service_time_ms / (1 - rho)
    queue_time = response_time - service_time_ms
    print(f'Util: {utilization_pct:3d}% | Response: {response_time:7.1f}ms | Queue: {queue_time:7.1f}ms')

Measuring Latency Correctly: Histograms, Bucket Boundaries, and Clock Sources

Measuring latency seems simple — record the start time, record the end time, subtract. In production, it is more complex. Clock source accuracy, histogram bucket boundaries, and measurement scope (wall clock vs CPU time) all affect the correctness of your latency data.

Histograms are the standard for latency measurement in Prometheus. They pre-aggregate observations into buckets at instrumentation time, then histogram_quantile computes percentiles at query time. The bucket boundaries you choose are permanent — you cannot change them without losing time series continuity.

package io.thecodeforge.performance;

import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.atomic.AtomicLong;

public class LatencyMeasurement {

    private static final double[] BUCKET_BOUNDARIES = {
        0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.5, 1.0, 2.5, 5.0
    };

    private final ConcurrentSkipListMap<Double, AtomicLong> buckets = new ConcurrentSkipListMap<>();
    private final AtomicLong sum = new AtomicLong(0);
    private final AtomicLong count = new AtomicLong(0);

    public LatencyMeasurement() {
        for (double boundary : BUCKET_BOUNDARIES) {
            buckets.put(boundary, new AtomicLong(0));
        }
        buckets.put(Double.POSITIVE_INFINITY, new AtomicLong(0));
    }

    public void observe(double latencySeconds) {
        count.incrementAndGet();
        sum.addAndGet((long) (latencySeconds * 1_000_000_000));
        for (var entry : buckets.tailMap(latencySeconds, true).entrySet()) {
            entry.getValue().incrementAndGet();
        }
    }

    public double percentile(double p) {
        long totalCount = count.get();
        if (totalCount == 0) return 0;
        double rank = p * totalCount;
        double prevBoundary = 0;
        long prevCount = 0;
        for (var entry : buckets.entrySet()) {
            long currentCount = entry.getValue().get();
            if (currentCount >= rank) {
                double currentBoundary = entry.getKey();
                if (currentBoundary == Double.POSITIVE_INFINITY) return prevBoundary;
                double bucketWidth = currentBoundary - prevBoundary;
                long bucketCount = currentCount - prevCount;
                if (bucketCount == 0) return currentBoundary;
                double fraction = (rank - prevCount) / bucketCount;
                return prevBoundary + fraction * bucketWidth;
            }
            prevBoundary = entry.getKey();
            prevCount = currentCount;
        }
        return prevBoundary;
    }
}

Frequently Asked Questions