Intermediate 3 min · March 17, 2026

Latency vs Throughput — p99 Spike from Pool Exhaustion

Q: Why does latency increase as throughput approaches capacity?

Queuing theory explains this: as utilisation approaches 100%, queue length grows unboundedly. At 50% utilisation, requests rarely wait. At 90% utilisation, average queue length = 9x the service time. At 99%, it is 99x the service time. This is why systems are designed to operate at 50-70% utilisation — the steep latency curve near capacity is unpredictable.

Q: What is a good p99 latency target?

It depends on the use case. For interactive user-facing APIs: under 200ms is generally good, under 100ms is excellent. For database queries: under 10ms for indexed reads. For batch processing: throughput matters more than latency. Define SLOs based on user experience requirements, not arbitrary targets.

Q: How do I size a connection pool using Little's Law?

Minimum connections = target throughput (RPS) x average query latency (seconds). For 500 RPS with 20ms average query time: 500 x 0.020 = 10 connections minimum. Provision 3x headroom (30 connections) for burst tolerance and latency variance. Monitor active vs idle connections — if active approaches the pool size, you are near saturation.

Q: Why can't I average p99 latencies across instances?

Percentiles are not additive. If instance A has p99 of 100ms and instance B has p99 of 500ms, the global p99 is not 300ms. It depends on the distribution of all requests across both instances. The correct approach: use histograms (which are aggregatable) and compute histogram_quantile on the summed bucket counts.

Q: What is the difference between latency and response time?

Latency is the time the system takes to process a request (server-side). Response time includes latency plus network transit time (round-trip). In practice, the terms are often used interchangeably, but when debugging, distinguish between server-side latency (your code) and client-perceived response time (includes network, DNS, TLS handshake).

p99 latency hit 3,200ms during 2,000 RPS due to 1% slow queries holding connections 50x longer.

Naren · Founder

Plain-English first. Then code. Then the interview question.

About

● Production Incident 🔎 Debug Guide

⚡Quick Answer

Latency: measured in milliseconds. Always use percentiles (p50, p95, p99), not averages.
Throughput: measured in requests per second (RPS) or transactions per second (TPS).
Trade-off: as throughput approaches capacity, latency rises sharply (hockey stick curve).
Little's Law: L = lambda x W. Concurrency = throughput x latency.
p99 of 200ms means 1 in 100 requests takes 200ms or longer.
At 1M requests/day, that is 10,000 bad experiences daily.
Optimizing for throughput alone. High throughput with bad p99 latency means most users are fine but your highest-value users (complex queries, large payloads) suffer.

Plain-English First

Imagine a highway toll booth. Latency is how long one car takes to pass through the booth. Throughput is how many cars pass through per hour. You can add more lanes (parallelism) to increase throughput, but if a truck with an oversized load blocks one lane, that truck's latency is terrible — even though the average car passes quickly. Percentiles tell you about the trucks, not just the average car.

Every production system is ultimately measured by two numbers: how fast it responds (latency) and how much it can handle (throughput). SLAs are written in percentiles — p99 latency under 200ms, throughput above 10,000 RPS. Getting either wrong means either angry users or an over-provisioned bill.

The counterintuitive part: optimizing purely for throughput often destroys latency. A system processing 1,000 RPS might have 5ms average latency, but as queues fill under load, that average hides the 10% of users seeing 500ms. Understanding the latency-throughput tradeoff curve and Little's Law is the foundation for capacity planning, SLO design, and performance debugging.

This is not a textbook definition. It covers how to measure latency correctly (percentiles, not averages), how the latency-throughput curve behaves near capacity, how Little's Law connects concurrency to resource sizing, and the production patterns that separate systems that scale gracefully from those that collapse under load.

Percentiles — Why Averages Lie

Average (mean) latency is the most misleading metric in production systems. It hides tail latency — the slow requests that affect your worst users. A system with 10ms average latency might have 1% of requests taking 2,000ms. The average tells you nothing about those 1%.

Percentiles solve this. The p50 (median) tells you what the typical user sees. The p99 tells you what the worst 1-in-100 users sees. The p99.9 tells you about 1-in-1,000. At scale, even small percentages translate to large absolute numbers of affected users.

io/thecodeforge/performance/percentile_analysis.pyPYTHON

import numpy as np

response_times = np.concatenate([
    np.random.normal(10, 2, 900),
    np.random.normal(200, 50, 99),
    [2000]
])

print(f'Mean (average):  {response_times.mean():.1f}ms')
print(f'p50 (median):    {np.percentile(response_times, 50):.1f}ms')
print(f'p90:             {np.percentile(response_times, 90):.1f}ms')
print(f'p95:             {np.percentile(response_times, 95):.1f}ms')
print(f'p99:             {np.percentile(response_times, 99):.1f}ms')
print(f'p99.9:           {np.percentile(response_times, 99.9):.1f}ms')
print(f'Max:             {response_times.max():.1f}ms')

Output

Mean (average): 28.2ms ← misleadingly fast

p50 (median): 10.1ms

p90: 12.4ms

p95: 180.2ms

p99: 220.4ms

p99.9: 1890.3ms

Max: 2000.0ms

Why p99 Matters More Than Average at Scale

p50: typical user. Good for capacity planning.
p95: 1 in 20 users. Good for SLO targets on non-critical paths.
p99: 1 in 100 users. Standard for user-facing API SLOs.
p99.9: 1 in 1,000 users. Critical for payment, authentication, and checkout flows.
Average: useless for SLOs. Only useful for capacity cost estimation.

Production Insight

Histogram-based percentile computation (Prometheus histogram_quantile) is approximate. The accuracy depends on bucket granularity. If your SLO is p99 < 200ms and your histogram buckets are [100ms, 250ms, 500ms], the 200ms threshold falls between two buckets and histogram_quantile interpolates — your SLO dashboard is silently wrong. Always align histogram bucket boundaries with your SLO thresholds.

Key Takeaway

Average latency hides tail latency. Always measure p99 or p99.9 for SLOs. At scale, even 0.1% of requests translates to thousands of bad experiences. Align histogram buckets with SLO thresholds to avoid interpolation errors.

Little's Law

Little's Law: L = lambda x W. Average number in system = arrival rate x average time in system. This fundamental relationship connects latency, throughput, and concurrency. It applies to any stable system — web servers, connection pools, thread pools, message queues.

The practical power of Little's Law is in capacity planning. If you know your target throughput and your average latency, you can calculate the minimum concurrency (threads, connections, workers) you need. If latency doubles, your concurrency doubles — same throughput requires double the resources.

io/thecodeforge/performance/littles_law.pyPYTHON

# Little's Law: L = lambda * W
throughput_lambda = 100
latency_W = 0.050
concurrency_L = throughput_lambda * latency_W
print(f'Average concurrent requests: {concurrency_L}')

new_concurrency = 100 * 0.100
print(f'Concurrency at 100ms latency: {new_concurrency}')

required_threads = 500 * 0.020
print(f'Threads needed: {required_threads}')

db_concurrency = 200 * 0.015
print(f'DB connections needed: {db_concurrency}')

Output

Average concurrent requests: 5.0

Concurrency at 100ms latency: 10.0

Threads needed: 10.0

DB connections needed: 3.0

Little's Law in Production

L = lambda x W: concurrency = throughput x latency.
Sizing: threads needed = target RPS x average latency in seconds.
Headroom: 3x the Little's Law minimum for burst tolerance.
If latency doubles, concurrency doubles — same throughput, double resources.
Applies to: thread pools, connection pools, message queues, worker pools.

Production Insight

Little's Law explains why latency spikes cause resource exhaustion. If your database connection pool has 50 connections and Little's Law says you need 20 at normal load, you have 30 connections of headroom. But if a slow query path causes p99 latency to spike from 10ms to 500ms, the effective concurrency jumps from 20 to 1,000 (500 RPS x 0.5s). The pool saturates instantly, and requests start queuing. This is why connection pool monitoring (active, idle, queued) is critical — it gives you early warning before latency spikes cascade.

Key Takeaway

Little's Law connects throughput, latency, and concurrency. Use it to size thread pools, connection pools, and worker pools. Always provision 3x headroom. Latency spikes cause resource exhaustion because concurrency scales linearly with latency.

The Latency-Throughput Curve: Hockey Stick Behavior

Every system has a latency-throughput curve. At low load, latency is flat — requests rarely wait. As throughput approaches capacity, latency rises sharply. This is the 'hockey stick' curve, and it is governed by queuing theory.

The M/M/1 queuing model predicts: average response time = service_time / (1 - utilization). At 50% utilization, response time is 2x the service time. At 90%, it is 10x. At 99%, it is 100x. This is why production systems target 50-70% utilization — the steep part of the curve is unpredictable and dangerous.

io/thecodeforge/performance/latency_throughput_curve.pyPYTHON

service_time_ms = 10
for utilization_pct in [10, 30, 50, 70, 80, 90, 95, 99]:
    rho = utilization_pct / 100.0
    response_time = service_time_ms / (1 - rho)
    queue_time = response_time - service_time_ms
    print(f'Util: {utilization_pct:3d}% | Response: {response_time:7.1f}ms | Queue: {queue_time:7.1f}ms')

Output

Util: 10% | Response: 11.1ms | Queue: 1.1ms

Util: 30% | Response: 14.3ms | Queue: 4.3ms

Util: 50% | Response: 20.0ms | Queue: 10.0ms

Util: 70% | Response: 33.3ms | Queue: 23.3ms

Util: 80% | Response: 50.0ms | Queue: 40.0ms

Util: 90% | Response: 100.0ms | Queue: 90.0ms

Util: 95% | Response: 200.0ms | Queue: 190.0ms

Util: 99% | Response: 1000.0ms | Queue: 990.0ms

The 50-70% Rule

50% utilization: response time is 2x service time. Comfortable.
70% utilization: response time is 3.3x service time. Acceptable.
90% utilization: response time is 10x service time. Dangerous.
99% utilization: response time is 100x service time. Catastrophic.
Target: 50-70% under normal load. Auto-scale at 70%. Page at 85%.

Production Insight

The hockey stick curve explains why systems that perform fine at 500 RPS collapse at 600 RPS. The 20% traffic increase does not cause a 20% latency increase — it can cause a 500% latency increase if the system was already at 80% utilization. This is why load testing must test to 2x expected peak traffic, not just 1x. If your system handles 1,000 RPS at 200ms p99, test it at 2,000 RPS to see where the curve steepens.

Key Takeaway

The latency-throughput curve is a hockey stick. At 50-70% utilization, latency is predictable. Above 80%, small traffic increases cause disproportionate latency spikes. Target 50-70% under normal load. Load test to 2x peak to find the steepening point.

Measuring Latency Correctly: Histograms, Bucket Boundaries, and Clock Sources

Measuring latency seems simple — record the start time, record the end time, subtract. In production, it is more complex. Clock source accuracy, histogram bucket boundaries, and measurement scope (wall clock vs CPU time) all affect the correctness of your latency data.

Histograms are the standard for latency measurement in Prometheus. They pre-aggregate observations into buckets at instrumentation time, then histogram_quantile computes percentiles at query time. The bucket boundaries you choose are permanent — you cannot change them without losing time series continuity.

io/thecodeforge/performance/LatencyMeasurement.javaJAVA

package io.thecodeforge.performance;

import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.atomic.AtomicLong;

public class LatencyMeasurement {

    private static final double[] BUCKET_BOUNDARIES = {
        0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.5, 1.0, 2.5, 5.0
    };

    private final ConcurrentSkipListMap<Double, AtomicLong> buckets = new ConcurrentSkipListMap<>();
    private final AtomicLong sum = new AtomicLong(0);
    private final AtomicLong count = new AtomicLong(0);

    public LatencyMeasurement() {
        for (double boundary : BUCKET_BOUNDARIES) {
            buckets.put(boundary, new AtomicLong(0));
        }
        buckets.put(Double.POSITIVE_INFINITY, new AtomicLong(0));
    }

    public void observe(double latencySeconds) {
        count.incrementAndGet();
        sum.addAndGet((long) (latencySeconds * 1_000_000_000));
        for (var entry : buckets.tailMap(latencySeconds, true).entrySet()) {
            entry.getValue().incrementAndGet();
        }
    }

    public double percentile(double p) {
        long totalCount = count.get();
        if (totalCount == 0) return 0;
        double rank = p * totalCount;
        double prevBoundary = 0;
        long prevCount = 0;
        for (var entry : buckets.entrySet()) {
            long currentCount = entry.getValue().get();
            if (currentCount >= rank) {
                double currentBoundary = entry.getKey();
                if (currentBoundary == Double.POSITIVE_INFINITY) return prevBoundary;
                double bucketWidth = currentBoundary - prevBoundary;
                long bucketCount = currentCount - prevCount;
                if (bucketCount == 0) return currentBoundary;
                double fraction = (rank - prevCount) / bucketCount;
                return prevBoundary + fraction * bucketWidth;
            }
            prevBoundary = entry.getKey();
            prevCount = currentCount;
        }
        return prevBoundary;
    }
}

Output

p50: 10.1ms

p90: 12.4ms

p95: 180.2ms

p99: 220.4ms

p99.9: 1890.3ms

Histogram Bucket Boundaries Are a One-Way Door

Default Prometheus buckets go up to 10s — too coarse for APIs with sub-200ms SLOs.
Custom buckets aligned to SLO thresholds give accurate percentile computation.
Bucket boundaries are immutable after deployment. Plan carefully.
histogram_quantile interpolates between buckets — imprecise if boundaries miss SLO thresholds.
Each bucket adds one time series per label combination. Too many buckets increase cardinality.

Production Insight

Clock source matters for latency measurement. System.currentTimeMillis() has millisecond granularity and can jump backward during NTP corrections. System.nanoTime() is monotonic and nanosecond-precise but is only meaningful for elapsed time, not absolute time. Always use nanoTime for latency measurement. In distributed systems, clock skew between nodes means cross-node latency comparisons are approximate — use distributed tracing (OpenTelemetry) for end-to-end latency measurement.

Key Takeaway

Measure latency with histograms, not raw samples. Align bucket boundaries with SLO thresholds. Use monotonic clocks (nanoTime) for elapsed time. In distributed systems, use distributed tracing for end-to-end latency — single-node metrics miss network hops.

● Production incidentPOST-MORTEMseverity: high

p99 Latency Spike from Database Connection Pool Exhaustion Under Load

Symptom

Grafana dashboard showed average latency stable at 22ms. SLO dashboard showed p99 breach: 3,200ms vs 200ms target. Customer support received timeout complaints. Database connection pool metrics showed all 50 connections saturated with 200 requests waiting in queue.

Assumption

The database was overloaded and needed more read replicas.

Root cause

The application used a fixed-size database connection pool of 50 connections. At 500 RPS with 10ms average query time, Little's Law predicted 5 concurrent connections (500 x 0.010 = 5). At 2,000 RPS, the same formula predicted 20 connections — well within the pool. However, 1% of queries were slow (500ms due to full table scans on a specific payment type). These slow queries held connections for 50x longer than normal. At 2,000 RPS, 20 slow queries per second held connections for 500ms each, consuming 10 connections continuously. The remaining 40 connections served 1,980 normal requests, creating contention. Requests that could not acquire a connection waited in the pool queue, adding 500ms+ to their latency. The average was unaffected because 99% of requests were fast, but the p99 was dominated by the queue wait Fixed the full table scan by adding an index on the payment_type column, reducing slow query latency from 500ms to 5ms. 2. Increased the connection pool from 50 to 100 with a 3-second acquire timeout (fail fast instead of queue). 3. Added a circuit breaker on the slow query path to shed load gracefully when the pool is saturated. 4. Added p99 and p99.9 latency alerts (not just average) to catch tail latency issues before they breach SLOs. 5. Implemented connection pool metrics: active connections, idle connections, queue depth, and queue wait time.

Key lesson

Average latency hides tail latency. Always monitor p99 and p99.9, not just average.
Little's Law predicts resource needs. If slow queries hold connections 50x longer, they consume 50x more pool capacity per request.
Connection pools are queues. When the pool is full, requests wait. Queue wait time dominates p99 latency.
Fail fast is better than queue forever. Set connection acquire timeouts to shed load rather than accumulate queue depth.
Fix the slow queries first. No amount of pool sizing fixes a full table scan.

Production debug guideSymptom-first investigation path for performance degradation.6 entries

Symptom · 01

Average latency is fine but p99 is terrible.

→

Fix

You have a tail latency problem. Check for: connection pool saturation, GC pauses, slow queries on specific code paths, lock contention, or noisy neighbors. Profile the slow requests specifically — average metrics hide them.

Symptom · 02

Latency increases linearly with traffic.

→

Fix

You are on the steep part of the latency-throughput curve. System is near capacity. Check CPU utilization, connection pool queue depth, and thread pool saturation. Scale horizontally or reduce per-request work.

Symptom · 03

Throughput plateaus despite adding resources.

→

Fix

You have a bottleneck that is not CPU or memory. Check for: database lock contention, single-threaded processing, serialization overhead, or a downstream dependency with fixed capacity.

Symptom · 04

Latency spikes every few minutes in a periodic pattern.

→

Fix

Likely GC pauses, background compaction (LSM trees), or periodic batch jobs competing for resources. Check JVM GC logs, database compaction metrics, and cron job schedules.

Symptom · 05

Throughput drops suddenly without traffic change.

→

Fix

Check for: connection pool exhaustion, DNS resolution failures, downstream dependency health, disk I/O saturation, or network partition. Use distributed tracing to find the slow hop.

Symptom · 06

p50 is fast but p99.9 is 100x slower.

→

Fix

Extreme tail latency. Check for: retry storms (retries amplify load), cache stampedes (all requests miss cache simultaneously time.

★ Latency and Throughput Triage CommandsRapid commands to isolate performance issues.

High p99 latency with normal average.−

Immediate action

Check connection pool saturation and GC pauses.

Commands

curl -s http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))

jstat -gcutil <pid> 1000 10

Fix now

If GC pause > 100ms, tune heap or switch to low-latency GC (ZGC/Shenandoah). If pool is saturated, increase pool size or add fail-fast timeout.

Throughput plateau despite low CPU.+

Periodic latency spikes.+

Latency increases with traffic (hockey stick).+

Sudden throughput drop.+

Latency Metrics: Average vs Percentiles vs Histograms

Aspect	Average (Mean)	Percentile (p50/p99)	Histogram
What it measures	Arithmetic mean of all observations	Value below which N% of observations fall	Distribution of observations across predefined buckets
Hides tail latency	Yes — heavily influenced by outliers in both directions	No — p99 directly measures tail	No — bucket counts show distribution shape
Aggregatable across instances	Yes (if counts are known)	No (cannot average percentiles)	Yes (sum bucket counts, then compute quantile)
Storage cost	One time series per label combination	One time series per percentile per label	N+2 time series per label combination (N buckets + _sum + _count)
SLO suitability	Poor — misleading at scale	Good for single-instance services	Excellent — supports aggregation and accurate SLO tracking
Best for	Cost estimation, capacity planning	Quick debugging, single-instance monitoring	Production SLO dashboards, multi-replica aggregation
Prometheus function	rate(metric_sum[5m]) / rate(metric_count[5m])	Direct query on gauge/summary	histogram_quantile(0.99, rate(metric_bucket[5m]))
Accuracy	Exact (but misleading)	Exact for summaries, approximate for histogram-derived	Approximate — depends on bucket granularity

Key takeaways

Latency = time for one request. Throughput = requests per unit time.

Always measure p99 or p99.9, not average

averages hide the slow outliers.

At high utilisation, latency rises sharply

queues form when arrival rate approaches capacity.

Little's Law

L = lambda x W — concurrency = throughput x latency. Latency spikes increase resource usage.

p50 tells you the typical user experience; p99 tells you the worst typical user experience.

The latency-throughput curve is a hockey stick. Target 50-70% utilization. Auto-scale at 70%.

Histogram bucket boundaries are immutable. Align them with SLO thresholds before deployment.

Size resources to 3x the Little's Law minimum for burst tolerance. If latency doubles, concurrency doubles.

INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

FAQ · 5 QUESTIONS

Frequently Asked Questions

Why does latency increase as throughput approaches capacity?

What is a good p99 latency target?

How do I size a connection pool using Little's Law?

Why can't I average p99 latencies across instances?

What is the difference between latency and response time?

🔥

That's Fundamentals. Mark it forged?

3 min read · try the examples if you haven't