TimeoutException in Headless: Viewport and UA Fixes

The suite is green on your laptop and red in CI. Same code, same locators, same waits — except CI runs headless, and there every other test dies with TimeoutException. You raise timeouts and the failures move but never leave. The page is not slower in CI; it is a different page: narrower layout, bolder bot defenses, and rendering quirks that only exist without a display.

Headless-only timeouts mislead because waits look like the problem. Engineers tune durations for a layout that will never satisfy the condition — a desktop button that does not exist in the mobile view, a widget blocked by a headless-detecting challenge. No timeout value fixes a condition that is structurally unreachable. The cure is making headless behave like headed, then debugging any remainder with eyes on.

This guide closes the headed-headless gap systematically. You will learn the viewport sizing that restores desktop layouts, the user-agent and flag hygiene that calms bot defenses, the GPU and font differences that stall rendering, and the headed-debug capture that shows you the page CI actually sees. By the end, headless is a deployment detail, not a second test suite.

The Viewport Gap: Size the Invisible Window

The single most common headless-only failure is an unset window size. Without explicit sizing, headless Chrome falls back to a small default viewport that responsive sites read as a phone, serving hamburger menus, stacked layouts, and hidden desktop controls. Your locators target a desktop page that was never rendered — every wait on it is doomed from the first poll, at any timeout value.

The fix is one flag plus verification: --window-size=1920,1080 in the shared driver factory, covering width and height together since responsive breakpoints watch both. Verify by logging driver.get_window_size at session start and screenshotting early failures into build artifacts. Teams that add the size assertion once never revisit this class — the log line makes regressions self-announcing.

Beware half-fixes: set_window_size after load can miss the initial layout pass on some pages, and maximizing a headless window is meaningless without a display manager. Set the size via flags before session start so first paint already uses desktop geometry. The snippet shows the factory shape with the log line. Viewport discipline is unglamorous and decisive — half of all headless mysteries end at this flag.

from selenium import webdriver

def make_headless_driver(width=1920, height=1080):
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_argument(f"--window-size={width},{height}")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    driver = webdriver.Chrome(options=options)
    size = driver.get_window_size()  # prove it in every build log
    print("viewport", size["width"], "x", size["height"])
    assert size["width"] >= 1280, f"viewport too narrow: {size}"
    return driver

User-Agents and Flags That Calm Bot Defenses

Headless browsers carry detectable signatures, and bot-mitigation vendors read them eagerly. Default headless user-agent strings, the webdriver navigator flag, missing plugin arrays, and datacenter IP ranges each add suspicion; combined, they route your tests into challenges, rate limits, or blank responses that look exactly like slow pages. Every wait on a challenged page times out innocently while the real story sits in the screenshot.

Hygiene starts with a standard desktop user-agent matching your pinned Chrome major version — never a phone string, never an ancient desktop one. Prefer --headless=new, which shares the headed pipeline and sheds the oldest detection markers. Keep automation honest: evading defenses with stealth plugins may violate the site's terms and rots with every vendor update. Where defenses persist, the durable answers are test environments with defenses relaxed or vendor-approved test keys, negotiated once and owned explicitly.

The snippet shows the flag and user-agent setup with a challenge detector you can run after load. Check page titles and body markers for challenge text before blaming waits — a five-line detector in setup saves hours of timeout tuning against a page that will never load. Identity first, patience second: no timeout outlasts a block.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

DESKTOP_UA = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/126.0.0.0 Safari/537.36")

def make_stealth_honest_driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1920,1080")
    options.add_argument(f"user-agent={DESKTOP_UA}")
    options.add_argument("--disable-blink-features=AutomationControlled")
    return webdriver.Chrome(options=options)

def is_challenged(driver):
    title = driver.title.lower()
    return any(k in title for k in ("just a moment", "attention required",
                                    "access denied", "verify you are"))

GPU, Fonts, and Rendering That Stall Waits

The old headless pipeline skipped GPU compositing, which changed canvas output, WebGL availability, and animation timing in ways headed runs never showed. Font gaps compound it: minimal CI images ship few typefaces, so text metrics shift, canvas screenshots differ pixel by pixel, and icon fonts render as empty boxes that visibility checks may still accept while visual assertions fail. These are rendering divergences, not speed problems, and patience cannot fix them.

Close the gap in layers. Adopt --headless=new to share the headed pipeline, add software GL fallbacks like --use-gl=swiftshader where the suite needs canvas determinism, and install the font packages your app uses into the CI image so metrics match design machines. Screenshot the same page headed and headless during migration and diff them — residual differences after flags and fonts are the true product bugs worth filing.

The snippet shows the rendering-hardened factory plus a comparison helper for migration week. Treat pixel assertions as environment-sensitive by design: run them against explicit baselines per mode or restrict them to headed lanes. Waits prove states; screenshots prove paint. When paint differs by pipeline, fix the pipeline first and trust the waits again afterward.

from selenium import webdriver

def make_render_ready_driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")  # shares headed pipeline
    options.add_argument("--window-size=1920,1080")
    options.add_argument("--use-gl=swiftshader")  # software GL fallback
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--font-render-hinting=none")
    return webdriver.Chrome(options=options)

# Image prep (Dockerfile): install the app's fonts so metrics match.
#   RUN apt-get update && apt-get install -y fonts-liberation \
#       fonts-dejavu-core fontconfig && fc-cache -f

Lazy Loading and Scroll Observers Without a Display

Infinite scrolls, lazy images, and reveal-on-scroll animations depend on scroll observers that behave differently without a real display. Headed scrolling fires intersection callbacks through the compositor; some headless configurations deliver them late or not at all until scripted scrolling forces the issue. Tests that wait for below-fold content then time out on elements no observer ever revealed — another structurally unreachable condition.

The deterministic answer is scripted scrolling as a first-class helper: scroll the target into view, advance through sentinel positions, and wait for the content marker after each step. Pair each scroll with a short explicit wait on the expected batch rather than one long wait on the final item, so progress is visible step by step. Where the app supports it, test hooks that disable virtualization or preload fixtures remove the observer from the equation entirely — faster and steadier than choreographing scrolls.

The snippet shows a scroll-until-loaded helper with bounded steps and a per-batch wait. Cap the steps so a genuinely broken feed fails in seconds, not at the suite timeout. Log loaded counts per step for evidence. Observers are contracts with the viewport: when the viewport is virtual, drive it explicitly and the content follows.

from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

ITEMS = (By.CSS_SELECTOR, ".feed .card")

def scroll_until(driver, minimum, max_steps=20, timeout=5):
    loaded = 0
    for _ in range(max_steps):
        loaded = len(driver.find_elements(*ITEMS))
        if loaded >= minimum:
            return loaded
        driver.execute_script(
            "window.scrollTo(0, document.body.scrollHeight);")
        try:
            WebDriverWait(driver, timeout).until(
                lambda d: len(d.find_elements(*ITEMS)) > loaded)
        except TimeoutException:
            break  # feed stalled: fail fast with the count below
    assert loaded >= minimum, f"only {loaded} cards loaded"
    return loaded

Headed Debug Capture: Seeing What CI Sees

Some divergences survive every flag, and then you need eyes. The headed-debug lane reproduces the CI environment with visibility: run the failing subset headed on identical hardware, or under Xvfb on the CI machine itself, capturing screenshots at each step plus page source at failure. Xvfb provides a virtual display Frederick Douglas would recognize — real geometry, headed pipeline — isolating display-mode effects from machine effects cleanly.

Build capture into the framework so it is always on: screenshots on failure saved to build artifacts, page source alongside, console logs where the app emits them. A failing headless test without artifacts is an unsolved mystery by choice; with artifacts it is a ten-minute read. Name artifacts by test and timestamp so parallel runs never overwrite each other.

The snippet shows a pytest fixture that captures on failure plus the Xvfb command for the diagnostic lane. Keep the diagnostic lane manual or nightly — Xvfb headed runs cost more than pure headless — but keep capture always on. Visibility is a feature of the test system, not a favor developers request after the fact. When CI can show you its screen, headless stops being a black box and starts being just another configuration of the same trusted suite.

import os
import time
import pytest

ARTIFACTS = "/tmp/ci-artifacts"

@pytest.fixture
def headed_debug(driver, request):
    yield driver  # test runs here
    if request.node.rep_call.failed:
        os.makedirs(ARTIFACTS, exist_ok=True)
        stamp = f"{request.node.name}-{int(time.time())}"
        driver.get_screenshot_as_file(f"{ARTIFACTS}/{stamp}.png")
        with open(f"{ARTIFACTS}/{stamp}.html", "w") as fh:
            fh.write(driver.page_source)

# Diagnostic lane on the CI machine with a virtual display:
#   xvfb-run -a -s "-screen 0 1920x1080x24" pytest tests/ -x -k headless_flake

Converging the Modes: One Suite, Both Displays

The end state is a single suite that passes in both modes, with differences confined to configuration. One driver factory takes a headed flag; CI runs headless with the hardened flags, developers run headed for visibility, and nightly runs sample the opposite mode to catch drift. Mode-specific skips are a code smell — each one concedes a divergence instead of fixing it, and the list grows until the modes are effectively two suites.

Enforce convergence with a mode-matrix job: the same tests, both modes, compared weekly. New divergences file bugs against the environment or the app with screenshots attached, and the matrix stays green by repair rather than exemption. Pin the browser milestone across both so version drift never masquerades as mode drift.

The payoff compounds: developers reproduce CI failures locally by flipping one flag, CI trusts local green builds, and headless becomes what it should be — a display detail. Convergence work feels slow until the month with zero headless-only tickets arrives. The snippet pattern is a factory parameterized by mode plus the matrix invocation. One suite, two displays, zero exemptions: that is the standard that ends headless-only timeouts as a category rather than as individual tickets.