httpx: Modern Async HTTP Client for Python – Complete Guide

In the world of Python HTTP clients, requests has been the go-to library for years. But as applications grow and demand for performance increases, you need a client that can handle modern requirements: async/await, HTTP/2, connection pooling, and robust error handling. Enter httpx – a fully featured HTTP client for Python 3.7+ that provides both synchronous and asynchronous APIs.

Why httpx? Imagine you're building a web scraper that fetches data from 100 APIs. With requests, you'd either block on each request or juggle threads. With httpx's async client, you can fire all 100 requests concurrently with minimal overhead. Or perhaps you're building a microservice that needs to make outbound calls without blocking the event loop – httpx integrates seamlessly with asyncio.

httpx is not just an async clone of requests. It brings native HTTP/2 support (faster multiplexing), automatic content decompression, cookie persistence, and a transport layer that you can customize for proxies or mock testing. It also supports type hints, making it a joy to use in modern Python codebases.

In this tutorial, you'll learn how to install httpx, make GET and POST requests, handle timeouts and retries, work with async clients, and debug production issues. We'll cover real-world patterns like connection pooling, streaming responses, and error handling. By the end, you'll be ready to replace requests with httpx in your next project.

Installing httpx

To get started with httpx, install it via pip. It's recommended to install the full package with HTTP/2 support. If you're working in an async environment, ensure you have anyio or asyncio available.

``bash pip install httpx ``

For HTTP/2 support, install the optional h2 dependency:

``bash pip install httpx[http2] ``

That's it! You're ready to make your first request.

import httpx
response = httpx.get('https://httpbin.org/get')
print(response.status_code)

Making GET and POST Requests

httpx provides a simple API similar to requests. Use httpx.get() for synchronous GET requests and httpx.post() for POST requests. You can pass query parameters, headers, and JSON data easily.

For POST requests with JSON, use the json parameter – httpx automatically sets the Content-Type header. You can also send form data with data.

Let's see examples:

import httpx

# GET request
response = httpx.get('https://httpbin.org/get', params={'key': 'value'})
print(response.json())

# POST with JSON
response = httpx.post('https://httpbin.org/post', json={'name': 'Alice'})
print(response.status_code)

# POST with form data
response = httpx.post('https://httpbin.org/post', data={'field': 'value'})
print(response.text)

Using the Client for Connection Pooling

For production applications, you should use httpx.Client (sync) or httpx.AsyncClient (async) instead of the top-level functions. The client maintains a connection pool, reuses TCP connections, and allows you to configure defaults like base URL, headers, and timeouts.

Using a client as a context manager ensures connections are properly closed. You can also set a base URL to avoid repeating the hostname.

import httpx

with httpx.Client(base_url='https://httpbin.org') as client:
    response = client.get('/get')
    print(response.status_code)
    
    # Another request reuses the same connection
    response = client.post('/post', json={'key': 'value'})
    print(response.status_code)

Async Requests with AsyncClient

One of httpx's standout features is native async support. Use httpx.AsyncClient with async/await to perform non-blocking HTTP requests. This is ideal for I/O-bound tasks like web scraping, API aggregation, or microservices.

You can run multiple requests concurrently using asyncio.gather() or asyncio.create_task(). The async client also supports context managers.

import asyncio
import httpx

async def fetch(url):
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        return response.status_code

async def main():
    urls = ['https://httpbin.org/get', 'https://httpbin.org/post']
    results = await asyncio.gather(*(fetch(url) for url in urls))
    print(results)

asyncio.run(main())

Timeouts and Retries

Production systems must handle network failures gracefully. httpx allows you to set timeouts for the entire request or individual phases (connect, read, write). You can also configure retries using a custom transport.

Timeouts can be set per-request or on the client. For retries, use httpx.HTTPTransport with the retries parameter. Note that retries only apply to certain exceptions (like connection errors) by default.

import httpx

# Timeout per request
response = httpx.get('https://httpbin.org/delay/3', timeout=5.0)

# Client-level timeout
with httpx.Client(timeout=httpx.Timeout(10.0, connect=5.0)) as client:
    response = client.get('https://httpbin.org/delay/2')

# Retries with transport
transport = httpx.HTTPTransport(retries=3)
with httpx.Client(transport=transport) as client:
    response = client.get('https://httpbin.org/status/500')  # will retry up to 3 times

Handling Errors and Exceptions

httpx raises exceptions for network errors, timeouts, and HTTP status codes (if you call raise_for_status()). Common exceptions include httpx.ConnectError, httpx.TimeoutException, and httpx.HTTPStatusError. Always wrap your requests in try-except blocks to handle failures gracefully.

You can also inspect the response for details even when an error occurs.

import httpx

url = 'https://httpbin.org/status/404'
try:
    response = httpx.get(url)
    response.raise_for_status()
except httpx.HTTPStatusError as e:
    print(f'HTTP error: {e.response.status_code}')
except httpx.ConnectError as e:
    print(f'Connection error: {e}')
except httpx.TimeoutException as e:
    print(f'Timeout: {e}')
except Exception as e:
    print(f'Unexpected error: {e}')

Streaming Responses and File Downloads

For large responses, you can stream the content to avoid loading everything into memory. Use client.stream() for sync or async with client.stream() for async. This is useful for downloading large files or processing real-time data.

You can iterate over the response bytes or read chunks.

import httpx

with httpx.Client() as client:
    with client.stream('GET', 'https://httpbin.org/bytes/1024') as response:
        for chunk in response.iter_bytes():
            print(f'Received {len(chunk)} bytes')

Custom Headers and Authentication

You can set custom headers on the client or per request. httpx supports various authentication methods: Basic, Digest, Bearer token, and custom auth classes. Use the auth parameter.

For bearer tokens, you can pass a tuple ('token', '') or use httpx.BearerAuth.

import httpx

# Custom headers
headers = {'User-Agent': 'MyApp/1.0'}
response = httpx.get('https://httpbin.org/headers', headers=headers)

# Basic auth
response = httpx.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass'))

# Bearer token
auth = httpx.BearerAuth('my-token')
response = httpx.get('https://httpbin.org/bearer', auth=auth)
print(response.status_code)

Testing with Mock Transports

httpx provides a mock transport for testing. You can simulate responses without making real HTTP calls. This is perfect for unit tests. Use httpx.MockTransport and pass a handler function that returns a response.

You can also use respx library for more advanced mocking.

import httpx

def handler(request):
    return httpx.Response(200, json={'message': 'mock'})

client = httpx.Client(transport=httpx.MockTransport(handler))
response = client.get('https://example.com/')
print(response.json())