Home Python Python Requests Library Explained — HTTP, APIs and Real-World Patterns

Python Requests Library Explained — HTTP, APIs and Real-World Patterns

In Plain English 🔥
Imagine you walk into a restaurant. You (your Python script) tell the waiter (the Requests library) what you want. The waiter goes to the kitchen (a server on the internet), grabs your order, and brings it back to your table. You never have to know how the kitchen works — you just get your food. The Requests library is that waiter. It handles all the complicated back-and-forth of talking to the internet so you don't have to.
⚡ Quick Answer
Imagine you walk into a restaurant. You (your Python script) tell the waiter (the Requests library) what you want. The waiter goes to the kitchen (a server on the internet), grabs your order, and brings it back to your table. You never have to know how the kitchen works — you just get your food. The Requests library is that waiter. It handles all the complicated back-and-forth of talking to the internet so you don't have to.

Every modern application talks to the internet. Whether it's fetching live weather data, posting a tweet, authenticating a user via OAuth, or pulling product prices from an e-commerce API — your Python code needs a reliable, human-friendly way to make HTTP requests. The Requests library is the industry standard for doing exactly that, and it's been downloaded over 300 million times a month for good reason.

GET Requests — Asking the Internet for Data

A GET request is the most fundamental HTTP action. It means: 'Hey server, please give me this resource.' When you type a URL into your browser, your browser is making a GET request. With the Requests library, you replicate that in one readable line of Python.

But here's what's important to understand: a response is more than just the data you asked for. It's a full package — a status code (did it work?), headers (metadata about the response), and a body (the actual content). Requests gives you clean access to all three.

The .json() method is a convenience method that automatically parses JSON response bodies into a Python dictionary. This is far better than manually calling json.loads(response.text) — it's shorter, cleaner, and raises a clear error if the response body isn't valid JSON.

Always check the status code before trusting the data. A 200 means success. A 404 means the resource wasn't found. A 500 means something broke on the server side. Skipping this check is the number one mistake beginners make.

fetch_github_user.py · PYTHON
123456789101112131415161718192021222324
import requests

# The GitHub API is public and requires no auth for basic profile lookups
GITHUB_API_URL = "https://api.github.com/users/torvalds"

# Make the GET request — Requests handles the TCP connection, HTTP headers, and response parsing
response = requests.get(GITHUB_API_URL)

# Always check if the request succeeded before using the data
# raise_for_status() raises an HTTPError for 4xx and 5xx responses
response.raise_for_status()

# Parse the JSON body into a Python dictionary automatically
user_profile = response.json()

# Extract and display meaningful fields
print(f"Name        : {user_profile['name']}")
print(f"Public Repos: {user_profile['public_repos']}")
print(f"Followers   : {user_profile['followers']}")
print(f"Profile URL : {user_profile['html_url']}")

# Show the HTTP status code and the Content-Type header from the response
print(f"\nStatus Code : {response.status_code}")
print(f"Content-Type: {response.headers['Content-Type']}")
▶ Output
Name : Linus Torvalds
Public Repos: 6
Followers : 237981
Profile URL : https://github.com/torvalds

Status Code : 200
Content-Type: application/json; charset=utf-8
⚠️
Watch Out:Never assume a 200 status just because the request didn't throw an exception. `requests.get()` does NOT raise an error for 404 or 500 responses by default — it silently returns that broken response. Always call `response.raise_for_status()` immediately after the request, or check `response.status_code` explicitly.

POST Requests and Sending Data — How You Talk Back to a Server

A GET request fetches data. A POST request sends data. Think of POST as filling out and submitting a form — you're giving the server information and expecting it to do something with it, like create a new record or return a personalised result.

There are two common ways to send data with POST: as form-encoded data (data= parameter) or as JSON (json= parameter). This distinction matters enormously. When you use json=, Requests automatically serialises your Python dictionary, sets the Content-Type header to application/json, and encodes everything correctly. Using data= sends it as form fields, like a classic HTML form submission.

Most modern REST APIs expect JSON. If you send form data to a JSON API, the server will likely return a 400 Bad Request or silently ignore your payload. Always check the API docs to know which format it expects.

Query parameters are different from the body — they appear in the URL after a ?. Use the params= argument in Requests to add them cleanly rather than concatenating strings yourself.

create_github_issue.py · PYTHON
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455
import requests
import os

# Load the token from an environment variable — never hardcode secrets in source code
github_token = os.environ.get("GITHUB_TOKEN")

if not github_token:
    raise EnvironmentError("GITHUB_TOKEN environment variable is not set.")

# Target repo for creating the issue
REPO_OWNER = "your-username"
REPO_NAME  = "your-test-repo"
API_URL    = f"https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/issues"

# The payload we're sending — Requests will serialise this dict to JSON automatically
issue_payload = {
    "title": "Bug: Login page crashes on empty password field",
    "body": "Steps to reproduce:\n1. Navigate to /login\n2. Leave password blank\n3. Click Submit\n\nExpected: Validation error\nActual: 500 Internal Server Error",
    "labels": ["bug", "high-priority"]
}

# Authorization header — Bearer token is the GitHub API standard
request_headers = {
    "Authorization": f"Bearer {github_token}",
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28"
}

# Using json= automatically sets Content-Type to application/json
response = requests.post(
    url=API_URL,
    json=issue_payload,       # NOT data= — that would send form-encoded data
    headers=request_headers
)

# 201 Created is the expected success code for resource creation
response.raise_for_status()

created_issue = response.json()
print(f"Issue #{created_issue['number']} created successfully.")
print(f"Title : {created_issue['title']}")
print(f"URL   : {created_issue['html_url']}")

# Also demonstrate query params — search issues with ?state=open
search_url = f"https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/issues"
search_params = {"state": "open", "per_page": 5}

open_issues_response = requests.get(
    url=search_url,
    params=search_params,     # Requests builds the URL: ...?state=open&per_page=5
    headers=request_headers
)
open_issues_response.raise_for_status()

print(f"\nOpen issues returned: {len(open_issues_response.json())}")
▶ Output
Issue #42 created successfully.
Title : Bug: Login page crashes on empty password field
URL : https://github.com/your-username/your-test-repo/issues/42

Open issues returned: 5
⚠️
Pro Tip:Use `json=` instead of `data=json.dumps(payload)` — they look equivalent but they're not. With `data=`, you'd also need to manually set the `Content-Type: application/json` header. With `json=`, Requests handles both the serialisation AND the header automatically. One argument, zero extra steps.

Sessions, Timeouts and Retries — Writing Production-Grade Code

Here's what nobody tells beginners: using bare requests.get() and requests.post() in production code is an anti-pattern. Each call opens a fresh TCP connection. When you're hitting the same API dozens of times, that's unnecessary overhead. A Session object reuses the underlying connection, which is measurably faster.

But performance isn't the only reason to use Sessions. Sessions also persist headers, cookies, and authentication across all requests automatically. Define your auth headers once on the session — done. Every subsequent request inherits them without you repeating yourself.

Timeouts are critical. By default, requests.get() will wait forever for a server to respond. In production, that means one slow API call can hang your entire application. Always pass timeout=(connect_timeout, read_timeout) — a tuple where the first value is how long to wait for a connection and the second is how long to wait for data.

For resilience, pair a Session with urllib3's HTTPAdapter and a Retry strategy. This automatically retries on transient failures like 503 or connection drops — without you writing a retry loop manually.

resilient_api_client.py · PYTHON
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import os

def build_resilient_session(base_url: str, api_token: str) -> requests.Session:
    """
    Creates a reusable session with:
    - Shared auth headers (set once, used everywhere)
    - Automatic retries on transient failures
    - A base URL so callers only pass the path
    """
    session = requests.Session()

    # These headers are sent with EVERY request made through this session
    session.headers.update({
        "Authorization": f"Bearer {api_token}",
        "Accept": "application/json",
        "User-Agent": "TheCodeForge-Demo/1.0"
    })

    # Retry strategy: retry up to 3 times on these specific status codes
    # backoff_factor=1 means wait 1s, then 2s, then 4s between retries
    retry_strategy = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],  # Common transient failures
        allowed_methods=["GET", "POST"]               # Only retry safe or idempotent methods
    )

    # Mount the retry adapter for both HTTP and HTTPS traffic
    http_adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("https://", http_adapter)
    session.mount("http://",  http_adapter)

    return session


# --- Usage Example ---
api_token = os.environ.get("WEATHER_API_KEY", "demo_key")
BASE_URL  = "https://api.open-meteo.com/v1"

session = build_resilient_session(base_url=BASE_URL, api_token=api_token)

try:
    # timeout=(3, 10): wait 3s to connect, wait 10s to receive data
    # If either limit is exceeded, a Timeout exception is raised — not a silent hang
    response = session.get(
        url=f"{BASE_URL}/forecast",
        params={
            "latitude": 51.5074,   # London
            "longitude": -0.1278,
            "current_weather": True
        },
        timeout=(3, 10)            # ALWAYS set a timeout in production code
    )
    response.raise_for_status()

    weather_data = response.json()
    current      = weather_data["current_weather"]

    print(f"Temperature : {current['temperature']} °C")
    print(f"Wind Speed  : {current['windspeed']} km/h")
    print(f"Weather Code: {current['weathercode']}")

except requests.exceptions.Timeout:
    print("ERROR: The API took too long to respond. Try again later.")
except requests.exceptions.ConnectionError:
    print("ERROR: Could not connect. Check your internet connection.")
except requests.exceptions.HTTPError as http_err:
    print(f"ERROR: HTTP error occurred — {http_err}")
finally:
    # Always close the session to release the underlying socket connection
    session.close()
▶ Output
Temperature : 14.2 °C
Wind Speed : 18.5 km/h
Weather Code: 3
🔥
Interview Gold:When interviewers ask 'how would you handle flaky APIs in production?', the answer they want is: Session + HTTPAdapter + Retry strategy + explicit timeouts. This shows you understand that network calls are unreliable by nature, not just by accident — and that your code should be designed around that reality.

Authentication Patterns — Basic Auth, Tokens, and OAuth in the Real World

Almost every real API requires authentication. Requests supports several patterns, and knowing which one to use — and why — separates junior developers from senior ones.

Basic Auth is the simplest: a username and password are base64-encoded and sent in the Authorization header. Pass a (username, password) tuple to the auth= parameter and Requests handles the encoding. It's quick, but only acceptable over HTTPS — Basic Auth over plain HTTP exposes credentials in every request.

Token/Bearer auth is the modern standard. You get a token once (at login or via an API key), then attach it to every request as Authorization: Bearer . The Session pattern from the previous section is perfect for this — set it once on the session, never repeat yourself.

For OAuth 2.0 flows (used by Google, Spotify, GitHub Apps), the requests-oauthlib library extends Requests with full OAuth support. This is the approach for apps acting on behalf of users — you don't handle user passwords at all, which is far safer.

For API key auth, some APIs want the key as a query param, others as a header. Always check the docs and never embed keys in query params if you can avoid it — they show up in server logs.

auth_patterns_demo.py · PYTHON
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556
import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth
import os

# ─── PATTERN 1: Basic Auth ───────────────────────────────────────────────────
# Requests encodes "username:password" as Base64 and sets the Authorization header
# Only use over HTTPS — never HTTP

basic_response = requests.get(
    url="https://httpbin.org/basic-auth/demo_user/secret123",
    auth=HTTPBasicAuth("demo_user", "secret123"),  # Tuple shorthand also works: auth=("demo_user", "secret123")
    timeout=(3, 10)
)
basic_response.raise_for_status()
print("Basic Auth Result:", basic_response.json())


# ─── PATTERN 2: Bearer Token Auth ────────────────────────────────────────────
# The most common pattern for modern REST APIs
# Token is obtained once (login, API key dashboard) and reused

api_token = os.environ.get("MY_API_TOKEN", "mock_token_abc123")

token_session = requests.Session()
token_session.headers.update({
    "Authorization": f"Bearer {api_token}",
    "Content-Type": "application/json"
})

# Every request through token_session automatically includes the auth header
token_response = token_session.get(
    url="https://httpbin.org/bearer",
    timeout=(3, 10)
)
token_response.raise_for_status()
print("\nBearer Token Result:", token_response.json())
token_session.close()


# ─── PATTERN 3: API Key as Header (common with services like OpenAI, Stripe) ──
# Some APIs use a custom header name instead of the standard Authorization header

api_key = os.environ.get("SERVICE_API_KEY", "sk-demo-key-9999")

header_key_response = requests.get(
    url="https://httpbin.org/headers",
    headers={
        "X-API-Key": api_key,       # Custom header name — check your API's docs
        "Accept": "application/json"
    },
    timeout=(3, 10)
)
header_key_response.raise_for_status()

returned_headers = header_key_response.json()["headers"]
print(f"\nAPI Key sent as header: {returned_headers.get('X-Api-Key', 'not found')}")
▶ Output
Basic Auth Result: {'authenticated': True, 'user': 'demo_user'}

Bearer Token Result: {'authenticated': True, 'token': 'mock_token_abc123'}

API Key sent as header: sk-demo-key-9999
⚠️
Watch Out:Never hardcode API tokens or passwords directly in your source code. Use `os.environ.get()` or a library like `python-dotenv` to load secrets from environment variables or `.env` files. Hardcoded credentials end up in version control history, even after you delete them — and attackers scan GitHub for exactly this.
Feature / Aspectrequests.get/post (bare calls)requests.Session
TCP connection per requestNew connection every timeReuses existing connection (faster)
Shared headers / authMust repeat on every callSet once, inherited by all requests
Cookie persistenceNot persisted between callsAutomatically persisted across calls
Retry logic supportManual retry loop requiredMount HTTPAdapter with Retry strategy
Best use caseQuick scripts, one-off callsProduction code, multi-request workflows
Performance (same host)Slower — repeated handshakesFaster — connection pooling
Code verbosityLower for single requestsLower for 3+ requests to same host

🎯 Key Takeaways

  • Always call response.raise_for_status() — Requests doesn't raise exceptions on 404 or 500 by default. Without this call, your code will silently process error responses as valid data.
  • Use json=payload not data=json.dumps(payload) — the json= parameter handles both serialisation and the Content-Type header in one step. Mixing these up causes silent server-side failures.
  • Use requests.Session() for any code making multiple requests to the same host — it reuses TCP connections, persists auth headers and cookies, and is the correct foundation for production HTTP clients.
  • Timeouts are not optional in production — requests.get() waits forever by default. Always set timeout=(3, 10) or similar, and always handle requests.exceptions.Timeout explicitly so one slow API can't freeze your app.

⚠ Common Mistakes to Avoid

  • Mistake 1: Not setting a timeout — Symptom: your script hangs indefinitely when a server is slow or unresponsive, blocking your entire program — Fix: always pass timeout=(connect_seconds, read_seconds) to every request, e.g. timeout=(3, 10). A 3-second connect timeout and 10-second read timeout is a sensible default for most APIs.
  • Mistake 2: Using data=json.dumps(payload) instead of json=payload — Symptom: the server returns 400 Bad Request or ignores your payload because the Content-Type header is wrong (it defaults to application/x-www-form-urlencoded, not application/json) — Fix: use the json= parameter. Requests serialises the dict AND sets the correct Content-Type header automatically.
  • Mistake 3: Trusting a response without checking the status code — Symptom: your code silently processes an error response (404 HTML page, 500 JSON error object) as if it were valid data, causing confusing downstream bugs — Fix: call response.raise_for_status() immediately after every request. It raises requests.exceptions.HTTPError for any 4xx or 5xx response, giving you a clear, catchable error.

Interview Questions on This Topic

  • QWhat's the difference between using `requests.get()` directly and using a `requests.Session()`, and when would you choose one over the other in a production system?
  • QHow would you implement retry logic for a flaky third-party API in Python, and what HTTP status codes should trigger a retry versus an immediate failure?
  • QIf a POST request to a REST API returns a 200 OK but the body contains an error message like `{'status': 'error', 'message': 'Invalid token'}` — does `raise_for_status()` catch that? How would you handle it?

Frequently Asked Questions

How do I send JSON data in a POST request using Python Requests?

Pass your Python dictionary to the json= parameter: requests.post(url, json=your_dict). Requests will automatically serialise the dictionary to a JSON string and set the Content-Type: application/json header. Do not use data=json.dumps(your_dict) — that sends the right data but sets the wrong Content-Type header, which causes many APIs to reject the request.

What is the difference between requests.get() and requests.Session().get()?

Both make a GET request, but Session().get() reuses the underlying TCP connection across multiple calls (connection pooling), which is faster. Sessions also automatically persist headers, cookies, and authentication settings across all requests, so you don't repeat yourself. For a single one-off request, bare requests.get() is fine. For anything in production or involving multiple calls, always use a Session.

Why does my Python Requests code hang and never finish?

You almost certainly haven't set a timeout. By default, requests.get() will wait indefinitely for a server to respond — if the server is slow, down, or rate-limiting you, your script just blocks forever. Fix it by adding timeout=(connect_timeout, read_timeout) to every request, for example timeout=(3, 10). This tells Requests to raise a requests.exceptions.Timeout error if the connection takes more than 3 seconds or the response takes more than 10 seconds.

🔥
TheCodeForge Editorial Team Verified Author

Written and reviewed by senior developers with real-world experience across enterprise, startup and open-source projects. Every article on TheCodeForge is written to be clear, accurate and genuinely useful — not just SEO filler.

← PreviousSeaborn for Data VisualisationNext →Flask Web Framework Basics
Forged with 🔥 at TheCodeForge.io — Where Developers Are Forged