Python Requests Library Explained — HTTP, APIs and Real-World Patterns
Every modern application talks to the internet. Whether it's fetching live weather data, posting a tweet, authenticating a user via OAuth, or pulling product prices from an e-commerce API — your Python code needs a reliable, human-friendly way to make HTTP requests. The Requests library is the industry standard for doing exactly that, and it's been downloaded over 300 million times a month for good reason.
GET Requests — Asking the Internet for Data
A GET request is the most fundamental HTTP action. It means: 'Hey server, please give me this resource.' When you type a URL into your browser, your browser is making a GET request. With the Requests library, you replicate that in one readable line of Python.
But here's what's important to understand: a response is more than just the data you asked for. It's a full package — a status code (did it work?), headers (metadata about the response), and a body (the actual content). Requests gives you clean access to all three.
The .json() method is a convenience method that automatically parses JSON response bodies into a Python dictionary. This is far better than manually calling json.loads(response.text) — it's shorter, cleaner, and raises a clear error if the response body isn't valid JSON.
Always check the status code before trusting the data. A 200 means success. A 404 means the resource wasn't found. A 500 means something broke on the server side. Skipping this check is the number one mistake beginners make.
import requests # The GitHub API is public and requires no auth for basic profile lookups GITHUB_API_URL = "https://api.github.com/users/torvalds" # Make the GET request — Requests handles the TCP connection, HTTP headers, and response parsing response = requests.get(GITHUB_API_URL) # Always check if the request succeeded before using the data # raise_for_status() raises an HTTPError for 4xx and 5xx responses response.raise_for_status() # Parse the JSON body into a Python dictionary automatically user_profile = response.json() # Extract and display meaningful fields print(f"Name : {user_profile['name']}") print(f"Public Repos: {user_profile['public_repos']}") print(f"Followers : {user_profile['followers']}") print(f"Profile URL : {user_profile['html_url']}") # Show the HTTP status code and the Content-Type header from the response print(f"\nStatus Code : {response.status_code}") print(f"Content-Type: {response.headers['Content-Type']}")
Public Repos: 6
Followers : 237981
Profile URL : https://github.com/torvalds
Status Code : 200
Content-Type: application/json; charset=utf-8
POST Requests and Sending Data — How You Talk Back to a Server
A GET request fetches data. A POST request sends data. Think of POST as filling out and submitting a form — you're giving the server information and expecting it to do something with it, like create a new record or return a personalised result.
There are two common ways to send data with POST: as form-encoded data (data= parameter) or as JSON (json= parameter). This distinction matters enormously. When you use json=, Requests automatically serialises your Python dictionary, sets the Content-Type header to application/json, and encodes everything correctly. Using data= sends it as form fields, like a classic HTML form submission.
Most modern REST APIs expect JSON. If you send form data to a JSON API, the server will likely return a 400 Bad Request or silently ignore your payload. Always check the API docs to know which format it expects.
Query parameters are different from the body — they appear in the URL after a ?. Use the params= argument in Requests to add them cleanly rather than concatenating strings yourself.
import requests import os # Load the token from an environment variable — never hardcode secrets in source code github_token = os.environ.get("GITHUB_TOKEN") if not github_token: raise EnvironmentError("GITHUB_TOKEN environment variable is not set.") # Target repo for creating the issue REPO_OWNER = "your-username" REPO_NAME = "your-test-repo" API_URL = f"https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/issues" # The payload we're sending — Requests will serialise this dict to JSON automatically issue_payload = { "title": "Bug: Login page crashes on empty password field", "body": "Steps to reproduce:\n1. Navigate to /login\n2. Leave password blank\n3. Click Submit\n\nExpected: Validation error\nActual: 500 Internal Server Error", "labels": ["bug", "high-priority"] } # Authorization header — Bearer token is the GitHub API standard request_headers = { "Authorization": f"Bearer {github_token}", "Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28" } # Using json= automatically sets Content-Type to application/json response = requests.post( url=API_URL, json=issue_payload, # NOT data= — that would send form-encoded data headers=request_headers ) # 201 Created is the expected success code for resource creation response.raise_for_status() created_issue = response.json() print(f"Issue #{created_issue['number']} created successfully.") print(f"Title : {created_issue['title']}") print(f"URL : {created_issue['html_url']}") # Also demonstrate query params — search issues with ?state=open search_url = f"https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/issues" search_params = {"state": "open", "per_page": 5} open_issues_response = requests.get( url=search_url, params=search_params, # Requests builds the URL: ...?state=open&per_page=5 headers=request_headers ) open_issues_response.raise_for_status() print(f"\nOpen issues returned: {len(open_issues_response.json())}")
Title : Bug: Login page crashes on empty password field
URL : https://github.com/your-username/your-test-repo/issues/42
Open issues returned: 5
Sessions, Timeouts and Retries — Writing Production-Grade Code
Here's what nobody tells beginners: using bare requests.get() and requests.post() in production code is an anti-pattern. Each call opens a fresh TCP connection. When you're hitting the same API dozens of times, that's unnecessary overhead. A Session object reuses the underlying connection, which is measurably faster.
But performance isn't the only reason to use Sessions. Sessions also persist headers, cookies, and authentication across all requests automatically. Define your auth headers once on the session — done. Every subsequent request inherits them without you repeating yourself.
Timeouts are critical. By default, requests.get() will wait forever for a server to respond. In production, that means one slow API call can hang your entire application. Always pass timeout=(connect_timeout, read_timeout) — a tuple where the first value is how long to wait for a connection and the second is how long to wait for data.
For resilience, pair a Session with urllib3's HTTPAdapter and a Retry strategy. This automatically retries on transient failures like 503 or connection drops — without you writing a retry loop manually.
import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry import os def build_resilient_session(base_url: str, api_token: str) -> requests.Session: """ Creates a reusable session with: - Shared auth headers (set once, used everywhere) - Automatic retries on transient failures - A base URL so callers only pass the path """ session = requests.Session() # These headers are sent with EVERY request made through this session session.headers.update({ "Authorization": f"Bearer {api_token}", "Accept": "application/json", "User-Agent": "TheCodeForge-Demo/1.0" }) # Retry strategy: retry up to 3 times on these specific status codes # backoff_factor=1 means wait 1s, then 2s, then 4s between retries retry_strategy = Retry( total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504], # Common transient failures allowed_methods=["GET", "POST"] # Only retry safe or idempotent methods ) # Mount the retry adapter for both HTTP and HTTPS traffic http_adapter = HTTPAdapter(max_retries=retry_strategy) session.mount("https://", http_adapter) session.mount("http://", http_adapter) return session # --- Usage Example --- api_token = os.environ.get("WEATHER_API_KEY", "demo_key") BASE_URL = "https://api.open-meteo.com/v1" session = build_resilient_session(base_url=BASE_URL, api_token=api_token) try: # timeout=(3, 10): wait 3s to connect, wait 10s to receive data # If either limit is exceeded, a Timeout exception is raised — not a silent hang response = session.get( url=f"{BASE_URL}/forecast", params={ "latitude": 51.5074, # London "longitude": -0.1278, "current_weather": True }, timeout=(3, 10) # ALWAYS set a timeout in production code ) response.raise_for_status() weather_data = response.json() current = weather_data["current_weather"] print(f"Temperature : {current['temperature']} °C") print(f"Wind Speed : {current['windspeed']} km/h") print(f"Weather Code: {current['weathercode']}") except requests.exceptions.Timeout: print("ERROR: The API took too long to respond. Try again later.") except requests.exceptions.ConnectionError: print("ERROR: Could not connect. Check your internet connection.") except requests.exceptions.HTTPError as http_err: print(f"ERROR: HTTP error occurred — {http_err}") finally: # Always close the session to release the underlying socket connection session.close()
Wind Speed : 18.5 km/h
Weather Code: 3
Authentication Patterns — Basic Auth, Tokens, and OAuth in the Real World
Almost every real API requires authentication. Requests supports several patterns, and knowing which one to use — and why — separates junior developers from senior ones.
Basic Auth is the simplest: a username and password are base64-encoded and sent in the Authorization header. Pass a (username, password) tuple to the auth= parameter and Requests handles the encoding. It's quick, but only acceptable over HTTPS — Basic Auth over plain HTTP exposes credentials in every request.
Token/Bearer auth is the modern standard. You get a token once (at login or via an API key), then attach it to every request as Authorization: Bearer . The Session pattern from the previous section is perfect for this — set it once on the session, never repeat yourself.
For OAuth 2.0 flows (used by Google, Spotify, GitHub Apps), the requests-oauthlib library extends Requests with full OAuth support. This is the approach for apps acting on behalf of users — you don't handle user passwords at all, which is far safer.
For API key auth, some APIs want the key as a query param, others as a header. Always check the docs and never embed keys in query params if you can avoid it — they show up in server logs.
import requests from requests.auth import HTTPBasicAuth, HTTPDigestAuth import os # ─── PATTERN 1: Basic Auth ─────────────────────────────────────────────────── # Requests encodes "username:password" as Base64 and sets the Authorization header # Only use over HTTPS — never HTTP basic_response = requests.get( url="https://httpbin.org/basic-auth/demo_user/secret123", auth=HTTPBasicAuth("demo_user", "secret123"), # Tuple shorthand also works: auth=("demo_user", "secret123") timeout=(3, 10) ) basic_response.raise_for_status() print("Basic Auth Result:", basic_response.json()) # ─── PATTERN 2: Bearer Token Auth ──────────────────────────────────────────── # The most common pattern for modern REST APIs # Token is obtained once (login, API key dashboard) and reused api_token = os.environ.get("MY_API_TOKEN", "mock_token_abc123") token_session = requests.Session() token_session.headers.update({ "Authorization": f"Bearer {api_token}", "Content-Type": "application/json" }) # Every request through token_session automatically includes the auth header token_response = token_session.get( url="https://httpbin.org/bearer", timeout=(3, 10) ) token_response.raise_for_status() print("\nBearer Token Result:", token_response.json()) token_session.close() # ─── PATTERN 3: API Key as Header (common with services like OpenAI, Stripe) ── # Some APIs use a custom header name instead of the standard Authorization header api_key = os.environ.get("SERVICE_API_KEY", "sk-demo-key-9999") header_key_response = requests.get( url="https://httpbin.org/headers", headers={ "X-API-Key": api_key, # Custom header name — check your API's docs "Accept": "application/json" }, timeout=(3, 10) ) header_key_response.raise_for_status() returned_headers = header_key_response.json()["headers"] print(f"\nAPI Key sent as header: {returned_headers.get('X-Api-Key', 'not found')}")
Bearer Token Result: {'authenticated': True, 'token': 'mock_token_abc123'}
API Key sent as header: sk-demo-key-9999
| Feature / Aspect | requests.get/post (bare calls) | requests.Session |
|---|---|---|
| TCP connection per request | New connection every time | Reuses existing connection (faster) |
| Shared headers / auth | Must repeat on every call | Set once, inherited by all requests |
| Cookie persistence | Not persisted between calls | Automatically persisted across calls |
| Retry logic support | Manual retry loop required | Mount HTTPAdapter with Retry strategy |
| Best use case | Quick scripts, one-off calls | Production code, multi-request workflows |
| Performance (same host) | Slower — repeated handshakes | Faster — connection pooling |
| Code verbosity | Lower for single requests | Lower for 3+ requests to same host |
🎯 Key Takeaways
- Always call
response.raise_for_status()— Requests doesn't raise exceptions on 404 or 500 by default. Without this call, your code will silently process error responses as valid data. - Use
json=payloadnotdata=json.dumps(payload)— thejson=parameter handles both serialisation and the Content-Type header in one step. Mixing these up causes silent server-side failures. - Use
requests.Session()for any code making multiple requests to the same host — it reuses TCP connections, persists auth headers and cookies, and is the correct foundation for production HTTP clients. - Timeouts are not optional in production —
requests.get()waits forever by default. Always settimeout=(3, 10)or similar, and always handlerequests.exceptions.Timeoutexplicitly so one slow API can't freeze your app.
⚠ Common Mistakes to Avoid
- ✕Mistake 1: Not setting a timeout — Symptom: your script hangs indefinitely when a server is slow or unresponsive, blocking your entire program — Fix: always pass
timeout=(connect_seconds, read_seconds)to every request, e.g.timeout=(3, 10). A 3-second connect timeout and 10-second read timeout is a sensible default for most APIs. - ✕Mistake 2: Using
data=json.dumps(payload)instead ofjson=payload— Symptom: the server returns 400 Bad Request or ignores your payload because the Content-Type header is wrong (it defaults to application/x-www-form-urlencoded, not application/json) — Fix: use thejson=parameter. Requests serialises the dict AND sets the correct Content-Type header automatically. - ✕Mistake 3: Trusting a response without checking the status code — Symptom: your code silently processes an error response (404 HTML page, 500 JSON error object) as if it were valid data, causing confusing downstream bugs — Fix: call
response.raise_for_status()immediately after every request. It raisesrequests.exceptions.HTTPErrorfor any 4xx or 5xx response, giving you a clear, catchable error.
Interview Questions on This Topic
- QWhat's the difference between using `requests.get()` directly and using a `requests.Session()`, and when would you choose one over the other in a production system?
- QHow would you implement retry logic for a flaky third-party API in Python, and what HTTP status codes should trigger a retry versus an immediate failure?
- QIf a POST request to a REST API returns a 200 OK but the body contains an error message like `{'status': 'error', 'message': 'Invalid token'}` — does `raise_for_status()` catch that? How would you handle it?
Frequently Asked Questions
How do I send JSON data in a POST request using Python Requests?
Pass your Python dictionary to the json= parameter: requests.post(url, json=your_dict). Requests will automatically serialise the dictionary to a JSON string and set the Content-Type: application/json header. Do not use data=json.dumps(your_dict) — that sends the right data but sets the wrong Content-Type header, which causes many APIs to reject the request.
What is the difference between requests.get() and requests.Session().get()?
Both make a GET request, but Session().get() reuses the underlying TCP connection across multiple calls (connection pooling), which is faster. Sessions also automatically persist headers, cookies, and authentication settings across all requests, so you don't repeat yourself. For a single one-off request, bare requests.get() is fine. For anything in production or involving multiple calls, always use a Session.
Why does my Python Requests code hang and never finish?
You almost certainly haven't set a timeout. By default, requests.get() will wait indefinitely for a server to respond — if the server is slow, down, or rate-limiting you, your script just blocks forever. Fix it by adding timeout=(connect_timeout, read_timeout) to every request, for example timeout=(3, 10). This tells Requests to raise a requests.exceptions.Timeout error if the connection takes more than 3 seconds or the response takes more than 10 seconds.
Written and reviewed by senior developers with real-world experience across enterprise, startup and open-source projects. Every article on TheCodeForge is written to be clear, accurate and genuinely useful — not just SEO filler.