Skip to content
Home Python Recursion in Python — 1200-Level JSON Hits Recursion Limit

Recursion in Python — 1200-Level JSON Hits Recursion Limit

Where developers are forged. · Structured learning · Free forever.
📍 Part of: Functions → Topic 7 of 11
1200-level nested JSON triggers RecursionError past Python's 1000 limit.
⚙️ Intermediate — basic Python knowledge assumed
In this tutorial, you'll learn
1200-level nested JSON triggers RecursionError past Python's 1000 limit.
  • Every recursive function needs a base case that returns without calling itself, and a recursive case that provably moves closer to that base case on every call — skip either and you get RecursionError.
  • Python pushes a new stack frame for every recursive call — the default limit is 1000, which is a deliberate design choice to keep tracebacks readable, not a bug to work around for normal use.
  • Naive recursive Fibonacci is O(2^n) and will time out on inputs above ~40 — a single @functools.lru_cache decorator converts it to O(n) with zero logic changes.
✦ Plain-English analogy ✦ Real code with output ✦ Interview questions
Quick Answer
  • Recursion solves self-similar problems by having a function call itself on smaller inputs
  • Every recursive function needs a base case (stops) and a recursive case (moves toward base)
  • Python uses the call stack: each call pushes a new frame; default limit is 1000 frames
  • Naive recursive Fibonacci is O(2^n) — one @lru_cache decorator drops it to O(n)
  • Use recursion for tree-shaped data; use iteration for flat sequences
  • Missing base case or not moving toward it = RecursionError: maximum recursion depth exceeded
🚨 START HERE

Quick Debug Cheat Sheet: Recursion Errors

One-liners and immediate actions for the most common recursion failures in production.
🟡

RecursionError on production data

Immediate ActionAdd depth counter and raise custom exception before RecursionError
Commands
import sys; print(sys.getrecursionlimit())
sys.setrecursionlimit(20000) # temporary, not recommended
Fix NowRewrite the recursive function to use an explicit stack (list) and loop.
🟠

Naive recursive Fibonacci extremely slow (n > 30)

Immediate ActionAdd memoisation decorator
Commands
@functools.lru_cache(maxsize=None)
from functools import lru_cache
Fix NowReplace recursion with iterative bottom-up DP using a loop.
🟡

Stack overflow from deep file system walk

Immediate ActionConvert to iterative using `os.walk` or explicit stack
Commands
import os; for root, dirs, files in os.walk(path): ...
stack = [path]; while stack: current = stack.pop()
Fix NowUse `os.walk()` which is iterative internally, or implement your own stack.
Production Incident

Production API Falls Over on Deeply Nested JSON Payload

A product catalog API crashed with `RecursionError` every time a supplier uploaded a deeply nested configuration file. The team had never seen it because all test data was shallow.
SymptomAPI returns 500 on specific supplier uploads. Error logs show: RecursionError: maximum recursion depth exceeded while calling a Python object. The error happens during JSON parsing of nested product categories.
AssumptionThe team assumed the recursion depth could never exceed a few dozen levels. They used json.loads() with a custom object_hook that recursively processes nested keys, but never tested with nesting beyond 100.
Root causeThe supplier's configuration file used nested categories up to 1200 levels deep. Python's default recursion limit is 1000. The recursive object_hook hit the limit and threw RecursionError, crashing the entire request.
FixReplaced the recursive object hook with an iterative stack-based approach. Added a max_depth parameter with a default of 200, raising a custom ValidationError for deeper nesting. Also added Sentry alerting for depth warnings.
Key Lesson
Never trust user-provided data depth — always impose a recursion limit or use iterative processing for untrusted input.Raise the recursion limit only after confirming no infinite recursion exists; it's a temporary bandage, not a fix.When processing nested JSON from external sources, consider iterative flattening or a depth-limited recursion wrapper.
Production Debug Guide

Symptom-driven strategy for common recursion failures in production.

RecursionError: maximum recursion depth exceededFirst, add a debug print of the input argument. If the argument never reaches the base case, fix the recursive step. If it does but depth is legitimately high, either increase sys.setrecursionlimit() temporarily (with caution) or rewrite iteratively.
Function returns unexpectedly slow or memory grows continuouslyCheck if the recursion is recomputing overlapping subproblems. Add @functools.lru_cache to memoize results. If caching doesn't help, profile with cProfile to identify repeated calls.
Traceback is so long it's unreadableUse faulthandler.enable() to dump C stack traces. For Python tracebacks, limit recursion depth in your function with a depth parameter and return an error instead of recursing deeper. Alternatively, log only the last 10 frames.
Recursive function works on small input but fails silently on larger dataThe data may be too deep for the default recursion limit. Add an explicit depth counter and raise a custom exception when it exceeds a safe threshold. Use iterative flattening for untrusted data.

Recursion sounds academic until you're staring at a problem that has no clean iterative solution — parsing a nested JSON structure, walking a file system, or solving a tree traversal. At that point, a recursive function is not just elegant, it's the natural language the problem is written in. Real codebases use recursion constantly: Django's template engine, Python's ast module, and virtually every algorithm that operates on trees or graphs rely on it.

The problem recursion solves is 'self-similar structure'. Some problems contain smaller versions of themselves — the classic example being a folder that contains other folders. Writing a loop to handle arbitrarily deep nesting is painful and brittle. A recursive function handles it gracefully because it says: 'I know how to process one folder. If there are subfolders, I'll just call myself on each of them.' The logic stays simple no matter how deeply nested the data gets.

By the end of this article you'll understand exactly how Python executes recursive calls using the call stack, why a missing base case crashes your program, how to write clean recursive solutions for real problems like tree traversal and directory scanning, and — critically — when recursion is the wrong tool and iteration is the better choice.

How Python Actually Executes a Recursive Function — The Call Stack

Every time you call a function in Python, the interpreter pushes a 'frame' onto the call stack — a small record containing the function's local variables and where to return when it finishes. When a function calls itself recursively, a brand new frame is pushed on top. This keeps happening until the base case is hit. Then each frame resolves from the top down, like peeling layers off an onion.

This is the mental model that makes recursion click. You're not replacing the current call — you're stacking a new one on top of it. The original call is patiently waiting, paused mid-execution, until the deeper calls resolve and hand their results back up the chain.

Python's default stack limit is 1000 frames. That's not a bug — it's a safety net. Without it, infinite recursion would silently eat all your RAM. Once you understand the stack, you understand why recursion has that limit, why it uses more memory than a loop, and why tail-call optimisation (which erases finished frames) is something Python deliberately doesn't implement — Guido van Rossum wanted tracebacks to stay readable.

call_stack_visualised.py · PYTHON
1234567891011121314151617181920212223242526
import sys

def countdown(current_number):
    """
    Counts down from current_number to 1, then prints 'Go!'.
    Each recursive call gets its own 'current_number' variable on the stack.
    """
    # BASE CASE — the simplest version of the problem we can answer directly.
    # Without this, the function would call itself forever and hit RecursionError.
    if current_number == 0:
        print("Go!")
        return  # stops the recursion and starts unwinding the stack

    # RECURSIVE CASE — we print the current number, then ask the function
    # to handle the rest of the countdown (current_number - 1).
    print(f"Stack depth: {current_number} | Calling countdown({current_number - 1})")
    countdown(current_number - 1)

    # This line runs AFTER the recursive call returns.
    # It proves the original call was paused, not destroyed.
    print(f"Stack depth: {current_number} | Back from countdown({current_number - 1}) — unwinding")

print(f"Python's default recursion limit: {sys.getrecursionlimit()}")
print("--- Starting countdown ---")
countdown(3)
▶ Output
Python's default recursion limit: 1000
--- Starting countdown ---
Stack depth: 3 | Calling countdown(2)
Stack depth: 2 | Calling countdown(1)
Stack depth: 1 | Calling countdown(0)
Go!
Stack depth: 1 | Back from countdown(0) — unwinding
Stack depth: 2 | Back from countdown(1) — unwinding
Stack depth: 3 | Back from countdown(2) — unwinding
🔥The Golden Rule of Recursion:
Every recursive function needs exactly two things: (1) a base case that returns without calling itself, and (2) a recursive case that moves closer to the base case with every call. If your recursive case doesn't get closer to the base case, you have infinite recursion — full stop.
📊 Production Insight
Python's 1000-frame limit is low enough that a single malformed JSON payload can crash your API.
If you're processing untrusted input, always add a depth counter and cap it before hitting the limit.
The limit is a safety net — not a target. Hitting it in production means your function is broken or your data is too deep.
🎯 Key Takeaway
Every recursive call pushes a frame; they unwind only after the base case.
A missing base case or non-converging recursion = RecursionError.
Write the base case first, then the recursive case that moves toward it.

Writing Real Recursive Solutions — Factorial, Fibonacci, and File Systems

Let's move from theory into three progressively real examples. Factorial is the classic teaching example — but we'll use it to lock in the base case / recursive case pattern. Fibonacci shows the classic performance trap beginners fall into. The file system walker is what recursion actually looks like in production code.

Factorial of n (written n!) means n × (n-1) × (n-2) × ... × 1. The definition itself is recursive: n! = n × (n-1)!. That makes the recursive solution feel like translating English directly into Python.

Fibonacci shows something important: naive recursion can be catastrophically slow because it recalculates the same values dozens of times. We'll compare the naive version with a memoised version using functools.lru_cache — a one-line fix that transforms it from toy code into something usable. The file system example is the payoff: showing you a real task where recursion is objectively the cleanest tool.

recursion_real_examples.py · PYTHON
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107
import os
import functools
import time

# ─────────────────────────────────────────────
# EXAMPLE 1: Factorial — the pattern template
# ─────────────────────────────────────────────
def factorial(number):
    """
    Returns the factorial of a non-negative integer.
    factorial(5) → 5 × 4 × 3 × 2 × 1 = 120
    """
    if number < 0:
        raise ValueError(f"Factorial is not defined for negative numbers, got {number}")

    # Base case: 0! and 1! are both defined as 1
    if number <= 1:
        return 1

    # Recursive case: n! = n × (n-1)!
    # We trust that factorial(number - 1) will return the right answer —
    # this trust is what makes recursion work. Focus on one level at a time.
    return number * factorial(number - 1)

print("=== Factorial ===")
for n in [0, 1, 5, 10]:
    print(f"  factorial({n}) = {factorial(n)}")


# ─────────────────────────────────────────────
# EXAMPLE 2: Fibonacci — naive vs memoised
# ─────────────────────────────────────────────
def fibonacci_naive(position):
    """
    Returns the Fibonacci number at the given position.
    fibonacci(0)=0, fibonacci(1)=1, fibonacci(6)=8
    WARNING: exponential time complexity O(2^n). Melts for n > 35.
    """
    if position <= 0:
        return 0
    if position == 1:
        return 1
    # This creates a binary tree of calls. fibonacci(6) calls fibonacci(5)
    # AND fibonacci(4). fibonacci(5) also calls fibonacci(4). That's wasteful.
    return fibonacci_naive(position - 1) + fibonacci_naive(position - 2)

# @lru_cache stores results so we never compute the same position twice.
# This converts the time complexity from O(2^n) to O(n). One decorator, massive gain.
@functools.lru_cache(maxsize=None)
def fibonacci_memoised(position):
    """Same logic as naive, but results are cached after the first call."""
    if position <= 0:
        return 0
    if position == 1:
        return 1
    return fibonacci_memoised(position - 1) + fibonacci_memoised(position - 2)

print("\n=== Fibonacci: Naive vs Memoised ===")
test_position = 35

start = time.perf_counter()
naive_result = fibonacci_naive(test_position)
naive_duration = time.perf_counter() - start

start = time.perf_counter()
memo_result = fibonacci_memoised(test_position)
memo_duration = time.perf_counter() - start

print(f"  fibonacci({test_position}) = {naive_result}")
print(f"  Naive time:     {naive_duration:.4f}s")
print(f"  Memoised time:  {memo_duration:.6f}s")
print(f"  Speedup:        ~{naive_duration / memo_duration:.0f}x faster")


# ─────────────────────────────────────────────
# EXAMPLE 3: Recursive file system walk
# This is real production-style recursion.
# ─────────────────────────────────────────────
def list_python_files(directory_path, indent_level=0):
    """
    Recursively lists all .py files under directory_path,
    showing folder structure with indentation.
    """
    indent = "  " * indent_level  # visual depth indicator

    try:
        entries = os.listdir(directory_path)
    except PermissionError:
        print(f"{indent}[Permission denied: {directory_path}]")
        return  # graceful base case — can't go deeper, so stop

    for entry_name in sorted(entries):
        full_path = os.path.join(directory_path, entry_name)

        if os.path.isdir(full_path):
            # It's a folder — recurse into it, one level deeper
            print(f"{indent}📁 {entry_name}/")
            list_python_files(full_path, indent_level + 1)

        elif entry_name.endswith(".py"):
            # It's a Python file — base case, just print it
            size_kb = os.path.getsize(full_path) / 1024
            print(f"{indent}  🐍 {entry_name} ({size_kb:.1f} KB)")

print("\n=== Python Files in Current Directory ===")
list_python_files(".")
▶ Output
=== Factorial ===
factorial(0) = 1
factorial(1) = 1
factorial(5) = 120
factorial(10) = 3628800

=== Fibonacci: Naive vs Memoised ===
fibonacci(35) = 9227465
Naive time: 2.8431s
Memoised time: 0.000021s
Speedup: ~135428x faster

=== Python Files in Current Directory ===
📁 examples/
🐍 call_stack_visualised.py (0.8 KB)
🐍 recursion_real_examples.py (2.1 KB)
💡Pro Tip: The 'Leap of Faith' Technique
When writing a recursive function, don't try to trace every call in your head — that way madness lies. Instead, assume the function already works correctly for smaller inputs. Your only job is to write the one-step reduction: 'if I had the answer for n-1, how do I get the answer for n?' Write that, add the base case, and trust the stack to do the rest.
📊 Production Insight
Naive recursive Fibonacci hits O(2^n) time — with n=50 it would take over a decade to compute.
In production, any overlapping-subproblem recursion must be memoised or converted to iteration.
The @lru_cache decorator is a one-line fix, but it adds memory overhead; for extreme cases use bottom-up DP.
🎯 Key Takeaway
Use @lru_cache to memoise recursive functions with overlapping subproblems.
Without it, exponential complexity is not theoretical — it's a production outage waiting to happen.
The leap of faith method works: trust the call to smaller input, write the one-step logic, and define the base case.

Recursion vs Iteration — Choosing the Right Tool Without Dogma

Recursion and iteration are equally powerful — any recursive function can be rewritten as a loop with an explicit stack, and vice versa. The real question is: which version is easier to read, maintain, and reason about for this specific problem?

Recursion wins when the data structure is inherently recursive — trees, graphs, nested lists, file systems, HTML/XML parsing. The recursive solution mirrors the shape of the data. The iterative version would require you to manually maintain a stack, which is just reimplementing what Python does for you automatically.

Iteration wins for linear sequences. Summing a list, processing rows in a CSV, generating a range of numbers — these are flat, sequential operations. Writing them recursively adds overhead and a risk of hitting the recursion limit with no benefit. Python also lacks tail-call optimisation, which means even a theoretically 'tail-recursive' function doesn't get the memory savings it would in Haskell or Scheme.

The honest rule: if you can draw the problem as a tree, recursion probably fits. If it's a straight line, use a loop.

recursion_vs_iteration.py · PYTHON
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879
# Comparing recursive and iterative approaches for two problems.
# Problem A: Sum a flat list — iteration wins clearly.
# Problem B: Flatten a deeply nested list — recursion wins clearly.

from typing import Any

# ─────────────────────────────────────────────
# PROBLEM A: Summing a flat list
# ─────────────────────────────────────────────
def sum_recursive(number_list):
    """Recursive sum — technically works, but awkward for a flat list."""
    if not number_list:          # base case: empty list sums to zero
        return 0
    # Take the first element, add it to the sum of everything else.
    # Python creates a new list slice on every call — that's O(n²) memory.
    return number_list[0] + sum_recursive(number_list[1:])

def sum_iterative(number_list):
    """Iterative sum — this is how you should actually do it."""
    running_total = 0
    for number in number_list:   # O(n) time, O(1) extra memory
        running_total += number
    return running_total

sample_numbers = [10, 20, 30, 40, 50]
print("=== Flat List Sum ===")
print(f"  Recursive: {sum_recursive(sample_numbers)}")   # works, but wasteful
print(f"  Iterative: {sum_iterative(sample_numbers)}")   # clean and efficient
print(f"  Built-in:  {sum(sample_numbers)}")             # just use this in real life


# ─────────────────────────────────────────────
# PROBLEM B: Flattening an arbitrarily nested list
# ─────────────────────────────────────────────
def flatten_nested_list(nested: Any) -> list:
    """
    Converts an arbitrarily nested list into a flat list.
    flatten_nested_list([1, [2, [3, 4]], 5]) → [1, 2, 3, 4, 5]

    The iterative version of this requires manually managing a stack.
    The recursive version just... describes what flat means.
    """
    flat_result = []

    for item in nested:
        if isinstance(item, list):
            # This item is itself a list — recurse into it.
            # We don't know how deep it goes, and we don't need to.
            flat_result.extend(flatten_nested_list(item))
        else:
            # Base case: it's a plain value, just collect it.
            flat_result.append(item)

    return flat_result

weirdly_nested = [1, [2, 3], [4, [5, [6, 7]]], 8, [9, [10]]]
print("\n=== Flatten Nested List ===")
print(f"  Input:  {weirdly_nested}")
print(f"  Output: {flatten_nested_list(weirdly_nested)}")

# For contrast, here's what the iterative version looks like.
# It's correct, but you have to manage the stack manually — more surface area for bugs.
def flatten_iterative(nested: Any) -> list:
    """Iterative flatten using an explicit stack. Compare complexity with above."""
    flat_result = []
    stack = list(nested)         # seed the stack with top-level items

    while stack:
        item = stack.pop(0)      # take from the front to preserve order
        if isinstance(item, list):
            stack = list(item) + stack   # push inner items back onto the front
        else:
            flat_result.append(item)

    return flat_result

print(f"  Iterative output: {flatten_iterative(weirdly_nested)}")
print("  Both give the same result. Recursive version is easier to reason about.")
▶ Output
=== Flat List Sum ===
Recursive: 150
Iterative: 150
Built-in: 150

=== Flatten Nested List ===
Input: [1, [2, 3], [4, [5, [6, 7]]], 8, [9, [10]]]
Output: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Iterative output: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Both give the same result. Recursive version is easier to reason about.
⚠ Watch Out: Recursion Depth on Real Data
If you're recursing over user-provided data — like a nested JSON payload from an API — you don't control how deep it goes. A malicious or malformed payload with 1001 levels of nesting will crash your app with RecursionError. Always add a max_depth parameter or switch to an iterative approach with an explicit stack for production code that handles untrusted input.
📊 Production Insight
Choosing recursion when iteration will do adds unnecessary stack pressure and memory overhead.
The default 1000-frame limit means that even a moderate depth (800 levels) can fail on a large list.
In production, use the shape of the data as your guide: tree = recursion, list = loop.
🎯 Key Takeaway
Recursion mirrors tree-shaped data; iteration mirrors flat sequences.
Python's lack of tail-call optimisation means recursion always costs O(depth) stack frames.
If you can't draw a tree, you probably don't need recursion.

Recursion Depth and Production Safety — Guarding Against Stack Overflow

The 1000-frame default limit is generous for most real-world trees (a file system deeper than 20 levels is rare). But external data — JSON, XML, user-defined configs — can go arbitrarily deep. A malicious payload with 2000 levels will kill your recursive parser before it can say 'RecursionError'.

The fix isn't to raise the limit. Raising the limit just moves the crash point and can cause Python to segfault if it exhausts real stack memory. The correct patterns are:

  1. Add a depth parameter with a safe maximum, and raise a custom exception.
  2. Use iterative traversal with an explicit stack (a list) for untrusted data.
  3. Use sys.setrecursionlimit() only as a last resort — and only after verifying no infinite recursion exists.

Here's a production-safe recursive function that never exceeds the limit:

safe_recursive_processor.py · PYTHON
123456789101112131415161718192021222324252627282930313233
MAX_RECURSION_DEPTH = 200

class RecursionDepthExceededError(Exception):
    """Raised when recursion goes deeper than the safety limit."""
    pass

def process_nested_data(data, depth=0):
    """
    Recursively processes nested data with an explicit depth guard.
    Raises RecursionDepthExceededError if depth exceeds MAX_RECURSION_DEPTH.
    """
    if depth > MAX_RECURSION_DEPTH:
        raise RecursionDepthExceededError(
            f"Max recursion depth {MAX_RECURSION_DEPTH} exceeded. "
            "Data is too deeply nested. Consider splitting the input."
        )

    if isinstance(data, dict):
        return {key: process_nested_data(value, depth + 1) for key, value in data.items()}
    elif isinstance(data, list):
        return [process_nested_data(item, depth + 1) for item in data]
    else:
        # Base case: plain value, process it
        return str(data)

# Example usage
try:
    deep_config = {"a": {"b": {"c": "value"}}}  # depth 3
    result = process_nested_data(deep_config)
    print("Processed:", result)
except RecursionDepthExceededError as e:
    print(f"Safety error: {e}")
▶ Output
Processed: {'a': {'b': {'c': 'value'}}}
Mental Model
Mental Model: The Stack Is a Shared Resource
Think of the call stack like a shared memory buffer — every recursive call consumes space, and the buffer is finite.
  • Python's default stack size is ~1 MB (but OS-dependent), shared across all calls.
  • Each frame uses roughly 8 KB overhead, so 1000 frames ≈ 8 MB of stack — safe for most systems.
  • But at 2000 frames you risk 16+ MB, which can cause segfaults or OOM in constrained environments.
  • A single recursive function shouldn't consume the whole stack — leave room for other call chains (logging, error handlers, etc.).
📊 Production Insight
A common pattern in incident debriefs: 'We increased the recursion limit and the app segfaulted next week.'
The real stack grows with every function call, not just your recursive one. Network calls, logging, and middleware all consume frames.
Limit depth explicitly in your recursive function rather than raising the global limit.
🎯 Key Takeaway
Guard recursion depth with a parameter, not a global limit.
Raise the global limit only after proving no other option exists.
Prefer iterative stacks for untrusted input — they don't blow the Python call stack.

Recursive Tree Traversals — The Production Pattern You Actually Use

You'll rarely hand-write a recursive factorial in production. But you will walk DOM trees, parse AST nodes, evaluate nested expressions, and serialize hierarchical data. These are all recursive by nature. The pattern is always the same: visit a node, process it, recurse into its children, combine results.

Let's look at a common production example: parsing nested HTML-like tags (or any nested structure) into a flat list of contents, preserving hierarchy with an indent level. This is essentially what linters, formatters, and template engines do internally.

The decision tree for when to use recursive traversal is simple: if the data has a 'children' attribute (or equivalent) and you need to process all descendants, recursion is almost always the clearest path.

recursive_html_parser.py · PYTHON
123456789101112131415161718192021222324252627282930
def parse_nested_tags(children, depth=0):
    """
    Recursively parse a list of tag objects with 'tag' and 'child_tags' fields.
    Returns a list of (depth, tag_name) tuples.
    This pattern applies to JSON, ASTs, and XML.
    """
    result = []
    for child in children:
        # Process current node
        result.append((depth, child['tag']))
        # Recurse into children if they exist
        if 'child_tags' in child:
            result.extend(parse_nested_tags(child['child_tags'], depth + 1))
    return result

# Example: simplified HTML-like structure
html_dom = [
    {'tag': 'div', 'child_tags': [
        {'tag': 'p', 'child_tags': [
            {'tag': 'span', 'child_tags': []},
            {'tag': 'strong', 'child_tags': []}
        ]},
        {'tag': 'img', 'child_tags': []}
    ]},
    {'tag': 'footer', 'child_tags': []}
]

print(parse_nested_tags(html_dom))
# Output: [(0, 'div'), (1, 'p'), (2, 'span'), (2, 'strong'), (1, 'img'), (0, 'footer')]
▶ Output
[(0, 'div'), (1, 'p'), (2, 'span'), (2, 'strong'), (1, 'img'), (0, 'footer')]
🔥The Recursive Bucket Brigade
When using recursion on trees, you're passing results upward (like buckets of water up a line). Each level adds its own contribution and passes the accumulated result downward (or upward). In the example above, each depth level just records its tag and extends with children. The recursion is effectively a depth-first traversal.
📊 Production Insight
A production logging system recursively serialised message metadata and hit the recursion limit on a 800-deep forwarding chain.
The fix was to switch to an explicit stack (list) for the serialisation, adding a max depth warning.
Always plan for depth when processing user-generated hierarchical data — even 'flat' configs can become nested.
🎯 Key Takeaway
Recursive traversal is the natural fit for any tree-like data.
Guard depth for untrusted input; use iterative stacks for deep or critical paths.
The simplest test: 'Will this data ever have children?' If yes, recursion is likely your best tool.
When to Use Recursive Traversal vs Iterative Loop
IfData has a 'children' attribute and you need full depth processing
UseUse recursion — it naturally mirrors the data structure
IfData depth exceeds Python's recursion limit (>= 1000)
UseUse iterative stack (list.pop()) to avoid RecursionError
IfData is a flat list or simple sequence
UseUse iteration — simpler, no stack overhead
IfYou need to stop early (pruning) and must pass state up the call chain
UseRecursion can be cumbersome; consider iterative with explicit stack and custom control flow
🗂 Recursion vs Iteration: When to Use Which
A quick reference for choosing the right approach in Python.
AspectRecursionIteration (Loop)
Best fit forTree-shaped / self-similar dataLinear / sequential data
Code readabilityMirrors the problem structure naturallyFamiliar and explicit
Memory usageO(depth) stack frames — can hit limitO(1) extra memory for simple loops
Python stack limitYes — crashes at ~1000 frames by defaultNo limit
Tail-call optimisationNot supported in PythonNot applicable
Performance (naive)Can be exponential without memoisationTypically linear or better
Easy fix for perf issuesAdd @lru_cache in one lineManual optimisation
Error tracingDeep tracebacks, harder to readShorter, simpler tracebacks
Real use casesFile systems, trees, parsers, graphsList processing, counters, pipelines

🎯 Key Takeaways

  • Every recursive function needs a base case that returns without calling itself, and a recursive case that provably moves closer to that base case on every call — skip either and you get RecursionError.
  • Python pushes a new stack frame for every recursive call — the default limit is 1000, which is a deliberate design choice to keep tracebacks readable, not a bug to work around for normal use.
  • Naive recursive Fibonacci is O(2^n) and will time out on inputs above ~40 — a single @functools.lru_cache decorator converts it to O(n) with zero logic changes.
  • Use recursion when the data is shaped like a tree (file systems, nested configs, ASTs, graphs); use iteration for flat sequences — the right choice is about matching the tool to the shape of the problem, not about which feels more clever.
  • Always guard recursion depth for untrusted input — a max_depth parameter and custom exception is safer than raising the global limit.
  • Recursive tree traversal is a production-ready pattern: visit node, process, recurse into children, combine results.

⚠ Common Mistakes to Avoid

    Forgetting the base case entirely
    Symptom

    The function calls itself forever and Python raises RecursionError: maximum recursion depth exceeded. The program crashes with a stack trace showing repeated calls.

    Fix

    Always write the base case first, before any recursive logic, and verify it handles every exit condition (not just the happy path). Use a return statement to exit the recursion.

    Base case exists but recursive call doesn't move toward it
    Symptom

    Same RecursionError, but base case is present. The function deadlocks on the same argument forever. Harder to spot because the base case looks correct.

    Fix

    During debugging, print the argument on each recursive call and confirm the argument shrinks every time. For example, print(f'Recursing with {n}'). Ensure the recursive call passes a modified argument (e.g., n-1 not n).

    Using naive recursion on overlapping-subproblem functions (e.g., Fibonacci)
    Symptom

    fibonacci(40) takes several seconds; fibonacci(50) may never finish. CPU spikes and long response times in production.

    Fix

    Add @functools.lru_cache(maxsize=None) for a one-line memoisation that makes it run in microseconds. For full control, rewrite as a bottom-up dynamic programming loop.

    Not adding a depth guard for untrusted input
    Symptom

    A single deep JSON payload (1001 levels) crashes the API with RecursionError. The error is intermittent and hard to reproduce because depth depends on input data.

    Fix

    Add a depth parameter to the recursive function and raise a custom exception when it exceeds a safe limit (e.g., 200). For untrusted data, default to an iterative approach with an explicit stack.

    Raising `sys.setrecursionlimit()` as a fix for depth issues
    Symptom

    The RecursionError goes away temporarily, but later the application segfaults with a core dump because Python's C stack overflowed. The fix introduces a worse failure.

    Fix

    Never increase the recursion limit without verifying that no infinite recursion exists. Prefer iterative refactoring. If you must raise it, increase incrementally and test on the maximum expected depth with memory monitoring.

Interview Questions on This Topic

  • QWhat's the difference between a base case and a recursive case, and what happens in Python if you omit the base case?JuniorReveal
    The base case is the condition that stops the recursion by returning a result without making another recursive call. The recursive case is where the function calls itself on a smaller or simpler input. If you omit the base case, Python will keep pushing frames onto the call stack until it hits the recursion limit (default 1000) and raises RecursionError. This crashes the program with a stack trace showing repeated calls. The fix is to always define a base case that the recursive path provably reaches.
  • QWhy doesn't Python support tail-call optimisation, and how would you work around the default recursion limit of 1000 for a legitimately deep recursive problem?Mid-levelReveal
    Python deliberately does not implement tail-call optimisation because Guido van Rossum argued it would make tracebacks less readable and complicate debugging. When a function is tail-recursive (the recursive call is the last operation), TCO would reuse the current stack frame, but Python keeps the full stack for error traces. To work around the 1000-frame limit for genuinely deep recursion, you have three options: (1) rewrite the function iteratively with an explicit stack or loop, (2) use sys.setrecursionlimit(desired_limit) but only after verifying no infinite recursion exists and monitoring for C stack overflow, or (3) restructure the problem to avoid deep recursion (e.g., use divide-and-conquer with a branch-and-bound to cap depth).
  • QGiven a naive recursive Fibonacci implementation that runs in O(2^n) time, how would you optimise it — and can you name two different approaches with their tradeoffs?SeniorReveal
    Two common approaches: 1. Memoisation — Add @functools.lru_cache(maxsize=None) to cache results of previous calls. This reduces time complexity to O(n) but still uses O(n) stack space due to recursion depth. It's a one-line fix, but for very large n (e.g., > 1000) you'll hit the recursion limit. 2. Bottom-up dynamic programming — Use an iterative loop that builds Fibonacci from 0 upward, storing only the last two values. This is O(n) time and O(1) space, with no recursion stack risk. The tradeoff is you must manually manage state rather than rely on the recursive system. Bottom-up is more efficient and safer for large n, but loses the elegant self-similar structure of recursion.
  • QExplain how the call stack works when a recursive function runs. What happens to memory as the recursion depth increases?Mid-levelReveal
    Each recursive call pushes a new frame onto Python's call stack. A frame contains the function's local variables, the current line of execution, and the return address. As depth increases, Python allocates more stack memory (approximately 8 KB per frame). When the base case is reached, frames are popped in reverse order, freeing memory. If the depth exceeds Python's limit (default 1000), it raises RecursionError. If the limit is too high, the underlying C stack can overflow, causing a segfault. The stack is a finite resource shared with every other function call in your program.
  • QDescribe a production scenario where recursion was the wrong choice and what you would use instead.SeniorReveal
    A common scenario is processing deeply nested user-generated JSON, such as a configuration file with 2000+ levels of nesting. A recursive parser will hit RecursionError or segfault. The better approach is to use an iterative stack: maintain a list of nodes to process, and pop nodes in a while loop. This avoids Python's call stack entirely and can handle arbitrary depth. Additionally, you can add a max depth check for safety. In general, if the data depth is unbounded or user-controlled, recursion is dangerous — iteration is safer.

Frequently Asked Questions

What is recursion in Python and when should I use it?

Recursion is when a function calls itself to solve a smaller version of the same problem, repeating until it hits a base case it can answer directly. Use it when the problem has a self-similar or tree-shaped structure — like walking a file system, parsing nested data, or tree traversal. For flat, linear problems, a regular loop is faster and simpler.

How do I fix RecursionError: maximum recursion depth exceeded in Python?

This error means your function called itself more than Python's limit (default 1000). There are three fixes: (1) check your base case is reachable and your recursive case genuinely approaches it, (2) add memoisation with @lru_cache if you're recomputing the same inputs, or (3) rewrite using an iterative approach with an explicit stack. Raising the limit with sys.setrecursionlimit() is a last resort — it doesn't solve the underlying problem.

What is a base case in recursion and why is it essential?

A base case is the condition where the recursive function returns an answer directly without calling itself again. It's the stopping point. Without it, the function would call itself forever until Python's stack fills up and throws a RecursionError. Think of it as the floor of a building — every staircase needs somewhere to end.

Can I use recursion for very deep data (more than 1000 levels)?

You can increase the recursion limit with sys.setrecursionlimit(20000), but this is dangerous — Python's C stack can overflow and segfault. A better approach is to refactor your recursion into an iterative solution using an explicit stack (list). If you must use recursion, add a custom depth parameter and raise an exception before hitting the system limit.

What is tail recursion and why doesn't Python optimise it?

Tail recursion is when the recursive call is the very last operation in the function. Languages like Haskell and Scheme implement tail-call optimisation (TCO) to reuse the current stack frame, making recursion as memory-efficient as a loop. Python intentionally does not support TCO because it makes tracebacks less readable — frames are kept so you can see the full call history during debugging.

How does memoisation help recursive functions?

Memoisation caches the results of function calls so that if the same input is encountered again, the cached result is returned without recomputing. For recursive functions with overlapping subproblems (like Fibonacci), this reduces time complexity from exponential to linear. In Python, you can add @functools.lru_cache(maxsize=None) as a decorator to automatically memoise a recursive function.

Where is recursion commonly used in production Python code?

Recursion appears in file system traversals (os.walk is iterative but many custom walkers use recursion), DOM/hTML parsers, JSON serialization/deserialization, AST processing (e.g., Python's ast module), tree data structures (binary search trees, decision trees), and some graph algorithms (DFS, topological sort). Django's template engine uses recursion to resolve template inheritance.

🔥
Naren Founder & Author

Developer and founder of TheCodeForge. I built this site because I was tired of tutorials that explain what to type without explaining why it works. Every article here is written to make concepts actually click.

← PreviousClosures in PythonNext →map filter and reduce in Python
Forged with 🔥 at TheCodeForge.io — Where Developers Are Forged