Python Closures — Late Binding Loop Bug in Rate Limiters
All endpoints returned 429 after one hit its limit—closures captured loop variables by reference.
20+ years shipping production Python across data and backend systems. Drawn from code that ran under real load.
- ✓Solid grasp of fundamentals
- ✓Comfortable reading code examples
- ✓Basic production concepts
- A closure is a nested function that remembers variables from its enclosing scope even after that scope exits
- Three conditions: nested function, references outer variable, outer function returns inner function
- Python stores captured variables in cell objects accessible via
__closure__ - Performance: closure creation is cheap (one cell per captured variable); no class overhead
- Production failure: forgetting
nonlocalwhen mutating causesUnboundLocalError— silent misdirection - Biggest mistake: assuming closures capture values by value in loops — they capture by reference, leading to the loop closure bug
Imagine you work at a coffee shop and your manager gives you a secret discount code before you go out to serve customers. Even after you've left the back office, you still remember that code and can use it whenever you need it. A Python closure is exactly that — a function that 'carries' a piece of its birthplace with it, even after that birthplace no longer exists. The inner function remembers the variables from the outer function that created it, like a note tucked into its pocket.
Most Python developers hit a wall around the same time: they understand functions, they understand variables, and then they write slightly more complex code and something weird happens — a function 'remembers' a value it seemingly shouldn't have access to anymore. That moment of confusion is actually you bumping into one of Python's most powerful and elegant features: closures. They're not an advanced academic concept — they're built into how Python evaluates and stores functions, and once you get them, a whole class of real-world problems becomes easy to solve.
The problem closures solve is surprisingly common: how do you give a function some persistent, private state without creating a whole class? Before closures, the only clean answer was to write a class with an __init__ method and self.some_value sprinkled everywhere — a lot of ceremony for something conceptually simple. Closures let you attach state to a function directly, keeping that state private and scoped exactly to where it's needed. They're also the engine under the hood of Python decorators, which means understanding closures is the key to understanding one of Python's most widely-used patterns.
By the end of this article you'll know exactly what makes a closure a closure (it's a specific three-part contract Python enforces), you'll be able to write closures that solve real problems like function factories and stateful callbacks, and you'll know the two gotchas that trip up almost every developer the first time they use closures in a loop. Let's build it up from scratch.
What Python Actually Needs to Form a Closure
A closure isn't magic — it's the result of three specific conditions being true at the same time. First, there must be a nested function (a function defined inside another function). Second, the inner function must reference at least one variable from the outer function's scope — not a global variable, not one of its own local variables, but one that belongs to the enclosing function. Third, the outer function must return the inner function. When all three are true, Python doesn't just return a plain function object. It returns a function object bundled together with a snapshot of the variables it referenced from outside. That bundle is the closure.
You can actually inspect this bundle yourself. Every closure in Python has a __closure__ attribute, which is a tuple of 'cell' objects. Each cell holds one of those remembered variables. If a function has no closure, __closure__ is None. This isn't just trivia — it's confirmation that Python is doing real work to preserve that state for you.
The variable that gets captured is called a 'free variable'. It's free because it isn't local to the inner function and isn't global — it floats between those two worlds, owned by the enclosing scope. Understanding this scoping layer (called the Enclosing scope in Python's LEGB rule) is the key to predicting how closures behave.
__closure__ and co_freevars — they are always available at runtime.__closure__ is not None to confirm.__closure__ is your proof — use it to verify closure behaviour.LEGB Scope Resolution — Step-by-Step Visual
Python resolves variable names using the LEGB rule: Local, Enclosing, Global, Built-in. Closures rely on the Enclosing (E) scope. When Python executes a nested function, it first looks for a variable in the function's local scope (L). If not found, it inspects the enclosing function's local scope (E) — that's where free variables live. Next comes the global module scope (G), then the built-in namespace (B) for functions like len and print. This chain is evaluated at runtime, which is why closures can see updated values of variables until they are called.
Understanding LEGB is crucial for debugging closures. If a variable unexpectedly resolves to a global instead of an enclosing scope, Python skipped the E because either the variable wasn't captured (not referenced in the inner function) or the inner function wasn't truly a closure (not returned). The diagram below walks through a typical scope chain for a closure.
__code__.co_freevars to confirm which variables are captured. If a variable is not in co_freevars, it's not part of the closure.co_freevars to verify which variables are captured.The 'nonlocal' Keyword — Mutating Enclosed Variables
By default, Python treats any assignment inside a nested function as creating a new local variable. To modify a variable from an enclosing (non-global) scope, you must declare it nonlocal. This keyword tells Python: 'bind this name to the variable in the nearest enclosing scope that is not global.' Without nonlocal, writing count += 1 inside a nested function raises UnboundLocalError because Python sees the assignment and considers count local, but then can't find an initial value before the increment.
nonlocal is commonly used in closures that maintain mutable state: counters, accumulators, caches, and retry trackers. It is different from global: nonlocal climbs up only through function scopes, while global bypasses all function scopes and goes straight to the module level.
A key nuance: nonlocal is only valid at the beginning of a function (before any code that uses the variable), and the variable must already exist in an enclosing function scope. If the variable is not defined in any enclosing function, Python raises a SyntaxError at compile time. This is a safety net — it prevents accidental creation of a new global or outer scope variable.
Below is an example that demonstrates correct nonlocal usage for a closure that accumulates a sum.
nonlocal and try to assign to a variable, Python creates a new local variable instead of modifying the outer one. This can lead to hard-to-find bugs where the outer variable never changes. Always add nonlocal as the first line of the inner function when you intend to mutate an enclosing variable.nonlocal sparingly — it makes state harder to track. For production code with multiple mutations, prefer a class with explicit methods. nonlocal shines when you need a single, lightweight stateful callable and a class would be overkill.nonlocal is required to mutate variables from the enclosing scope. It prevents UnboundLocalError and ensures the outer variable is changed.Closures With Mutable State — Building a Counter Without a Class
So far the closure variables have been read-only inside the inner function. But what if you need the inner function to update that variable — to keep a running count, for example? This is where Python's nonlocal keyword comes in. Without it, any assignment inside the inner function creates a brand-new local variable and shadows the outer one rather than updating it. Python treats assignment as declaration by default.
nonlocal is your explicit instruction to Python: 'when I assign to this name, go up to the enclosing scope and update the variable there, don't create a new local one.' It's the closure equivalent of the global keyword, but scoped to the enclosing function rather than the module level. You should reach for nonlocal only when you genuinely need mutable state in a closure — if you're using it everywhere, a class is probably the cleaner choice.
The stateful counter pattern is a textbook example, but the real-world version is more interesting: rate limiters, request counters, retry trackers, and progress accumulators all use this exact shape. The closure gives each counter its own private state, completely isolated from any other counter you create from the same factory.
nonlocal and write call_count += 1 directly, Python raises UnboundLocalError: local variable 'call_count' referenced before assignment. Python sees the assignment and marks call_count as local to the inner function — then can't find a value for it before the +=. The fix is always just one word: nonlocal.nonlocal for complex state machines; it becomes hard to debug.nonlocal is mandatory for assignments to outer variables.nonlocal.The Classic Loop-Closure Gotcha (And Two Ways to Fix It)
Here's the trap that catches almost every developer writing closures inside a loop for the first time. You want to create a list of functions, each capturing a different loop variable. You write what looks like perfectly reasonable code, run it, and every single function in the list returns the same value — the last value the loop variable had. This isn't a bug in Python. It's the correct behaviour once you understand what closures actually capture.
A closure captures a variable by reference, not by value. That means all the inner functions point to the same index variable in the enclosing scope. By the time you call any of those functions, the loop has finished and index has settled at its final value. Every function looks up index right then, at call time, and sees the same thing.
There are two clean fixes. The first is to use a default argument in the inner function (def greet(name=name):), which forces the current value to be captured by value at definition time. The second is to wrap the inner function in another function call that immediately captures the current value in its own closure scope. Both work — the default argument approach is slightly more Pythonic for simple cases.
Closures and Decorators: The Hidden Relationship
Python decorators are the most widespread real-world use of closures. Every time you write @cache or @staticmethod, you're relying on the fact that a function can wrap another function and carry state. The decorator pattern is a closure: the outer function (the decorator) accepts a function, and the inner function (the wrapper) captures that function and adds behaviour before/after calling it.
Understanding this relationship demystifies decorators. When you see @decorator, Python calls decorator(func) and stores the returned closure in place of func. That returned closure references the original function and any configuration passed to the decorator factory. This means you can create parameterised decorators (like @retry(max_attempts=3)) by nesting three levels: outer factory returns a decorator, which returns a wrapper closure.
Be careful with decorator stacking: each decorator adds one layer of closure indirection. Too many layers can hurt readability and performance. Also, functools.wraps is essential to preserve the original function's metadata — without it, the wrapped closure will have the wrong __name__ and __doc__, breaking introspection tools and documentation generators.
@timer_decorator as saying: replace slow_add with the closure that timer_decorator(slow_add) returns.- The outer function receives the decorated function as an argument.
- The inner wrapper function captures that function as a free variable.
functools.wrapsis critical to preserve the original function's identity.- Parameterised decorators require an additional outer factory level.
- Each decorator adds one closure cell — stay mindful of nesting depth.
functools.wraps, your decorated function loses its original name and docstring — breaking automated docs, debugging, and serialisation.@functools.wraps in decorator wrappers; it's a one-line insurance policy.functools.wraps preserves the original function's metadata.Closure vs Class: Memory and Syntax Comparison
When building stateful callables, developers often wonder whether to use a closure or a class. Both can hold state and expose behaviour, but they differ in syntax overhead, memory usage, and capabilities. This section breaks down the practical trade‑offs so you can make an informed decision.
Memory – A closure stores captured variables in cell objects (one cell per free variable). Each cell is approximately 56 bytes on 64‑bit Python, plus the captured object itself. A closure function object adds around 56 bytes of overhead. In contrast, a class instance has an overhead of about 64 bytes plus the instance dictionary (which can be substantial if many attributes are added dynamically). For a single stateful callable with one or two variables, closures are more memory‑efficient.
Syntax – Closures are written as nested functions; the state is implicitly captured. Classes require a class definition, an __init__ method, and explicit self references. For simple cases, closures are far less verbose. However, if you need multiple methods that share state, a closure becomes awkward (you must return a tuple of functions), while a class naturally supports any number of methods.
Introspection – Class attributes are directly accessible via instance.attribute. Closure state is hidden behind __closure__ cells, which makes debugging harder. If you need to inspect or log internal state regularly, a class is clearer.
Inheritance – Closures cannot be subclassed. Classes fully support inheritance, which is essential for polymorphic dispatch.
Pickling – Closures can be pickled only if the outer function is importable and all captured variables are picklable. Class instances are generally easier to pickle.
Below is a comparison table summarising the key differences:
Why You Should Care About Variable Capture (And When It Bites You)
Closures exist because Python needs to carry around state without allocating a class. The inner function "captures" variables from the enclosing scope — not copies, not snapshots, but live references. That's the power and the pain.
When you return a nested function, it brings along a cell object for each free variable. Inspect it with __closure__ and you'll see tuples of cells. Each cell holds a pointer to the current value. If that value changes before the closure executes, you get the loop-gotcha (already covered), but there's a subtler trap: capturing mutable objects like lists or dicts. You think you're saving a state snapshot; you're actually saving a leash to a shared kennel.
Watch for this in async callbacks. A closure capturing request from a loop iteration will see the last value if the callback fires after the loop finishes. The fix: default argument binding or a factory function that eagerly evaluates. Know your capture semantics, or your production bug will be a silent, data-corrupting ghost.
.copy() to break the reference chain.Factory Functions — Why You Build Functions That Build Functions
Factory functions are the classic closure use case. You've got a function that takes a parameter and returns a new function hard-wired with that parameter. No class, no __init__, no boilerplate. Just a closure that remembers what you gave it.
Example: an API rate limiter. You need different rate limits per endpoint. Write one factory, get back throttled callers that each carry their own wait time. No global state, no inheritance tree, no nonsense. Just a cell with a float.
Here's the production angle: factory functions let you parameterize behavior at definition time, not call time. That means you can precompute static data, validate arguments once, and fail fast during module import instead of silently at 3 AM. If the factory raises, you know immediately that your configuration is broken. That's worth more than a pretty class hierarchy.
When to avoid? If you need multiple methods or serialization, use a class. But for a single-method state-holder, a factory closure is less code, less cognitive load, and easier to test in isolation.
Memoization With Closures — Caching Without a Class
Memoization is the poster child for closures. You wrap a pure function with a cache dictionary that lives in the closure's scope. The first call computes and stores; subsequent calls hit the cache. No global dict, no class instance, no decorator syntax needed if you want to keep it explicit.
The closure captures the cache dict. Every invocation of the inner function sees the same dict cell. That's the entire trick — a mutable container that persists across calls. Python's functools.lru_cache does this with fancier features (like max size and thread safety), but under the hood it's exactly this pattern: a dict in a closure.
Why not just use a class? Because a closure-based memoizer has zero public API surface. No one can accidentally clear the cache from outside. No one can inspect internals. That's encapsulation without the ceremony of private attributes. If the function is pure and the computation is expensive, this pattern saves you from writing a class that's nothing but __init__, __call__, and one private dict.
Performance note: the dict lookup is O(1) average. But the closure itself adds a tiny overhead per call — around 50-100ns in CPython. For memoization, that's noise compared to any real computation.
Stop Using Closures As Crappy Classes
You already have classes. Use them. Closures capture state by accident, not by design. Every time you find yourself writing a closure that holds five variables and returns three different inner functions, you've built a class with extra steps and zero readability.
The production rule: if your closure has more than one captured variable, or if it returns more than one function, refactor to a class. Closures shine for single-responsibility factories — one input, one output, zero surprises. Classes are explicit. Closures are sneaky. Don't make your team debug captured state at 3 AM because someone forgot which scope a variable lives in.
Your senior engineer will thank you. Your future self will curse you for the alternative.
Partial Application: The Closure You Didn't Know You Had
functools.partial is a closure without the nonsense. You get the same variable capture, but it's readable, tested, and doesn't require you to manually nest functions. It's the closure pattern that landed in the standard library because people kept writing buggy versions of it.
The WHY: partial binds arguments without executing the function. That's closure behavior — capturing values — without the scope resolution headaches or nonlocal keyword. Your team reads it instantly. No nested defs. No wondering which frame a variable belongs to.
Real-world win: database query factories. You bind a connection once, then call the partial repeatedly. Performance identical to a hand-rolled closure. Maintenance cost: zero. Don't write what the standard library already gave you.
Closures in Decorators: Capturing State
Decorators are a powerful feature in Python that allow you to modify or enhance functions without changing their definition. Under the hood, decorators often rely on closures to capture state from the enclosing scope. When you define a decorator that takes arguments or maintains internal state, closures enable the inner wrapper function to access variables from the outer decorator function even after the decorator has returned.
Consider a simple retry decorator that retries a function a specified number of times on failure. The decorator function retry takes max_attempts as an argument and returns a wrapper that captures this variable via closure. The wrapper then calls the original function and retries on exceptions, decrementing a counter each time. The closure ensures that max_attempts is available to the wrapper each time the decorated function is called.
```python import functools
def retry(max_attempts): def decorator(func): @functools.wraps(func) def wrapper(args, kwargs): attempts = 0 while attempts < max_attempts: try: return func(args, **kwargs) except Exception as e: attempts += 1 if attempts == max_attempts: raise return None return wrapper return decorator
@retry(max_attempts=3) def unstable_network_call(): import random if random.random() < 0.7: raise ConnectionError("Network error") return "Success"
print(unstable_network_call()) ```
In this example, max_attempts is captured by the closure wrapper. Each time unstable_network_call is invoked, the wrapper accesses the captured max_attempts to control the retry loop. This pattern is common in real-world decorators like caching, rate limiting, and logging, where state must be preserved across multiple calls.
Key point: The closure captures the variable max_attempts from the enclosing scope of the retry function. Even after retry returns, the wrapper retains access to max_attempts. This is the essence of closures in decorators: they allow decorators to carry configuration or state without using global variables or classes.
nonlocal Statement: Modifying Captured Variables
In Python, closures can capture variables from the enclosing scope, but by default, you cannot reassign them. The nonlocal statement allows you to modify variables that are in an outer (but not global) scope. Without nonlocal, assigning to a variable inside a nested function creates a new local variable, shadowing the outer one. This is crucial when you need to update state across multiple calls to the inner function.
Consider a closure-based counter that increments a count each time it's called. Without nonlocal, the inner function would create a new local count variable, and the counter would never increment. With nonlocal, the inner function can modify the outer count variable, preserving state across calls.
```python def make_counter(): count = 0 def counter(): nonlocal count count += 1 return count return counter
counter_a = make_counter() print(counter_a()) # 1 print(counter_a()) # 2
counter_b = make_counter() print(counter_b()) # 1 (independent state) ```
In this example, nonlocal count tells Python that count is not local to counter but is defined in the enclosing scope of make_counter. Each call to counter_a modifies the same count variable, so the counter increments. Each call to make_counter creates a new closure with its own count, so counter_b starts at 1.
Without nonlocal, the code would raise an UnboundLocalError or silently create a local variable, breaking the closure's intended behavior. The nonlocal statement is essential for closures that need to maintain mutable state, such as accumulators, caches, or state machines.
Note: nonlocal works only for variables in enclosing function scopes, not global or module-level variables. For global variables, use the global statement.
nonlocal is commonly used in decorators that maintain state (e.g., caching decorators) and in callback functions that need to accumulate results. Be mindful of variable naming to avoid shadowing and ensure readability.nonlocal statement is required to modify captured variables in a closure; without it, assignments create new local variables instead of updating the outer scope.Closures vs Classes: When to Use Each
Both closures and classes can encapsulate state and behavior, but they have different strengths. Closures are lightweight, functional-style constructs that capture variables from an enclosing scope. Classes are more feature-rich, supporting inheritance, methods, and multiple instances with shared behavior. Choosing between them depends on the complexity of the state and the need for extensibility.
Use closures when: - You need a simple callable with minimal state (e.g., a counter, a multiplier). - You want to avoid defining a full class for a single function. - You are working in a functional programming style or need to pass stateful functions as arguments. - The state is small and unlikely to grow.
Use classes when: - You need multiple methods that operate on the same state. - You want to define a type with its own identity (e.g., for isinstance checks). - You need inheritance or polymorphism. - The state is complex or involves many attributes.
Consider a simple multiplier: a closure can capture a factor and return a function that multiplies its argument by that factor. A class would require defining a __call__ method. The closure is more concise.
```python # Closure approach def make_multiplier(factor): def multiply(x): return x * factor return multiply
times2 = make_multiplier(2) print(times2(5)) # 10
# Class approach class Multiplier: def __init__(self, factor): self.factor = factor def __call__(self, x): return x * self.factor
times2_class = Multiplier(2) print(times2_class(5)) # 10 ```
For a counter with multiple operations (increment, reset, get value), a class is more appropriate because it groups related methods. A closure would require returning multiple functions or a dictionary, which is less elegant.
In summary, closures are best for simple, single-method stateful functions, while classes excel when you need a full-fledged object with multiple behaviors and potential for extension.
sorted() or map().Rate Limiter with Closures: When Late Binding Breaks Production
- Always force value capture when creating closures in a loop: use a default argument or a factory helper.
- Test closure behaviour with a small loop before rolling out to production.
- Add a quick assertion to verify that each returned function has different
__closure__cell contents.
func.__closure__[0].cell_contents on each generated function to confirm different values.UnboundLocalError: local variable 'x' referenced before assignment inside a nested functionnonlocal x at the top of the inner function. Python treats assignments as local by default; nonlocal explicitly lifts the assignment to the enclosing scope.func.__closure__ for large objects. If a closure captures a large data structure, it stays alive as long as the function exists. Consider using weak references or making a copy to avoid accidental retention.args, *kwargs and passes them through. The closure captures the original function, but the wrapper signature must match the expected call pattern.print([f.__closure__[0].cell_contents for f in list_of_funcs])Check if all cell_contents are identical.def f(val=loop_var): or wrap in a helper function.| File | Command / Code | Purpose |
|---|---|---|
| closure_anatomy.py | def make_multiplier(factor): # Outer function defines 'factor' | What Python Actually Needs to Form a Closure |
| nonlocal_example.py | def make_accumulator(initial=0): | The 'nonlocal' Keyword |
| api_request_counter.py | def make_request_counter(endpoint_name): | Closures With Mutable State |
| loop_closure_fix.py | user_names = ["alice", "bob", "carol"] | The Classic Loop-Closure Gotcha (And Two Ways to Fix It) |
| closure_decorator_example.py | def timer_decorator(func): | Closures and Decorators |
| closure_vs_class.py | def make_counter_closure(initial=0): | Closure vs Class |
| CaptureTrap.py | callbacks = [] | Why You Should Care About Variable Capture (And When It Bite |
| RateLimiterFactory.py | def rate_limiter(max_calls, period_seconds): | Factory Functions |
| ClosureMemoizer.py | def memoize(func): | Memoization With Closures |
| ClassVsClosure.py | class CounterClosure: | Stop Using Closures As Crappy Classes |
| PartialVsClosure.py | from functools import partial | Partial Application |
| retry_decorator.py | def retry(max_attempts): | Closures in Decorators |
| nonlocal_counter.py | def make_counter(): | nonlocal Statement |
| closure_vs_class.py | def make_multiplier(factor): | Closures vs Classes |
Key takeaways
__closure__, not just in a stack frame that disappears.nonlocal explicitly whenever an inner function needs to assign to (not just read) a variable from the enclosing scopeUnboundLocalError.functools.wraps is essential to preserve function metadata.__closure__ to debug closure issues in productionInterview Questions on This Topic
Can you explain what a closure is in Python and describe the three conditions required to form one? Then walk me through what `__closure__` and `__code__.co_freevars` actually contain.
__closure__ is a tuple of cell objects, each cell holding the runtime value of one captured free variable. __code__.co_freevars is a tuple of strings with the names of those free variables, determined at compile time.Frequently Asked Questions
20+ years shipping production Python across data and backend systems. Drawn from code that ran under real load.
That's Functions. Mark it forged?
13 min read · try the examples if you haven't