StopIteration in Python: Iterators Explained

Your loop processes 9,800 rows then dies with a bare StopIteration and no message. Or worse — it processes 9,800 of 10,000 rows, exits cleanly, and nobody notices 200 rows vanished. Both are the same signal handled two ways: an iterator ran dry, and the code around it either caught the signal correctly or let a stray one masquerade as the end of data.

StopIteration is Python's iterator exhaustion protocol. Every for-loop depends on it: the loop calls next() repeatedly and stops when StopIteration arrives. That machinery is invisible until you call next() manually, write a generator, or raise StopIteration yourself inside generator code — three situations where the signal escapes its normal channel and becomes either a crash or silent truncation.

The sharpest edge is PEP 479. Since Python 3.7, any StopIteration that bubbles out of a generator frame is converted to RuntimeError: generator raised StopIteration. Code that worked for years by letting a helper's StopIteration end a generator now crashes with a different error pointing at a different line. You'll learn the exhaustion protocol cold, the next()-with-default habit, the PEP 479 fix, and how to spot loops that swallow real bugs as fake end-of-data.

The Exhaustion Protocol: next(), __next__, and the Signal

Iteration in Python is a two-method contract. An iterable defines __iter__ to produce an iterator; the iterator defines __next__ to return one item per call and raise StopIteration when empty. Built-in next(it) just calls it.__next__(), and for-loops wrap that call in a try that catches StopIteration and exits. The whole 10,000-row loop you write as two lines is secretly thousands of next() calls plus one caught exception.

Calling next() yourself skips the safety net. There's no loop to catch the signal, so the first call past the end raises StopIteration into your code like any other exception. That's correct behavior — the iterator is telling the truth — but it surprises developers who think of next() as returning items forever. It doesn't; it returns items until it can't, then raises.

Internalize this: StopIteration is not an error in the protocol sense. It's the protocol's 'done' message. It becomes an error only when it arrives somewhere unprepared — a manual call with no try, a generator frame under PEP 479, or a loop that mistakes a helper's crash for the data's end. Every fix in this article is about delivering that message to code ready to hear it.

Sketch the call chain on paper when one escapes: which iterator raised, which frame caught or missed it, and which default would have kept the message in its proper channel.

it = iter([10, 20])
print(next(it))  # 10
print(next(it))  # 20
try:
    next(it)  # exhausted: raises StopIteration
    print("never reached")
except StopIteration:
    print("exhausted: StopIteration raised as designed")
print("for-loop swallows it:", [x for x in iter([10, 20])])

next() With a Default: The One-Habit Fix for Manual Calls

The two-argument form next(iterator, default) converts exhaustion from an exception into a return value. When items remain, you get the next item exactly as before. When the iterator is empty, you get your default instead of a traceback. One extra argument removes the entire crash class for manual calls — polling loops, token parsers, lookahead readers, and merge joins all get simpler.

The default should be something your code can branch on. None works when None never appears as real data. When None is legitimate data — nullable columns, optional tokens — use a module-level sentinel like _MISSING = object() and test with is. Identity comparison against a unique object can't collide with data, so data Nones flow through while true exhaustion takes the sentinel branch.

Apply it everywhere a helper consumes a sub-iterator: token streams, chunk readers, multi-line record assemblers. The refund bug was a single bare next(token_iter) in a helper that met a short memo. A default plus an if-branch would have ended the record instead of the export. Grep your codebase for bare next( calls today — each is a future truncation or page waiting for the right short input.

Log short-record branches with their input offsets so the first truncated memo shows up as a counted event instead of a silent row-count gap.

_MISSING = object()

def parse_tokens(tokens):
    it = iter(tokens)
    out = []
    while True:
        tok = next(it, _MISSING)
        if tok is _MISSING:
            return out  # clean end, no exception
        out.append(tok.upper())

print(parse_tokens(["a", "b"]))
print(parse_tokens([]))
it = iter([None, None])
print(next(it, _MISSING) is _MISSING, next(it, _MISSING) is _MISSING)
print(next(it, _MISSING) is _MISSING)  # only this is True: real Nones pass through

PEP 479: When StopIteration Inside a Generator Becomes RuntimeError

Before Python 3.7, a StopIteration bubbling out of a generator frame silently ended the generator — which meant helper bugs disguised themselves as finished data. PEP 479 closed that hole: now any StopIteration that would escape a generator frame is converted to RuntimeError: generator raised StopIteration, with the original as context. The generator crashes loudly instead of truncating quietly.

In practice this means two things. First, if you see RuntimeError naming a generator, look inside that generator for a bare next() on another iterator or an explicit raise StopIteration — that's the true culprit wearing a new exception type. Second, the fix is never to catch RuntimeError around the generator; it's to stop the StopIteration at its source with a default, or replace raise StopIteration with a plain return, which is the correct way to end a generator.

Migration-era code still carries the old idiom: helpers that raise StopIteration to signal 'done' to a calling generator. Rewrite those helpers to return a sentinel or raise a domain exception like ValueError, and rewrite generators to return instead of raising. The 6-day silent truncation predated full PEP 479 coverage; on modern Python the same bug pages you at 2 a.m. instead — louder, but at least visible.

def gen_with_stray():
    it = iter([])
    yield next(it)  # StopIteration escapes generator frame -> RuntimeError

try:
    list(gen_with_stray())
except RuntimeError as exc:
    print("PEP 479 converted it to", type(exc).__name__ + ":", exc)

def gen_fixed():
    it = iter([])
    first = next(it, None)
    if first is None:
        return  # correct way to end a generator: plain return
    yield first

print("fixed generator yields:", list(gen_fixed()))

For-Loops Swallow StopIteration: The Silent Truncation Trap

A for-loop stops on the first StopIteration it sees, with no way to ask who sent it. Normally the sender is the loop's own iterator announcing clean end. But any StopIteration raised by code inside the loop body — a helper's bare next(), a property that iterates internally, a nested generator's leak — arrives through the same channel and ends the loop with identical silence. Your 10,000-row export becomes a 9,800-row export with exit code 0.

Defend on two fronts. First, eliminate stray sources: no bare next() inside loop bodies or generator helpers, only defaulted calls with explicit branches. Second, verify completeness outside the loop: compare emitted counts against the source manifest, assert checksums, and page on gaps larger than a small tolerance like 5 rows. The monitoring check 'file exists and non-empty' is what let 9,800 rows pass as healthy for 6 days.

When counts already differ, bisect fast: run the generator to a list and print len() versus the manifest, then binary-search the input for the short record. The culprit is almost always a record shape your test data never included — multi-line memos, empty chunks, trailing delimiters. Add that exact shape to your fixtures so the regression test fails before the next format change ships.

def records(raw_lines):
    it = iter(raw_lines)
    while True:
        line = next(it, None)
        if line is None:
            return
        if line == "MULTI":  # short memo: needs a continuation line
            cont = next(it, None)
            if cont is None:
                yield line + ":<truncated-memo>"  # don't end the export
            else:
                yield line + "+" + cont
        else:
            yield line

src = ["r1", "MULTI", "memo", "r2", "MULTI"]  # trailing short memo
out = list(records(src))
print(len(out), out)
assert len(out) == 4, out  # completeness gate: source had 5 lines, 4 records

Iterators Are Single-Use: Exhaustion Looks Like Missing Data

Iterators are consumed as you read them — once exhausted, they stay exhausted. Code that passes one iterator to two consumers, or loops over the same generator twice, gets full data the first time and zero rows the second, with no exception to explain the emptiness. The second loop just sees immediate StopIteration and exits, looking exactly like an empty source.

Symptoms include a validation pass that always reports zero rows after the load pass, or a debug print(list(it)) that 'fixes' nothing but empties the iterator before the real loop. The cure is to materialize shared data with list() once when it fits in memory, or to re-create the iterator per pass with a factory function like make_rows() instead of sharing one object. When memory is tight, restructure to a single pass that computes both results together.

Teach your team the one-line diagnostic: print(sum(1 for _ in it)) consumes, so never probe an iterator you still need. If a function receives an iterator parameter, document whether it consumes it — 'consumes the iterator' in the docstring prevents the next caller from reusing the husk.

When a pipeline needs two passes over data too large to materialize, split it into two generator factories fed from the same re-readable source instead of sharing one live iterator.

def make_rows():
    return iter(["r1", "r2", "r3"])

shared = make_rows()
first = list(shared)
second = list(shared)  # husk: already exhausted
print("first:", first, "second:", second)
print("factory gives fresh:", list(make_rows()), list(make_rows()))
rows = list(make_rows())  # materialize once, reuse freely
print("materialized:", len(rows), "rows twice:", rows, rows)

Custom Iterators: Raise StopIteration Exactly Once, at the End

When you write __next__ yourself, you own the signal. The contract is narrow: return the next item while items exist, and raise StopIteration exactly when they run out — never to signal errors, skips, or bad records. Raising it early truncates every consumer; raising it for corrupt input teaches loops to treat corruption as completion.

Keep three rules. One: validate inputs in __init__ or the factory, not in __next__, so configuration errors raise ValueError before iteration starts. Two: handle bad records inside __next__ by skipping to the next good item or yielding a quarantined record — reserve StopIteration for true end. Three: make repeated calls after exhaustion keep raising StopIteration rather than restarting or erroring differently, so consumers like list() and for-loops behave uniformly.

Test the edges explicitly: empty source yields zero items then StopIteration, single-item source yields once then stops, and a source with a corrupt middle record still delivers its good neighbors. Those three tests catch every custom-iterator truncation before it meets 10,000 production rows.

Add a fourth test that keeps calling __next__ past exhaustion to lock in the repeat-StopIteration contract your consumers rely on.

class Batches:
    def __init__(self, rows, size):
        if size <= 0:
            raise ValueError("size must be positive")
        self._rows = list(rows)
        self._size = size
        self._i = 0
    def __iter__(self):
        return self
    def __next__(self):
        if self._i >= len(self._rows):
            raise StopIteration  # true end only
        chunk = self._rows[self._i:self._i + self._size]
        self._i += self._size
        return chunk

print(list(Batches(range(5), 2)))
print(list(Batches([], 2)))
try:
    Batches([1], 0)
except ValueError as exc:
    print("config error, not StopIteration:", exc)