Python List Comprehensions — The 500K Email OOM Crash

Every Python program that works with data — scraping websites, processing CSVs, filtering API responses — spends a huge amount of time building new lists from old ones. How you do that has a direct impact on how readable your code is and, to a meaningful degree, how fast it runs. List comprehensions are Python's built-in answer to this everyday problem, and they're one of the first things experienced Python developers reach for when they see a for-loop that builds a list.

Before list comprehensions existed, building a filtered or transformed list meant writing a loop, declaring an empty list, and calling .append() on every iteration — four to six lines of boilerplate just to express a single idea. That ceremony buries the actual intent of the code under a pile of scaffolding. List comprehensions collapse all of that into one expression that reads almost like a sentence, making your intent immediately obvious to the next developer — or to yourself six months later.

By the end of this article you'll know exactly how list comprehensions work under the hood, when they're the right tool and when they're not, how to layer in filtering and nesting without creating unreadable one-liners, and the two or three mistakes that trip up almost every developer the first time they use them in a real project. You'll also walk away with concrete answers to the interview questions that come up every time this topic surfaces.

The Anatomy of a List Comprehension — Reading It Like a Sentence

A list comprehension has three parts, and the order they sit in the expression mirrors the order you'd describe them out loud. The structure is: [expression for item in iterable if condition]. Read it left to right and it says: 'Give me expression, for each item in iterable, but only if condition is true.' The if clause is completely optional — leave it out and every item gets transformed.

The reason the expression comes first — before the for — is that it puts the most important thing front and centre. You're telling the reader immediately what each element of the new list will look like. The loop mechanics and the filter are supporting details that follow.

Under the hood, Python compiles a list comprehension into bytecode that's slightly faster than an equivalent for loop with .append(). That's because .append() has to look up the method on the list object every single iteration, whereas the comprehension's internal C-level loop skips that lookup. For small lists the difference is negligible, but at tens of thousands of items it starts to matter.

One crucial mental model: a list comprehension always produces a brand-new list. It never mutates the original. If you're iterating over temperatures and writing [t * 1.8 + 32 for t in temperatures], your original temperatures list is completely untouched.

# ── Basic transformation: convert Celsius readings to Fahrenheit ──
celsius_readings = [0, 20, 37, 100]

# Traditional loop approach — 4 lines to say one thing
fahrenheit_loop = []
for temp in celsius_readings:
    fahrenheit_loop.append(temp * 1.8 + 32)

# List comprehension — same result, one line, reads like English
# 'Give me (temp * 1.8 + 32) for each temp in celsius_readings'
fahrenheit_comp = [temp * 1.8 + 32 for temp in celsius_readings]

print("Loop result:  ", fahrenheit_loop)
print("Comprehension:", fahrenheit_comp)
print("Same output?  ", fahrenheit_loop == fahrenheit_comp)

# ── Adding a filter: only convert temps above freezing ──
above_freezing_f = [temp * 1.8 + 32 for temp in celsius_readings if temp > 0]
print("Above freezing:", above_freezing_f)

Filtering with Conditions — The if Clause That Changes Everything

The if clause at the end of a comprehension is a gate: only items that pass the test make it into the output list. This is where list comprehensions really start to earn their keep in real-world code, because filtering and transforming at the same time is something you do constantly — think 'get me all active users and format their names' or 'find all log lines that contain an error and strip the timestamp'.

There's an important distinction to keep straight: the if at the end of the comprehension (after the for) is a filter — it controls which items are included. A conditional expression inside the output expression (using value_if_true if condition else value_if_false) is a transformation — it changes what an item becomes. You can use both in the same comprehension, and knowing which is which prevents a lot of confusing bugs.

For example, [score if score >= 50 else 0 for score in exam_scores] transforms every failing score to zero but keeps passing scores as-is. Compare that to [score for score in exam_scores if score >= 50] which simply drops failing scores entirely. The first changes items; the second removes them. These are fundamentally different operations, and mixing them up produces wrong results silently — Python won't complain either way.

exam_scores = [72, 45, 88, 31, 95, 50, 60, 29]

# ── Filter only: remove failing scores (below 50) from the list ──
passing_scores = [score for score in exam_scores if score >= 50]
print("Passing scores:", passing_scores)

# ── Transform only: convert to letter grades using inline conditional ──
# The ternary expression is the OUTPUT, not a filter — every item survives
letter_grades = [
    "A" if score >= 90
    else "B" if score >= 75
    else "C" if score >= 60
    else "D" if score >= 50
    else "F"
    for score in exam_scores
]
print("Letter grades:", letter_grades)

# ── Combined: filter AND transform in one expression ──
# Only passing scores, and map them to 'Pass: <score>'
passing_labelled = [
    f"Pass: {score}"          # transformation applied to survivors
    for score in exam_scores
    if score >= 50             # only scores that pass this gate get transformed
]
print("Labelled passes:", passing_labelled)

# ── Real-world pattern: clean a list of raw strings from user input ──
raw_tags = [" python ", "  ", "Django", "", " REST api "]
cleaned_tags = [
    tag.strip().lower()        # strip whitespace and normalise case
    for tag in raw_tags
    if tag.strip()             # filter out blank or whitespace-only strings
]
print("Cleaned tags:", cleaned_tags)

Nested Comprehensions and Real-World Data — When One Loop Isn't Enough

Sometimes your data isn't a flat list — it's a list of lists. Think a spreadsheet (rows of rows), a game board, or a JSON response that returns a list of orders, each containing a list of items. Nested list comprehensions let you flatten or transform these structures without resorting to nested loops that take up half a screen.

The mental model for reading nested comprehensions is the same 'right to left' trick, but applied twice. In [cell for row in grid for cell in row], you read it as: 'for each row in grid, for each cell in that row, give me cell.' The outermost loop always comes first after the expression.

That said, nesting deeper than two levels is almost always a code smell. If you find yourself writing three for clauses inside one comprehension, stop and ask whether a helper function or a regular loop would be clearer. Readability is the whole point — a comprehension that requires five minutes to decode has failed at its one job.

A genuinely common real-world use case is flattening API response data: an endpoint returns paginated results as a list of pages, each page containing a list of records, and you need one flat list to work with. A two-level comprehension handles this in one expressive line.

# ── Flattening a matrix (list of lists) ──
game_board = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]

# Read: 'give me cell, for each row in game_board, for each cell in that row'
all_cells = [cell for row in game_board for cell in row]
print("Flattened board:", all_cells)

# ── Real-world: flatten paginated API results ──
# Simulate three pages of user records returned by an API
paginated_users = [
    [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}],
    [{"id": 3, "name": "Carol"}],
    [{"id": 4, "name": "Dave"}, {"id": 5, "name": "Eve"}],
]

# Flatten all pages into a single list and extract just the names
all_user_names = [
    user["name"]              # extract the 'name' field from each user dict
    for page in paginated_users   # outer loop: iterate over pages
    for user in page              # inner loop: iterate over users on each page
]
print("All user names:", all_user_names)

# ── Generating a multiplication table as a 2D list ──
# This builds a list of lists — comprehension produces a list,
# and the expression is itself a comprehension
multiplication_table = [
    [row * col for col in range(1, 6)]   # inner comprehension: one row
    for row in range(1, 6)               # outer comprehension: iterate over rows
]

for row in multiplication_table:
    print(row)

When NOT to Use a List Comprehension — Knowing the Limits

List comprehensions have a superpower, but like all superpowers, using them in the wrong situation creates problems. The most important rule is this: if the comprehension doesn't fit on two lines and still read clearly, it's time to switch to a regular loop.

The other big consideration is memory. A list comprehension always builds the entire list in memory immediately. If you're working with a million records and only need to consume them one at a time — say, writing them to a file line by line — you should use a generator expression instead. The syntax is identical, but with round brackets instead of square ones: (expression for item in iterable). A generator is lazy: it produces one item at a time on demand and holds almost nothing in memory at once.

There's also a subtler reason to avoid comprehensions: side effects. If the main purpose of your loop is to do something — print to the console, update a database, send a request — rather than to produce a value, a comprehension is the wrong tool. Using a comprehension purely for its side effects, and throwing away the resulting list, is a code smell that confuses readers and wastes memory.

Finally, never use a list comprehension when a built-in function already does the job more clearly. sum(), max(), filter(), and map() all exist precisely for common cases. Knowing when to reach for them instead is what separates intermediate from advanced Python.

import sys

product_prices = [12.99, 5.49, 89.00, 3.25, 45.50, 7.80]

# ── List comprehension: fine when you need the full list in memory ──
discounted_prices = [price * 0.9 for price in product_prices]
print("Discounted list:", discounted_prices)
print("Memory (list):", sys.getsizeof(discounted_prices), "bytes")

# ── Generator expression: use when you only need to iterate once ──
# Identical syntax, but with () instead of []
# Nothing is computed until you actually iterate
discounted_gen = (price * 0.9 for price in product_prices)
print("Memory (generator):", sys.getsizeof(discounted_gen), "bytes")

# Consume the generator once — after this it's exhausted
total_discounted = sum(discounted_gen)
print("Total after discount: $", round(total_discounted, 2))

# ── Anti-pattern: comprehension used purely for side effects ──
# This works but is misleading — it builds a list nobody uses
# BAD: [print(price) for price in product_prices]  # don't do this

# GOOD: use a regular for loop when you're doing work, not building a list
print("\n--- Price List ---")
for price in product_prices:
    print(f"  ${price:.2f}")  # side effect (printing) — loop is the right tool

# ── When a built-in is clearer than a comprehension ──
high_value_items = list(filter(lambda p: p > 10, product_prices))
print("\nHigh value (filter):", high_value_items)

# Or with a comprehension — equally readable here, your call
high_value_comp = [p for p in product_prices if p > 10]
print("High value (comp): ", high_value_comp)

Performance Deep Dive — When Comprehension Speed Actually Matters

The common wisdom says list comprehensions are ~10–35% faster than for loops with .append(). That's true, but the real question is: does that speedup matter in your use case? For small lists (a few hundred items), the difference is microseconds — not worth sacrificing readability. For large lists (hundreds of thousands or millions), the difference can be seconds, which might matter in a latency-sensitive pipeline.

Where comprehensions really shine is in data processing scripts, API response cleaning, and batch transformations. But be aware: if your comprehension calls a function that does I/O (database, file, network), the I/O cost will completely dominate — the comprehension's speedup becomes irrelevant.

There's another hidden cost: error handling. If a comprehension raises an exception, you lose all context — you can't easily tell which item caused it. In a for loop, you can wrap the transformation in a try/except and log the offending data. Debugging a comprehension that crashes in production often requires rewriting it as a loop just to add logging.

Also, be careful with heavily nested comprehensions. Each level of nesting adds overhead from multiple C loops. A two-level comprehension with a filter and a ternary may still be faster than a nested loop, but profile before you commit.

A practical tip: for numerical data, consider using NumPy instead of a list comprehension. A NumPy array operation is written in C and runs orders of magnitude faster than any Python-level comprehension. np.square(arr) vs [x**2 for x in arr] — the NumPy version is 10-50x faster on large arrays.

import timeit

# ── Benchmark: list comprehension vs for loop ──
setup = '''
data = list(range(10000))
'''

comprehension = '''
result = [x * 2 for x in data]
'''

loop = '''
result = []
for x in data:
    result.append(x * 2)
'''

comp_time = timeit.timeit(comprehension, setup, number=1000)
loop_time = timeit.timeit(loop, setup, number=1000)

print(f"Comprehension: {comp_time:.4f}s")
print(f"For loop:      {loop_time:.4f}s")
print(f"Speedup:       {loop_time/comp_time:.2f}x")

# ── The real trap: comprehension that calls a function ──
def transform(x):
    return x * 2  # simulate work

comprehension_func = '''
result = [transform(x) for x in data]
'''

loop_func = '''
result = []
for x in data:
    result.append(transform(x))
'''

comp_func_time = timeit.timeit(comprehension_func, setup + '\nfrom __main__ import transform', number=1000)
loop_func_time = timeit.timeit(loop_func, setup + '\nfrom __main__ import transform', number=1000)

print(f"\nWith function call:")
print(f"Comprehension: {comp_func_time:.4f}s")
print(f"For loop:      {loop_func_time:.4f}s")
print(f"Speedup:       {loop_func_time/comp_func_time:.2f}x")