Python collections — Counter Drops Zero-Count Keys

Every Python developer hits the same wall eventually. You are writing a word-frequency counter and you keep asking 'does this key exist yet?' before incrementing it. You are modelling a playing card and passing around a plain tuple, quietly hoping nobody accesses index 0 when they meant index 1. You are building a task processor and discovering, two weeks too late, that list.pop(0) is O(n) and your queue is now a bottleneck. These are not exotic edge cases — they are the everyday friction points that the collections module was designed to eliminate, and it has been shipping with Python since version 2.4.

The module solves a specific class of problem: data-wrangling tasks that are just awkward enough with built-in types to force you into boilerplate, but not complex enough to justify a third-party library. Instead of writing a three-line 'if key not in dict' guard every time you want a default value, defaultdict handles it in zero extra lines. Instead of sorting a list of tuples and trying to remember which index means 'price', namedtuple gives every field a readable name. The module trades a small learning investment for a measurable reduction in repetitive, error-prone code.

The module provides nine types in total. This article gives you deep coverage of the six you will reach for most often in production code — Counter, defaultdict, namedtuple, deque, OrderedDict, and ChainMap. The remaining three — UserDict, UserList, and UserString — are base classes designed for safely subclassing Python's built-in dict, list, and str types. They exist because inheriting directly from dict or list can produce subtle method-override bugs; the User variants route all operations through a single internal data store so your overrides always fire correctly. They are worth knowing about, but they solve a library-author problem rather than an everyday application problem, so they sit outside the scope of this article.

By the end you will know exactly which collection to reach for when you are counting things, building queues, working with structured data, or layering configuration. You will understand the performance trade-offs, the traps that catch experienced engineers, and how to talk about these types confidently in a technical interview.

Counter — Count Anything in One Line

Counter is a dict subclass built for one job: tallying. Hand it any iterable — a string, a list, a generator of log lines — and it returns a dict-like object where each key is an element and each value is how many times that element appeared. No initialisation loop, no conditional increment, no boilerplate of any kind.

Beyond raw counting, Counter ships with helper methods that make it genuinely composable in real pipelines. most_common(n) returns the n highest-frequency elements sorted descending — ideal for building leaderboards or word-cloud datasets. You can add two Counters together with + to merge tallies from separate data sources, making it trivial to combine weekly batch results without a manual merge loop.

The right mental model is the multiset, also called a bag. A bag is like a set that remembers multiplicity — it tracks not just what exists, but how many copies of each thing exist. When you need to know not just whether something is present but how many times it appears, Counter is your type. Real-world examples include analysing HTTP access logs to find the most-requested endpoints, scoring a Scrabble hand by letter frequency, or detecting duplicate records in a data pipeline.

One behaviour that consistently catches engineers: Counter arithmetic silently drops keys whose result is zero or negative. This is deliberate — a multiset with zero copies of an item simply does not contain that item. But when you are comparing two frequency distributions and you need to know which keys went to zero, that silent removal will bite you. The production incident section covers exactly this failure. The safe habit is to verify your expected keys still exist after any Counter arithmetic before passing results downstream.

Counter also returns 0 for missing keys rather than raising KeyError. This is enormously convenient for frequency checks — you can ask 'how many times did this word appear?' without guarding against the word being absent entirely.

from collections import Counter

# Analysing customer support feedback — a realistic text-processing pipeline
feedback_words = [
    "slow", "buggy", "slow", "great", "slow",
    "buggy", "excellent", "great", "slow", "excellent"
]

# Counter tallies every element automatically — no manual dict initialisation needed
word_tally = Counter(feedback_words)
print("Full tally:", word_tally)
# Output: Counter({'slow': 4, 'buggy': 2, 'great': 2, 'excellent': 2})

# most_common gives you a ranked list — top 3 complaints surfaced instantly
top_issues = word_tally.most_common(3)
print("Top 3 words:", top_issues)
# Output: [('slow', 4), ('buggy', 2), ('great', 2)]

# Counters support + arithmetic to merge two batches of feedback
week2_feedback = Counter(["slow", "excellent", "excellent", "buggy"])
combined = word_tally + week2_feedback
print("Combined over 2 weeks:", combined)
# Output: Counter({'slow': 5, 'excellent': 4, 'buggy': 3, 'great': 2})

# Missing keys return 0 — no KeyError, no guard clause needed
print("Count of 'terrible':", word_tally["terrible"])
# Output: Count of 'terrible': 0

# Subtraction — this is where engineers get caught out
# Counter drops any key whose result is zero or negative
difference = word_tally - week2_feedback
print("Difference (keys with zero result are gone):", difference)
# Output: Counter({'slow': 3, 'great': 2})
# Notice 'buggy' and 'excellent' are completely absent — their counts cancelled out
# This is intentional multiset behaviour, but silent enough to cause real bugs

# Safe comparison when zero-count keys matter:
# Use a plain dict and subtract manually
safe_diff = {
    key: word_tally.get(key, 0) - week2_feedback.get(key, 0)
    for key in word_tally
}
print("Safe difference (zero counts preserved):", safe_diff)
# Output: {'slow': 3, 'buggy': 0, 'great': 2, 'excellent': 0}

defaultdict — Stop Writing 'if key not in dict' Forever

A defaultdict is a dict that automatically creates a value for a key the moment you first access it. You supply a callable — int, list, set, or any zero-argument function — when you construct it. That callable is invoked to produce the default value whenever a missing key is accessed. No KeyError, no setdefault gymnastics, no conditional guard.

The canonical use-case is grouping. You have a list of (student, subject) pairs and you want a dict that maps each student to a list of their subjects. With a plain dict you write three lines per insertion: check if the key exists, create an empty list if not, then append. With defaultdict(list) you just append — the empty list materialises automatically on first access.

Under the hood, defaultdict overrides __missing__, which is the method dict calls when a key lookup fails. This means defaultdict behaves identically to a regular dict in every other way — you can iterate it, pass it anywhere a dict is expected, and convert it with dict() for JSON serialisation. The only difference is that absent-key access never raises; it constructs.

One detail that trips up engineers: the default_factory callable is evaluated at __missing__ time, not at construction time. This means every missing key gets its own fresh default value — two different missing keys each get their own independent empty list, not references to the same list. That is the correct behaviour, and it is why defaultdict is safe for grouping.

For nested defaultdicts, you need to pass a callable that itself returns a defaultdict. The pattern is:

nested = defaultdict(lambda: defaultdict(list))

This works because the lambda is called fresh for each new outer key, producing a brand-new inner defaultdict(list) each time. The common mistake is writing defaultdict(defaultdict(list)) — this evaluates defaultdict(list) immediately, producing a single shared instance, and then passes that one instance as the factory. Every outer key ends up pointing to the same inner dict, causing all groups to bleed into each other. The factory must be a callable that returns a new object each time, not an object itself.

from collections import defaultdict

# Raw enrolment data — each tuple is (student_name, subject)
enrolments = [
    ("Alice", "Maths"),
    ("Bob", "Science"),
    ("Alice", "Science"),
    ("Charlie", "Maths"),
    ("Bob", "History"),
    ("Alice", "History"),
]

# defaultdict(list) creates an empty list automatically for any new key
student_subjects = defaultdict(list)

for student, subject in enrolments:
    # No 'if student not in student_subjects' guard needed — it just works
    student_subjects[student].append(subject)

print("Student subjects:", dict(student_subjects))
# Output: {'Alice': ['Maths', 'Science', 'History'], 'Bob': ['Science', 'History'], 'Charlie': ['Maths']}

# defaultdict(int) is perfect for manual counting when Counter is too heavy
vote_counts = defaultdict(int)
votes = ["Alice", "Bob", "Alice", "Charlie", "Alice", "Bob"]
for candidate in votes:
    vote_counts[candidate] += 1  # int() returns 0, so += 1 works on first access

print("Vote counts:", dict(vote_counts))
# Output: {'Alice': 3, 'Bob': 2, 'Charlie': 1}

# defaultdict(set) builds unique-value groups without any dedup logic
page_visitors = defaultdict(set)
visits = [("home", "user_1"), ("home", "user_2"), ("home", "user_1"), ("about", "user_1")]
for page, user in visits:
    page_visitors[page].add(user)  # set deduplicates automatically

print("Unique visitors per page:", dict(page_visitors))
# Output: {'home': {'user_1', 'user_2'}, 'about': {'user_1'}}

# --- The nested defaultdict pattern ---
# Correct: lambda returns a fresh defaultdict(list) for each new outer key
course_grades = defaultdict(lambda: defaultdict(list))
course_grades["Maths"]["Alice"].append(92)
course_grades["Maths"]["Bob"].append(85)
course_grades["Science"]["Alice"].append(88)
print("Course grades:", dict(course_grades["Maths"]))
# Output: {'Alice': [92], 'Bob': [85]}

# Wrong pattern — do NOT do this:
# broken = defaultdict(defaultdict(list))
# Every outer key would share the SAME inner defaultdict instance
# Appending to broken['Maths']['Alice'] would also affect broken['Science']['Alice']

namedtuple — Give Your Tuples a Memory

A plain tuple is positional amnesia. (52.3, -1.8) means nothing until you remember whether index 0 is latitude or longitude. namedtuple fixes this by generating a tuple subclass at runtime where every position has a name. It is immutable like a tuple, memory-efficient like a tuple, but readable like an object.

The magic is that namedtuple generates a real class — complete with __repr__, __eq__, and field access by attribute name. You get all the benefits of a lightweight data class without writing a single method. This is why namedtuple shows up throughout the standard library: os.stat_result, sys.version_info, and socket.getaddrinfo all return namedtuples or namedtuple-like objects.

The right mental model: namedtuple sits neatly between raw tuples (too opaque) and full classes (too heavy). It is the right choice when your data is immutable, has a fixed number of fields, and you want readable field access without defining a class. When you need mutability, default values, or methods, reach for dataclasses instead.

A few details worth knowing before you ship code with namedtuple. First, _asdict() returns a plain dict in Python 3.8 and later — it returned an OrderedDict in earlier versions, so if you have old code or documentation that says OrderedDict, update it. Second, _replace() creates a full copy of the tuple, which is the correct behaviour for an immutable type but becomes expensive when you are calling it millions of times inside a tight loop — at that scale, a list of plain dicts or a pandas DataFrame is a better fit. Third, default values are supported via the defaults parameter from Python 3.6.1 onwards, but the older workaround of using the class-based typing.NamedTuple form reads more cleanly when you have many fields with defaults.

from collections import namedtuple

# Define the structure once — this generates a real class called 'Product'
Product = namedtuple('Product', ['name', 'price', 'stock', 'category'])

# Instantiate like a class — no index guessing, no positional amnesia
laptop = Product(name="ProBook 450", price=899.99, stock=12, category="Electronics")
headphones = Product(name="SoundWave Pro", price=149.99, stock=35, category="Audio")
desk = Product(name="Standing Desk", price=399.00, stock=5, category="Furniture")

# Access fields by name — code reads like a sentence
print(f"{laptop.name} costs £{laptop.price} and has {laptop.stock} units in stock.")
# Output: ProBook 450 costs £899.99 and has 12 units in stock.

# Still a tuple underneath — indexing and unpacking both work
print("Price via index:", laptop[1])
# Output: Price via index: 899.99

# _replace creates a new instance with one field changed (immutability is preserved)
updated_laptop = laptop._replace(stock=10)
print("Updated stock:", updated_laptop.stock)
# Output: Updated stock: 10

# Sorts, filters, and list comprehensions all work naturally
catalogue = [laptop, headphones, desk]
by_price = sorted(catalogue, key=lambda product: product.price)
for item in by_price:
    print(f"{item.name}: £{item.price}")
# Output:
# SoundWave Pro: £149.99
# Standing Desk: £399.0
# ProBook 450: £899.99

# _asdict() returns a plain dict in Python 3.8+ (not an OrderedDict as in earlier versions)
print(type(laptop._asdict()))  # <class 'dict'>
print(laptop._asdict())
# Output: {'name': 'ProBook 450', 'price': 899.99, 'stock': 12, 'category': 'Electronics'}

# namedtuple with defaults (Python 3.6.1+)
# Last N field names get the default values from left to right
ProductWithDefault = namedtuple(
    'ProductWithDefault',
    ['name', 'price', 'stock', 'category'],
    defaults=['Uncategorised']  # only 'category' gets a default
)
backup_drive = ProductWithDefault(name="BackupDrive 2TB", price=79.99, stock=50)
print(backup_drive.category)
# Output: Uncategorised

deque — The Double-Ended Queue That Outperforms Lists

A Python list is secretly bad at one thing: inserting or removing elements from the front. list.pop(0) and list.insert(0, item) are O(n) operations because Python has to shift every other element in memory after the removal point. For small lists you will never notice. For a queue processing thousands of events per second, or a rolling window accumulating sensor data, it is a hidden bottleneck that appears gradually as your data grows.

deque (pronounced 'deck', short for double-ended queue) solves this with O(1) appends and pops from both ends. It is backed by a doubly-linked list of fixed-size blocks, so adding or removing from either end is always a pointer update — no shifting, no reallocation. Use deque any time your data structure is conceptually a queue (first-in, first-out), a stack (last-in, first-out), or a sliding window over the most recent N items.

The maxlen parameter is one of deque's most useful features. When set, the deque automatically discards items from the opposite end when it fills. This gives you a rolling window — a fixed-size buffer that always holds the most recent N items — in zero extra code. Think: last 100 log lines for a live tail view, last 10 sensor readings for an average calculation, or last 5 user actions for an undo buffer.

Deque does not support O(1) random access by index. Accessing deque[500] traverses nodes from the nearest end, making it O(n). This is not a bug or an oversight — it is an inherent property of the linked-block structure that gives you O(1) ends. If you need fast random access by index alongside fast ends, neither list nor deque is a perfect fit. In practice, most queue and window patterns access only the ends, so this is rarely a real constraint.

One more thing: thread safety. Individual append and popleft operations are atomic in CPython due to the GIL, but the GIL is increasingly opt-in rather than guaranteed — Python 3.13 introduced a free-threaded build mode. Compound operations such as 'check if non-empty, then popleft' are never atomic, regardless of implementation. For thread-safe producer-consumer patterns, use queue.Queue, which is designed for exactly that purpose.

from collections import deque

# --- USE CASE 1: Efficient task queue ---
# Simulating a print job queue — FIFO with occasional priority bumps
print_queue = deque()

print_queue.append("Invoice_March.pdf")       # Normal priority — goes to the right
print_queue.append("Report_Q1.xlsx")
print_queue.appendleft("URGENT_Contract.pdf")  # High priority — jumps to the front

print("Queue state:", list(print_queue))
# Output: Queue state: ['URGENT_Contract.pdf', 'Invoice_March.pdf', 'Report_Q1.xlsx']

# Process jobs FIFO — popleft is O(1), list.pop(0) would be O(n)
while print_queue:
    job = print_queue.popleft()
    print(f"Printing: {job}")
# Output:
# Printing: URGENT_Contract.pdf
# Printing: Invoice_March.pdf
# Printing: Report_Q1.xlsx

print()

# --- USE CASE 2: Rolling window with maxlen ---
# Keeping a live feed of the last 4 server response times in milliseconds
response_times = deque(maxlen=4)  # Automatically discards the oldest when full

readings = [120, 135, 98, 210, 87, 310, 95]
for reading in readings:
    response_times.append(reading)
    avg = sum(response_times) / len(response_times)
    print(f"Added {reading}ms | Window: {list(response_times)} | Avg: {avg:.1f}ms")

# The window slides automatically — no manual eviction code needed

print()

# --- USE CASE 3: Why list.pop(0) hurts at scale ---
import timeit

# Build a list and a deque with 100,000 elements
test_list = list(range(100_000))
test_deque = deque(range(100_000))

list_time = timeit.timeit(lambda: test_list.insert(0, 0), number=1_000)
deque_time = timeit.timeit(lambda: test_deque.appendleft(0), number=1_000)

print(f"list.insert(0) for 1000 ops: {list_time:.4f}s")
print(f"deque.appendleft() for 1000 ops: {deque_time:.4f}s")
# deque is typically 10-100x faster for front insertions on large collections

OrderedDict — Order Matters When Order Matters

Since Python 3.7, regular dicts preserve insertion order as a language guarantee — not an implementation detail. So you might reasonably ask: why does OrderedDict still exist? The answer is two specific capabilities a plain dict will never have: move_to_end() for reordering entries in place, and order-sensitive equality where two OrderedDicts are considered equal only if they contain the same keys in the same sequence.

move_to_end(key, last=True) moves an existing key to the end of the dictionary — or to the front if you pass last=False. This single method makes OrderedDict the natural foundation for an LRU cache. You maintain a fixed-capacity OrderedDict: on every cache hit, move the accessed key to the end (most recently used). When the cache is full and a new key arrives, pop the first item (least recently used) with popitem(last=False). The whole mechanism fits in about fifteen lines without any additional data structures.

Order-sensitive equality is the other differentiator. Two regular dicts with the same keys and values are always equal regardless of insertion order. Two OrderedDicts are equal only if the key order also matches. This matters when you are testing serialised output — for example, verifying that a configuration dict round-trips through JSON in a specific field order, or asserting that a pipeline produces results in a defined sequence.

The cost is memory. OrderedDict maintains an internal doubly-linked list to track ordering, which roughly doubles its memory footprint versus a plain dict. For small collections this is irrelevant. For millions of keys it is significant. If you do not need move_to_end() or order-sensitive equality, a plain dict is strictly better — both faster and lighter. That is the practical decision rule: start with a plain dict, upgrade to OrderedDict only when you reach for one of those two specific features.

from collections import OrderedDict

class LRUCache:
    """Fixed-capacity cache that evicts the least recently used item."""

    def __init__(self, capacity: int):
        self.cache = OrderedDict()
        self.capacity = capacity

    def get(self, key):
        if key not in self.cache:
            return -1
        # Move to end — this marks the key as most recently used
        self.cache.move_to_end(key)
        return self.cache[key]

    def put(self, key, value):
        if key in self.cache:
            # Refresh position of existing key
            self.cache.move_to_end(key)
        elif len(self.cache) >= self.capacity:
            # Evict the oldest entry — the first item in the OrderedDict
            evicted_key, _ = self.cache.popitem(last=False)
            print(f"  [evict] '{evicted_key}' removed (LRU)")
        self.cache[key] = value

# Simulate a cache with capacity 3
cache = LRUCache(capacity=3)

print("PUT A=1"); cache.put('A', 1); print(" State:", list(cache.cache.items()))
print("PUT B=2"); cache.put('B', 2); print(" State:", list(cache.cache.items()))
print("PUT C=3"); cache.put('C', 3); print(" State:", list(cache.cache.items()))
print("GET A  ->", cache.get('A'));  print(" State:", list(cache.cache.items()))
print("GET B  ->", cache.get('B'));  print(" State:", list(cache.cache.items()))
print("PUT D=4"); cache.put('D', 4); print(" State:", list(cache.cache.items()))
print("GET C  ->", cache.get('C'));  print(" State:", list(cache.cache.items()))
print("GET D  ->", cache.get('D'));  print(" State:", list(cache.cache.items()))

# Demonstrating order-sensitive equality — the plain dict difference
from collections import OrderedDict
ord1 = OrderedDict([('x', 1), ('y', 2)])
ord2 = OrderedDict([('y', 2), ('x', 1)])
plain1 = {'x': 1, 'y': 2}
plain2 = {'y': 2, 'x': 1}

print("\nOrderedDict equal (same order)?  ", ord1 == ord2)   # False — order differs
print("OrderedDict equal (diff order)?  ", ord1 == ord2)   # False
print("plain dict equal (any order)?    ", plain1 == plain2) # True — order ignored

ChainMap — Layer Multiple Dictionaries Without Merging

ChainMap takes multiple dictionaries and presents them as a single logical mapping without copying any data. Lookups search each dict in the order you supplied until the key is found; writes modify only the first dict. This is precisely the pattern you need for layered configuration: command-line flags override environment variables, which override config file settings, which override hard-coded defaults — and you want all four layers to remain independent so each can be updated without touching the others.

The memory efficiency comes from holding references to the original dicts rather than creating copies. When you update one of the underlying dicts, ChainMap reflects the change immediately on the next lookup. This makes it correct for runtime-configurable systems where overrides may be temporary — for example, a request-scoped context that adds per-request flags on top of global application settings, then discards the override when the request ends.

new_child() adds a new empty dict to the front of the chain and returns a new ChainMap — leaving the original untouched. This is useful for scoped overrides: create a child context, let it shadow the parent for the duration of a block, then discard it. parents returns the rest of the chain without the first mapping, which is how you walk up through the layers.

Two behaviours are worth knowing precisely before you rely on ChainMap in production. First, writes. Setting a key through ChainMap always writes to the first mapping — even if that key already exists in a deeper layer. If you want to update a key in a specific deeper layer, you must directly access that dict. This is intentional and correct for the override model, but it surprises engineers who expect assignment to update the key wherever it lives. Second, len() and keys(). ChainMap.len() and ChainMap.keys() count unique keys across all mappings — a key shadowed by an earlier layer is counted once, not multiple times. If you are trying to count all occurrences of a key across every layer, you need to iterate through each underlying dict manually.

from collections import ChainMap

# Three configuration layers, each a plain dict — no copies, no merging
defaults = {
    'host': 'localhost',
    'port': 5432,
    'dbname': 'app',
    'debug': False,
    'timeout': 30,
}

environment = {
    'host': 'prod-db.example.com',
    'debug': True,   # env enables debug mode
}

# Parsed from sys.argv in a real application
cli_overrides = {
    'port': 15432,
}

# Priority: CLI > environment > defaults
config = ChainMap(cli_overrides, environment, defaults)

# Lookups search from left to right — first match wins
print(f"host   : {config['host']}")     # from environment
print(f"port   : {config['port']}")     # from CLI
print(f"debug  : {config['debug']}")    # from environment
print(f"dbname : {config['dbname']}")   # from defaults
print(f"timeout: {config['timeout']}")  # from defaults

print()

# Writes go to the first mapping only
config['port'] = 5433
print(f"After write — CLI dict port : {cli_overrides['port']}")   # 5433
print(f"After write — env dict port : {environment.get('port', 'not set')}")
print(f"After write — defaults port : {defaults['port']}")
# Only cli_overrides was modified — the other dicts are unchanged

print()

# len() and keys() count unique keys across all layers
print("Unique keys:", list(config.keys()))
print("Total unique key count:", len(config))
# 'host', 'debug' appear in multiple layers but are counted once each

print()

# new_child() adds a temporary override layer — original chain is untouched
request_config = config.new_child({'timeout': 5, 'request_id': 'req_abc123'})
print(f"Request timeout  : {request_config['timeout']}")     # 5 (overridden)
print(f"Global timeout   : {config['timeout']}")             # 30 (unchanged)
print(f"Request host     : {request_config['host']}")        # from environment via parent chain

# parents gives you the chain without the child layer
print(f"Parent chain timeout: {request_config.parents['timeout']}")  # 30