C++ Memory Pools — How Exhaustion Crashed a Trading Engine

In a game engine rendering 120 frames per second, a trading system processing 10 million orders a minute, or an embedded firmware loop that must respond in under 50 microseconds, one thing will kill you faster than a logic bug: unpredictable memory allocation latency. The default new and delete operators are general-purpose tools — they handle any size, any time, with synchronization locks baked in. That generality has a cost, and in performance-critical C++ that cost is often unacceptable.

The standard heap allocator, whether it's glibc's ptmalloc, jemalloc, or TCMalloc, must maintain bookkeeping metadata, walk free-lists of variable sizes, coalesce adjacent freed blocks, and acquire locks in multithreaded contexts. Every call to new can trigger a system call, invalidate cache lines, and introduce allocations that are milliseconds apart in memory yet logically adjacent — fragmenting your address space until reallocation itself becomes a bottleneck. Memory pool allocators solve this by inverting the model: instead of asking the OS for memory on demand, you reserve a large contiguous arena upfront and serve allocations from it yourself, with full knowledge of your objects' sizes and lifetimes.

By the end of this article you'll understand how pool allocators work at the bit level, how to implement a thread-safe fixed-size pool and a more flexible slab-style allocator from scratch, how to plug them into STL containers via a custom allocator, and exactly when the tradeoffs make sense — and when they don't. You'll also leave with the mental model interviewers are probing for when they ask about custom allocators in systems programming interviews.

What is a Memory Pool Allocator?

A memory pool allocator (or fixed-size allocator) pre-allocates a large contiguous block of memory and divides it into slots of equal size. When you request an allocation, it returns a pointer to the next free slot — no searching, no coalescing. When you free, it marks the slot as available, often by pushing it onto a free list.

You'll find pool allocators in game engines, trading systems, embedded firmware, and custom STL allocators. They shine when you allocate many objects of the same type (e.g., particles, network packets, database rows) and you control their lifetimes tightly.

The key difference from general-purpose allocators is that pools don't handle arbitrary sizes. That's a feature, not a bug — by giving up generality, you gain speed and predictability.

// TheCodeForge — Fixed-size pool allocator
#include <cstddef>
#include <cassert>
#include <new>

namespace io::thecodeforge {

class FixedPool {
    struct Slot {
        union {
            Slot* next;  // free list pointer
            char data[1]; // placeholder, actual size at runtime
        };
    };

    Slot* pool_start_;
    Slot* free_head_;
    size_t slot_size_;
    size_t slot_count_;

public:
    FixedPool(size_t slot_size, size_t slot_count)
        : slot_size_(slot_size), slot_count_(slot_count) {
        if (slot_size < sizeof(Slot*))
            slot_size_ = sizeof(Slot*); // ensure we can store free ptr

        pool_start_ = static_cast<Slot*>(::operator new(slot_size_ * slot_count_));
        free_head_ = pool_start_;
        for (size_t i = 0; i < slot_count_ - 1; ++i) {
            Slot* current = reinterpret_cast<Slot*>(
                reinterpret_cast<char*>(pool_start_) + i * slot_size_);
            current->next = reinterpret_cast<Slot*>(
                reinterpret_cast<char*>(pool_start_) + (i + 1) * slot_size_);
        }
        // last slot points to null
        Slot* last = reinterpret_cast<Slot*>(
            reinterpret_cast<char*>(pool_start_) + (slot_count_ - 1) * slot_size_);
        last->next = nullptr;
    }

    void* allocate() {
        if (!free_head_) throw std::bad_alloc();
        Slot* slot = free_head_;
        free_head_ = slot->next;
        return slot;
    }

    void deallocate(void* ptr) {
        Slot* slot = static_cast<Slot*>(ptr);
        slot->next = free_head_;
        free_head_ = slot;
    }

    ~FixedPool() {
        ::operator delete(pool_start_);
    }
};

} // namespace

Slab Allocator: Handling Multiple Sizes

A slab allocator maintains several fixed-size pools (slabs) for different size classes. When you request 24 bytes, it returns a chunk from the 32-byte slab. This trades external fragmentation for some internal fragmentation per slab, but still avoids the overhead of walking free lists of variable sizes.

The Linux slab allocator popularised this approach. It also caches recently freed objects in a per-CPU cache to avoid locking entirely.

Implementing a simple slab allocator means creating an array of FixedPool objects, each with a size from a predefined size table (usually powers of two plus some mid-range entries: 16, 32, 64, 128, ...). The allocator rounds up the requested size to the next slab size.

In production, slab allocators work well for networking stacks and kernel-level memory management.

// TheCodeForge — Slab allocator with fixed size classes
#include <array>
#include <cstddef>
#include <new>
#include <algorithm>

namespace io::thecodeforge {

template <size_t... Sizes>
class SlabAllocator {
    static constexpr std::array<size_t, sizeof...(Sizes)> kSizes = {Sizes...};

    struct Slab {
        size_t slot_size;
        FixedPool* pool;
        Slab(size_t sz) : slot_size(sz), pool(nullptr) {}

        void ensure_pool(size_t count) {
            if (!pool)
                pool = new FixedPool(slot_size, count);
        }
    };

    Slab slabs_[sizeof...(Sizes)];  // one per size class
    size_t slab_count_ = sizeof...(Sizes);

public:
    SlabAllocator(size_t slots_per_slab = 256) {
        size_t idx = 0;
        ((slabs_[idx++] = Slab(Sizes)), ...);  // C++17 fold expression
    }

    void* allocate(size_t bytes) {
        for (auto& slab : slabs_) {
            if (bytes <= slab.slot_size) {
                slab.ensure_pool(256);
                return slab.pool->allocate();
            }
        }
        // no matching slab — fallback to malloc
        return ::operator new(bytes);
    }

    void deallocate(void* ptr, size_t bytes) {
        for (auto& slab : slabs_) {
            if (bytes <= slab.slot_size) {
                slab.pool->deallocate(ptr);
                return;
            }
        }
        ::operator delete(ptr);
    }
};

using SmallObjectAllocator = SlabAllocator<8, 16, 32, 64, 128, 256, 512, 1024>;

} // namespace

Thread-Safety Considerations

The simple pool above uses a single free list protected by nothing. That's safe only in a single-threaded context. In multi-threaded environments, you can't share the free_head_ pointer without protection — threads will race, corrupt the list, and crash.

Three common strategies: 1. Global mutex around allocate/deallocate — simple but kills performance under contention. 2. Thread-local pools — each thread has its own pool, no sharing. Objects must be freed on the same thread that allocated them, or you need a transfer mechanism. 3. Lock-free free list — use atomic compare-and-swap (CAS) to manipulate the head pointer. This is the fastest option for moderate contention.

Let's implement a lock‑free pool using std::atomic<Slot*>.

// TheCodeForge — Lock-free fixed-size pool
#include <atomic>
#include <cstddef>
#include <new>

namespace io::thecodeforge {

class LockFreePool {
    struct Slot {
        std::atomic<Slot*> next;
    };

    Slot* pool_start_;
    std::atomic<Slot*> free_head_;
    size_t slot_size_;
    size_t slot_count_;

public:
    LockFreePool(size_t slot_size, size_t slot_count)
        : slot_size_(slot_size), slot_count_(slot_count) {
        if (slot_size < sizeof(std::atomic<Slot*>))
            slot_size_ = sizeof(std::atomic<Slot*>);

        pool_start_ = reinterpret_cast<Slot*>(::operator new(slot_size_ * slot_count_));
        for (size_t i = 0; i < slot_count_; ++i) {
            Slot* s = reinterpret_cast<Slot*>(
                reinterpret_cast<char*>(pool_start_) + i * slot_size_);
            s->next.store(i + 1 < slot_count_ ?
                reinterpret_cast<Slot*>(reinterpret_cast<char*>(pool_start_) + (i + 1) * slot_size_) : nullptr,
                std::memory_order_relaxed);
        }
        free_head_.store(pool_start_, std::memory_order_relaxed);
    }

    void* allocate() {
        Slot* old_head = free_head_.load(std::memory_order_acquire);
        Slot* new_head;
        do {
            if (!old_head) return nullptr;  // pool exhausted
            new_head = old_head->next.load(std::memory_order_relaxed);
        } while (!free_head_.compare_exchange_weak(old_head, new_head,
                                                    std::memory_order_acq_rel,
                                                    std::memory_order_acquire));
        return old_head;
    }

    void deallocate(void* ptr) {
        Slot* slot = static_cast<Slot*>(ptr);
        Slot* old_head = free_head_.load(std::memory_order_acquire);
        do {
            slot->next.store(old_head, std::memory_order_relaxed);
        } while (!free_head_.compare_exchange_weak(old_head, slot,
                                                    std::memory_order_acq_rel,
                                                    std::memory_order_acquire));
    }

    ~LockFreePool() {
        ::operator delete(pool_start_);
    }
};

} // namespace

Plugging Into the STL: Custom Allocators

The STL containers like std::vector and std::map accept custom allocators via the template parameter. This lets you make std::vector> use your pool instead of new/delete.

A custom allocator must satisfy the Allocator concept: it needs allocate() and deallocate() methods, a rebind struct, and equality operators. The C++17 std::pmr::memory_resource provides a higher-level interface, but we'll show the classic approach.

Here's an adaptor that wraps our FixedPool into an STL allocator.

// TheCodeForge — STL-compatible pool allocator
#include <memory>
#include <cstddef>
#include "io/thecodeforge/fixed_pool.h"

namespace io::thecodeforge {

template <typename T>
class PoolAllocator {
public:
    using value_type = T;

    PoolAllocator(FixedPool& pool) noexcept : pool_(&pool) {}

    template <typename U>
    PoolAllocator(const PoolAllocator<U>& other) noexcept
        : pool_(other.pool_) {}

    T* allocate(std::size_t n) {
        if (n != 1) throw std::bad_alloc();  // pool is single-slot only
        return static_cast<T*>(pool_->allocate());
    }

    void deallocate(T* p, std::size_t n) noexcept {
        if (n == 1) pool_->deallocate(p);
    }

    template <typename U>
    struct rebind {
        using other = PoolAllocator<U>;
    };

    // equality: must share the same pool
    friend bool operator==(const PoolAllocator& a, const PoolAllocator& b) {
        return a.pool_ == b.pool_;
    }
    friend bool operator!=(const PoolAllocator& a, const PoolAllocator& b) {
        return a.pool_ != b.pool_;
    }

private:
    FixedPool* pool_;
    // allow cross-type copy
    template <typename U> friend class PoolAllocator;
};

} // namespace

When Pool Allocators Lose: Trade-offs and Failure Modes

Pool allocators are not a silver bullet. They fail badly when object sizes vary widely, lifetimes are unpredictable, or memory must be returned to the OS. Here are the problems that catch teams in production:

1. Internal fragmentation — Fixed-size slots waste space if objects are smaller than the slot. A 64-byte pool for 16-byte objects wastes 75% of memory.
2. Memory blow-up under load — Pools never shrink. If an allocator reserves 1MB for 1024 objects of 1KB, a single peak usage event pins that 1MB forever.
3. No way to return memory to OS — The ::operator new inside the pool is freed only when the pool destructs. Long-lived pools are effectively memory leaks from the OS perspective.
4. Debugging difficulty — Use-after-free errors inside a pool are hard to detect because the slot memory still belongs to the pool and appears valid. Standard tools like ASan often can't catch intra-pool violations unless you poison freed slots.

For these reasons, pool allocators are best for hot paths where allocation counts are high and object sizes are uniform. For cold paths or unpredictable workloads, stick with general-purpose allocators.

// TheCodeForge — demonstrating pool pitfalls
#include <iostream>
#include <memory>
#include <vector>

struct Particle { int x, y, vx, vy; };  // 16 bytes

// Suppose pool is sized for 1000 particles at 64 bytes each
struct LargePool {
    char data[64 * 1000];
    size_t next = 0;

    void* alloc() {
        if (next >= 1000) return nullptr;
        return data + (next++ * 64);
    }
    // ... no free() - pool never releases memory
};

int main() {
    LargePool pool;
    std::vector<Particle*> vec;
    // worst-case: each particle uses only 16 bytes but pool wastes 48
    for (int i = 0; i < 1000; ++i)
        vec.push_back(static_cast<Particle*>(pool.alloc()));
    std::cout << "Wasted space: " << (64-16)*1000 << " bytes\n";
    // pool memory never freed until program exit
    return 0;
}

Why the Default Allocator Betrays You Under Load

You don't notice malloc fragmentation until your latency spikes 200x. The heap is a shared resource. Every thread's deallocation can dirty a cache line another thread needs. Pool allocators solve this by treating memory as a flat array of fixed-size slots. No fragmentation, no coalescing, no kernel calls. You reserve a contiguous block once, then hand out slots with a pointer bump or a free list. When every allocation is O(1) and every deallocation just pushes a pointer back onto a stack, your allocator stops being a bottleneck. The real win is CPU cache behavior: sequential allocations land on adjacent cache lines. The WHY is simple: malloc pays for generality you don't need. A pool pays for exactly your workload.

// io.thecodeforge
#include <cstddef>
#include <cstdint>
#include <new>

template<typename T, size_t PoolSize = 4096>
class PoolAllocator {
    alignas(T) char pool_[PoolSize * sizeof(T)];
    T* next_;
public:
    PoolAllocator() : next_(reinterpret_cast<T*>(pool_)) {}

    T* allocate() {
        if (next_ >= reinterpret_cast<T*>(pool_ + PoolSize * sizeof(T)))
            throw std::bad_alloc();
        return new (next_++) T();
    }

    void deallocate(T*) noexcept {
        // Fixed pool: no per-slot free. Reset pool_ pointer to start for bulk reuse.
    }

    void reset() { next_ = reinterpret_cast<T*>(pool_); }
};

Stack Allocator: The Arena Pattern for Transient Work

Your game loop or request handler allocates thousands of small objects, then frees them all at once. Default allocators thrash the heap for no reason. The arena (or stack allocator) is your weapon: grab a large contiguous buffer, bump a pointer for each allocation, and reset the pointer when the frame or request ends. No deallocation calls during the hot path. No fragmentation. No thread contention if each thread owns its arena. The WHY: your allocation pattern is LIFO or bulk-free. You don't need individual frees. This is how game engines handle per-frame allocations and how HTTP servers handle per-request buffers. The cost is one: you cannot free a single object—only the whole arena. That constraint forces you to write code that batches lifetimes.

// io.thecodeforge
#include <cstddef>
#include <cstdint>

class Arena {
    char* const start_;
    char* current_;
    const size_t size_;
public:
    Arena(size_t bytes) 
        : start_(static_cast<char*>(::operator new(bytes)))
        , current_(start_), size_(bytes) {}

    ~Arena() { ::operator delete(start_); }

    void* alloc(size_t sz, size_t align) {
        // Align pointer up, then check capacity
        uintptr_t raw = reinterpret_cast<uintptr_t>(current_);
        uintptr_t aligned = (raw + align - 1) & ~(align - 1);
        if (aligned + sz > reinterpret_cast<uintptr_t>(start_) + size_)
            return nullptr;
        current_ = reinterpret_cast<char*>(aligned + sz);
        return reinterpret_cast<void*>(aligned);
    }

    void reset() { current_ = start_; }
};

std::pmr::memory_resource: Polymorphic Allocators

Polymorphic allocators, introduced in C++17 via std::pmr::memory_resource, decouple container allocation from the underlying memory source. Instead of hardcoding new/delete or a custom allocator type, containers like std::pmr::vector accept a memory_resource* at construction, allowing runtime selection of allocation strategies. This is especially useful in trading engines where different subsystems (e.g., order book, market data) may require distinct pool behaviors.

A memory_resource is an abstract base class with two pure virtual functions: do_allocate(size_t bytes, size_t alignment) and do_deallocate(void* p, size_t bytes, size_t alignment). The standard provides concrete resources: std::pmr::new_delete_resource() (default heap), std::pmr::null_memory_resource() (throws std::bad_alloc on any allocation), and std::pmr::monotonic_buffer_resource (a fast arena allocator).

To use a pool, you can wrap it in a custom memory_resource or use the standard pool resources. For example, a monotonic buffer resource can be used for transient work: ```cpp #include #include #include

int main() { std::array buffer; std::pmr::monotonic_buffer_resource pool(buffer.data(), buffer.size()); std::pmr::vector vec(&pool); vec.push_back(42); // allocates from buffer // No deallocation needed; buffer is released when pool is destroyed } `` This pattern eliminates per-element free` overhead, ideal for high-frequency trading loops where allocation bursts are followed by bulk teardown.

#include <pmr/memory_resource>
#include <pmr/vector>
#include <array>

int main() {
    std::array<std::byte, 1024> buffer;
    std::pmr::monotonic_buffer_resource pool(buffer.data(), buffer.size());
    std::pmr::vector<int> vec(&pool);
    vec.push_back(42);
}

C++23: std::pmr::synchronized_pool_resource and unsynchronized_pool_resource

C++23 introduces two new standard pool resources: std::pmr::synchronized_pool_resource and std::pmr::unsynchronized_pool_resource. These are thread-safe and non-thread-safe versions, respectively, of a pool allocator that manages fixed-size blocks. They are designed to reduce fragmentation and improve cache locality for frequent allocations of similar sizes.

Both resources maintain pools of memory chunks, each chunk subdivided into blocks of a specific size. When a request arrives, the resource picks the pool whose block size is large enough (rounded up to the nearest power of two). This minimizes waste and speeds up allocation/deallocation. The synchronized variant uses mutexes internally, making it safe for concurrent access; the unsynchronized variant is faster but requires external synchronization.

Example usage: ```cpp #include <pmr/memory_resource> #include <pmr/vector> #include <pmr/synchronized_pool_resource>

int main() { std::pmr::synchronized_pool_resource pool; // uses new_delete as upstream std::pmr::vector<int> vec(&pool); vec.reserve(1000); // allocates from pool // Thread-safe allocations from multiple threads } `` These resources are ideal for trading engines where multiple threads allocate and deallocate objects of varying sizes (e.g., order objects, trade records). The synchronized variant can replace custom thread-safe pools, reducing code complexity. However, they still rely on an upstream resource (default new_delete`) for large allocations, so monitor for unexpected upstream calls.

#include <pmr/memory_resource>
#include <pmr/vector>
#include <pmr/synchronized_pool_resource>

int main() {
    std::pmr::synchronized_pool_resource pool;
    std::pmr::vector<int> vec(&pool);
    vec.reserve(1000);
}

Custom Memory Resource Implementation Pattern

When standard pool resources don't meet your latency or fragmentation requirements, implementing a custom memory_resource gives you full control. The pattern involves inheriting from std::pmr::memory_resource and overriding do_allocate, do_deallocate, and do_is_equal. This allows you to integrate existing pool allocators (e.g., a slab allocator) into the polymorphic allocator framework.

A minimal custom resource that wraps a simple bump allocator: ```cpp #include #include

class BumpResource : public std::pmr::memory_resource { char buffer; size_t capacity; size_t offset = 0; public: BumpResource(char buf, size_t cap) : buffer(buf), capacity(cap) {} private: void do_allocate(size_t bytes, size_t alignment) override { // Align offset size_t aligned = (offset + alignment - 1) & ~(alignment - 1); if (aligned + bytes > capacity) throw std::bad_alloc(); void ptr = buffer + aligned; offset = aligned + bytes; return ptr; } void do_deallocate(void, size_t, size_t) override { / no-op */ } bool do_is_equal(const memory_resource& other) const noexcept override { return this == &other; } }; `` This bump allocator is extremely fast (no free) and perfect for transient work. For a full-featured pool, you'd manage free lists and reuse blocks. The do_is_equal` method is critical for container operations like swap; it should return true only if both resources are identical (same pool).

Custom resources allow you to implement advanced strategies: thread-local caches, NUMA-aware allocation, or leak detection. In a trading engine, you might create a resource that allocates from a pre-allocated shared memory segment for inter-process communication.

#include <pmr/memory_resource>
#include <cstddef>

class BumpResource : public std::pmr::memory_resource {
    char* buffer;
    size_t capacity;
    size_t offset = 0;
public:
    BumpResource(char* buf, size_t cap) : buffer(buf), capacity(cap) {}
private:
    void* do_allocate(size_t bytes, size_t alignment) override {
        size_t aligned = (offset + alignment - 1) & ~(alignment - 1);
        if (aligned + bytes > capacity) throw std::bad_alloc();
        void* ptr = buffer + aligned;
        offset = aligned + bytes;
        return ptr;
    }
    void do_deallocate(void*, size_t, size_t) override {}
    bool do_is_equal(const memory_resource& other) const noexcept override {
        return this == &other;
    }
};