Polars: Fast DataFrame Library for Python – A Complete Guide

If you've ever worked with pandas on a dataset that doesn't fit into memory or takes minutes to process, you know the pain. Enter Polars: a DataFrame library built in Rust that leverages modern hardware with multi-threading and cache-efficient algorithms. Polars offers both eager and lazy APIs, allowing you to build complex query plans that are optimized before execution. It's designed for speed and memory efficiency, often outperforming pandas by 10-100x on common operations. In this tutorial, you'll learn the core concepts of Polars, including expressions, lazy evaluation, and how to integrate it into your data pipeline. We'll cover real-world scenarios like reading large CSV files, filtering, grouping, and joining datasets. By the end, you'll be able to write production-ready Polars code that is both fast and expressive.

Getting Started with Polars

First, install Polars via pip: pip install polars. Polars has no dependencies on pandas or numpy, making it lightweight. Import it as import polars as pl. Let's create a simple DataFrame:

import polars as pl

df = pl.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'city': ['New York', 'London', 'Tokyo']
})
print(df)

Expressions: The Building Blocks

In Polars, operations are expressed using expressions. An expression is a combination of column references, functions, and operators. Expressions are composable and can be reused. For example, to select columns and create new ones:

import polars as pl

df = pl.DataFrame({
    'a': [1, 2, 3],
    'b': [4, 5, 6]
})

# Expression: select and transform
result = df.select([
    pl.col('a'),
    pl.col('b') * 2,
    (pl.col('a') + pl.col('b')).alias('sum')
])
print(result)

Lazy vs Eager Execution

Polars supports two modes: eager (default) and lazy. In eager mode, operations are executed immediately. In lazy mode, you build a query plan that is optimized and executed only when you call .collect(). Lazy evaluation can significantly improve performance by reducing the number of passes over data and enabling predicate pushdown. To use lazy mode, start with .lazy() on a DataFrame or use pl.scan_csv() for files.

import polars as pl

# Eager: immediate execution
df_eager = pl.DataFrame({'x': [1, 2, 3]})
result_eager = df_eager.filter(pl.col('x') > 1)
print('Eager result:', result_eager)

# Lazy: build query plan
q = df_eager.lazy().filter(pl.col('x') > 1)
print('Query plan:')
print(q.explain())
result_lazy = q.collect()
print('Lazy result:', result_lazy)

Reading and Writing Data

Polars supports many file formats: CSV, Parquet, JSON, Arrow, and more. For large files, use lazy scanning. For example, reading a CSV lazily: q = pl.scan_csv('data.csv'). Then you can apply transformations and collect. Writing is straightforward: df.write_csv('output.csv') or df.write_parquet('output.parquet'). For lazy queries, use .sink_csv() or .sink_parquet() to write without collecting.

import polars as pl

# Lazy read CSV
q = pl.scan_csv('data.csv')

# Filter and select
q_filtered = q.filter(pl.col('value') > 100).select(['id', 'value'])

# Write to Parquet without collecting
q_filtered.sink_parquet('output.parquet')

# Alternatively, collect and write
# df = q_filtered.collect()
# df.write_parquet('output.parquet')

GroupBy and Aggregations

Polars provides powerful groupby operations. Use .groupby() followed by .agg() with expressions. For example, to compute mean and sum per group:

import polars as pl

df = pl.DataFrame({
    'group': ['A', 'A', 'B', 'B'],
    'value': [10, 20, 30, 40]
})

result = df.groupby('group').agg([
    pl.col('value').mean().alias('mean_value'),
    pl.col('value').sum().alias('sum_value')
])
print(result)

Joins and Concatenation

Polars supports all standard joins: inner, left, outer, cross, and anti. Use .join() with a key. For example:

import polars as pl

df1 = pl.DataFrame({'id': [1, 2], 'name': ['Alice', 'Bob']})
df2 = pl.DataFrame({'id': [1, 3], 'score': [95, 87]})

# Inner join
inner = df1.join(df2, on='id', how='inner')
print('Inner join:')
print(inner)

# Left join
left = df1.join(df2, on='id', how='left')
print('Left join:')
print(left)

Handling Missing Data

Polars represents missing values as null. You can drop nulls with .drop_nulls() or fill them with .fill_null(). For example:

import polars as pl

df = pl.DataFrame({'a': [1, None, 3], 'b': [None, 5, 6]})
print('Original:')
print(df)

# Drop rows with any null
print('Drop nulls:')
print(df.drop_nulls())

# Fill nulls with a value
print('Fill nulls:')
print(df.fill_null(0))

Performance Tips and Best Practices

To get the most out of Polars: 1) Use lazy evaluation for complex pipelines. 2) Avoid collecting intermediate results. 3) Use appropriate data types (e.g., pl.Int32 instead of pl.Int64 if possible). 4) Use pl.scan_* for file reading. 5) Use streaming for out-of-core processing. 6) Profile with .explain() and pl.Profile. 7) Use column selection to reduce memory. 8) Prefer vectorized operations over loops.

import polars as pl

# Example: lazy pipeline with streaming
q = (
    pl.scan_csv('large.csv')
    .filter(pl.col('value') > 0)
    .groupby('category')
    .agg(pl.col('value').mean())
)

# Execute with streaming
result = q.collect(streaming=True)
print(result)