Pandas DataFrame — Merge Explosion from Duplicate Keys

Every data-driven Python project — from analysing sales figures to training a machine-learning model — eventually hits the same wall: raw data is messy, scattered, and weirdly shaped. You've got CSV files, database query results, and API responses all telling different stories in different formats. Pandas DataFrames exist precisely to be the one place where all that chaos comes together and starts making sense. They're the reason data scientists can do in five lines what would take fifty lines of raw Python loops.

What a DataFrame Actually Is (and Why It's Not Just a List of Lists)

A lot of beginners assume a DataFrame is just a fancy 2D list. It isn't — and that distinction matters the moment your data gets real. A DataFrame is a labelled, column-typed, index-aware structure. Every column has a name and a dtype. Every row has an index. That metadata is what lets you write df['revenue'].sum() instead of sum(row[3] for row in my_list) — it's not just cleaner syntax, it's a fundamentally different contract with your data.

When you load a CSV or query a database, Pandas infers the type of each column automatically. Integers stay integers, strings become object dtype, dates can become datetime64. This typing means Pandas can delegate heavy number-crunching to NumPy under the hood — which is why operations on millions of rows stay fast.

The index is another superpower people underuse. By default it's just 0, 1, 2… but set it to something meaningful — a customer ID, a timestamp — and suddenly lookups, joins, and time-series resampling become trivially simple.

import pandas as pd

# Build a small sales dataset manually — in real life this comes from a CSV or DB
sales_data = {
    'order_id':    [1001, 1002, 1003, 1004],
    'customer':    ['Alice', 'Bob', 'Alice', 'Carol'],
    'product':     ['Laptop', 'Mouse', 'Keyboard', 'Monitor'],
    'revenue':     [1200.00, 25.50, 89.99, 349.00],
    'order_date':  pd.to_datetime(['2024-01-15', '2024-01-17', '2024-01-17', '2024-01-20'])
}

df = pd.DataFrame(sales_data)

# Set order_id as the index — now we can look up orders by ID, not by position
df = df.set_index('order_id')

print(df)
print("\n--- Column dtypes ---")
print(df.dtypes)
print("\n--- Quick summary stats for revenue ---")
print(df['revenue'].describe())

Selecting Data the Right Way — loc, iloc, and When Each One Saves You

Data selection is where most intermediate developers have a messy relationship with Pandas. They mix bracket notation, loc, and iloc interchangeably until something breaks and they don't know why. Let's fix that.

loc is label-based. You give it index labels and column names — the human-readable identifiers. iloc is position-based. You give it integer row and column positions — like a zero-indexed 2D array. They look similar but they're entirely different tools.

The practical rule: use loc when you're thinking about your data ('give me Alice's orders'), and use iloc when you're thinking about structure ('give me the first 10 rows for a preview'). The bracket shorthand df['column'] is fine for single column access, but the moment you're selecting rows AND columns together, always use loc or iloc explicitly. Ambiguity here is a direct path to the dreaded SettingWithCopyWarning.

Boolean indexing — filtering with a condition — plugs into loc perfectly and is how you'll do the majority of real-world filtering.

import pandas as pd

sales_data = {
    'order_id':   [1001, 1002, 1003, 1004, 1005],
    'customer':   ['Alice', 'Bob', 'Alice', 'Carol', 'Bob'],
    'product':    ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Webcam'],
    'revenue':    [1200.00, 25.50, 89.99, 349.00, 79.95],
    'region':     ['North', 'South', 'North', 'West', 'South']
}

df = pd.DataFrame(sales_data).set_index('order_id')

# loc: label-based — give me Alice's rows, and only show product and revenue columns
alice_orders = df.loc[df['customer'] == 'Alice', ['product', 'revenue']]
print("--- Alice's orders (loc with boolean mask) ---")
print(alice_orders)

# iloc: position-based — give me rows 0 and 1, columns 0 to 2 (slice is exclusive at end)
first_two_rows_preview = df.iloc[0:2, 0:3]
print("\n--- First 2 rows preview (iloc) ---")
print(first_two_rows_preview)

# Chained boolean conditions — high-value orders in North region
high_value_north = df.loc[(df['revenue'] > 100) & (df['region'] == 'North'), :]
print("\n--- High-value North orders ---")
print(high_value_north)

# Single value lookup by known index label — fast O(1) access
order_1004_revenue = df.loc[1004, 'revenue']
print(f"\nRevenue for order 1004: £{order_1004_revenue}")

Merge Join Types — Inner, Left, Right, Outer with Visual Reference

When merging DataFrames with pd.merge(), the how parameter determines which rows survive. Think of two circles in a Venn diagram: the left DataFrame is one circle, the right DataFrame is the other. The overlap is where keys match.

Inner join (how='inner') keeps only rows where the key exists in BOTH DataFrames — the intersection. Use this when you only want complete records: orders that have matching customer details, transactions with valid product IDs.

Left join (how='left') keeps ALL rows from the left DataFrame, and adds columns from the right where keys match. Missing matches become NaN. This is your default for enrichment: "keep every order, attach customer names if available".

Right join (how='right') is the mirror — keep all rows from the right. Less common, but useful when the right table is your master list (e.g., all products) and you want to see which ones had activity.

Outer join (how='outer') keeps all rows from both sides. Missing matches become NaN on either side. Use for complete audits: all customers AND all orders, even orphaned records.

The critical production insight: if duplicate keys exist on the "one" side of what you assume is a one-to-many relationship, Pandas silently creates a Cartesian product. Always verify uniqueness before merging.

import pandas as pd

left = pd.DataFrame({'key': ['A', 'B', 'C'], 'left_val': [1, 2, 3]})
right = pd.DataFrame({'key': ['B', 'C', 'D'], 'right_val': [4, 5, 6]})

# Inner — only B and C (keys in both)
inner = pd.merge(left, right, on='key', how='inner')
print("Inner:\n", inner.to_string(index=False))

# Left — A, B, C (all left, D missing from right)
left_join = pd.merge(left, right, on='key', how='left')
print("\nLeft:\n", left_join.to_string(index=False))

# Right — B, C, D (all right, A missing from left)
right_join = pd.merge(left, right, on='key', how='right')
print("\nRight:\n", right_join.to_string(index=False))

# Outer — A, B, C, D (all rows from both)
outer = pd.merge(left, right, on='key', how='outer')
print("\nOuter:\n", outer.to_string(index=False))

GroupBy and Aggregation — Turning Raw Rows Into Business Insights

This is where DataFrames stop being a data container and start being an analytical tool. groupby() splits your DataFrame into logical buckets — by customer, by region, by month — and then you apply a calculation to each bucket. The result is a summarised DataFrame that answers real questions.

Under the hood, groupby uses a split-apply-combine strategy: it splits the data by the group key, applies your aggregation function independently to each group, then combines the results back into a single structure. Understanding this model helps you predict what the output will look like before you run it.

The real power shows up with agg(), which lets you apply multiple different functions to different columns in a single pass — no need to run five separate groupby calls. For reporting and dashboards, this pattern is indispensable.

You can also chain groupby with transform() when you need to add a group-level statistic back as a new column on every row — for example, adding each customer's total spend alongside their individual orders.

import pandas as pd

orders = pd.DataFrame({
    'order_id':  [1001, 1002, 1003, 1004, 1005, 1006],
    'customer':  ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'],
    'region':    ['North', 'South', 'North', 'West', 'South', 'North'],
    'revenue':   [1200.00, 25.50, 89.99, 349.00, 79.95, 430.00],
    'units':     [1, 2, 1, 1, 3, 2]
})

# --- Simple groupby: total revenue per customer ---
revenue_by_customer = orders.groupby('customer')['revenue'].sum().reset_index()
revenue_by_customer.columns = ['customer', 'total_revenue']
print("--- Revenue by customer ---")
print(revenue_by_customer)

# --- Multi-metric aggregation: different stats per column in one pass ---
summary = orders.groupby('region').agg(
    total_revenue=('revenue', 'sum'),    # sum of revenue per region
    order_count=('order_id', 'count'),   # how many orders per region
    avg_units=('units', 'mean')          # average units per order per region
).reset_index()
print("\n--- Regional summary ---")
print(summary)

# --- transform: add each customer's total spend as a new column on every row ---
# Useful for calculating what percentage of a customer's total each order represents
orders['customer_total'] = orders.groupby('customer')['revenue'].transform('sum')
orders['pct_of_customer_total'] = (orders['revenue'] / orders['customer_total'] * 100).round(1)
print("\n--- Orders with customer share % ---")
print(orders[['order_id', 'customer', 'revenue', 'customer_total', 'pct_of_customer_total']])

Aggregation Patterns — agg() vs transform() vs apply()

Pandas gives you three tools for grouped operations, and choosing the wrong one leads to slow code or confusing output. Here's the decision tree.

groupby().agg() reduces rows — one output row per group. Use this when you want a summary table: total sales per region, average order value per customer. The output shape is (number_of_groups, number_of_aggregated_columns). Always follow with .reset_index() to promote the group key back to a regular column.

groupby().transform() keeps the original row count — it broadcasts group-level results back to every row in the original DataFrame. Use this when you need to add a group statistic as a new column: each order gets the customer's total spend, each product gets its category average. Perfect for calculating percentages, z-scores, or normalised values within groups.

groupby().apply() is the escape hatch — it passes each group as a DataFrame to a custom function and stitches the results back together. It's flexible but slow because it can't use vectorised operations. Use it only when agg() and transform() can't express your logic (e.g., fitting a separate regression model per group).

The performance difference matters at scale: agg() and vectorised transform() run at C speed; apply() with Python functions runs at Python speed, often 100x slower.

import pandas as pd

df = pd.DataFrame({
    'customer': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'],
    'region': ['North', 'South', 'North', 'West', 'South', 'North'],
    'revenue': [1200, 25, 90, 349, 80, 430],
    'units': [1, 2, 1, 1, 3, 2]
})

# agg() — reduces rows: one row per customer
print("--- agg(): one row per customer ---")
customer_summary = df.groupby('customer').agg(
    total_revenue=('revenue', 'sum'),
    avg_units=('units', 'mean'),
    order_count=('revenue', 'count')
).reset_index()
print(customer_summary)

# transform() — keeps shape: add customer total to each row
print("\n--- transform(): customer total on every row ---")
df['customer_total'] = df.groupby('customer')['revenue'].transform('sum')
df['pct_of_customer'] = (df['revenue'] / df['customer_total'] * 100).round(1)
print(df[['customer', 'revenue', 'customer_total', 'pct_of_customer']])

# apply() — custom complex logic per group
print("\n--- apply(): custom function per group ---")
def revenue_range(group):
    return pd.Series({'min_revenue': group['revenue'].min(),
                      'max_revenue': group['revenue'].max(),
                      'revenue_span': group['revenue'].max() - group['revenue'].min()})

range_by_customer = df.groupby('customer').apply(revenue_range).reset_index()
print(range_by_customer)

Merging DataFrames — The SQL Join You Already Know, In Python

Real-world data is rarely in one place. You've got an orders table, a customers table, and a products table — all separate. Merging DataFrames is how you bring them together, and it maps directly onto SQL JOINs you may already know.

pd.merge() is the workhorse. You specify the two DataFrames, the key column(s) to join on, and the type of join: inner (only matching rows), left (all rows from the left, matched where possible), right, or outer (all rows from both). Understanding which join type to use is more important than memorising the syntax.

A common real-world pattern is a left join against a reference table — you have a list of orders and want to enrich each one with customer details, keeping all orders even if a customer record is somehow missing.

After merging, always check the row count. If you expected 500 rows and got 2000, you've hit a one-to-many relationship you didn't account for — a classic data explosion caused by duplicate keys on the right-hand DataFrame.

import pandas as pd

# Table 1: raw orders — just IDs, no names
orders_df = pd.DataFrame({
    'order_id':   [1001, 1002, 1003, 1004, 1005],
    'customer_id':[201,  202,  201,  203,  204],    # 204 has no customer record yet
    'revenue':    [1200.00, 25.50, 89.99, 349.00, 79.95]
})

# Table 2: customer reference data — note customer 204 is missing
customers_df = pd.DataFrame({
    'customer_id': [201, 202, 203],
    'customer_name': ['Alice', 'Bob', 'Carol'],
    'tier': ['Gold', 'Silver', 'Gold']
})

# Inner join: only orders where we have a matching customer record
inner_merged = pd.merge(orders_df, customers_df, on='customer_id', how='inner')
print("--- Inner join (drops order 1005 — no customer record) ---")
print(inner_merged)

# Left join: keep ALL orders, fill missing customer info with NaN
left_merged = pd.merge(orders_df, customers_df, on='customer_id', how='left')
print("\n--- Left join (keeps order 1005 with NaN for missing customer) ---")
print(left_merged)

# Practical check: did we unexpectedly gain or lose rows?
print(f"\nOriginal orders: {len(orders_df)} rows")
print(f"After left join:  {len(left_merged)} rows")
print(f"Rows with no customer match: {left_merged['customer_name'].isna().sum()}")

Handling Missing Data — The Patterns That Save Your Pipeline

Real-world data is never clean. Missing values creep in from JSON nulls, empty cells in CSVs, and incomplete database records. Pandas represents missing data as NaN (for numeric) and None (for object columns), but they behave oddly if you're not careful.

The most common mistake is trying to check df['col'] == None or df['col'] == float('nan') — both always return False because NaN is not equal to itself by IEEE 754 definition. You must use df['col'].isna() or pd.isnull() instead.

Three strategies for handling missing data: 1. Drop rows with df.dropna() — cruel but fast. 2. Fill with a constant or computed value using df.fillna(). Common choices: column mean, median, or a placeholder like 0 or 'Unknown'. 3. Forward or backward fill (method='ffill' or 'bfill') for time series — carry the last known value forward.

The right choice depends on the business context. Dropping rows with missing key identifiers is fine. Dropping 40% of your data because a single optional column has gaps is not.

import pandas as pd

# Messy sales data with missing values
messy = pd.DataFrame({
    'order_id': [1001, 1002, 1003, 1004, 1005],
    'customer': ['Alice', 'Bob', None, 'Carol', 'Alice'],
    'revenue':  [1200.00, None, 89.99, 349.00, None],
    'region':   ['North', 'South', None, 'West', 'North']
})

print("--- Raw messy data ---")
print(messy)

# Detect missing
print("\n--- Missing value counts ---")
print(messy.isna().sum())

# Drop rows where customer is missing (critical field)
cleaned = messy.dropna(subset=['customer'])
print("\n--- After dropping rows with missing customer ---")
print(cleaned)

# Fill missing revenue with column median (reasonable imputation)
median_revenue = messy['revenue'].median()
messy['revenue'] = messy['revenue'].fillna(median_revenue)
print(f"\n--- Filled missing revenue with median ({median_revenue}) ---")
print(messy[['order_id', 'revenue']])

# Forward fill region (carry last known region forward, if ordering makes sense)
messy['region'] = messy['region'].fillna(method='ffill')
print("\n--- After forward filling region ---")
print(messy[['order_id', 'region']])

Handling Missing Data — A Decision Framework

Missing data (NaN or None) is inevitable in real-world pipelines. The right strategy depends on the column's role and the percentage of missingness. Here's a framework used in production data pipelines.

Primary key columns (customer_id, order_id) — if missing, you cannot identify the record. Drop these rows immediately with dropna(subset=['key_column']). Never impute primary keys.

Numeric features with <5% missing — dropping rows has minimal impact. Use dropna(subset=['numeric_col']).

Numeric features with 5-30% missing — impute with median (robust to outliers) or mean (if distribution is symmetric). df['col'].fillna(df['col'].median())

Numeric features with >30% missing — create an indicator column first (df['col_missing'] = df['col'].isna()), then impute with 0 or median. This preserves the signal that the value was originally missing.

Categorical features — fill with a placeholder like 'Unknown' or 'MISSING'. df['category'].fillna('Unknown')

Time-series data — forward fill (method='ffill') carries the last known value forward. Only valid if rows are meaningfully ordered.

The golden rule: never drop or impute without quantifying the impact. Run df.isna().sum() / len(df) * 100 to see percentages before deciding.

import pandas as pd

# Sample messy data
sales = pd.DataFrame({
    'order_id': [1001, 1002, 1003, 1004, 1005, 1006],
    'customer_id': [201, 202, None, 203, 204, 202],
    'revenue': [1200.00, None, 89.99, 349.00, None, 430.00],
    'region': ['North', 'South', None, 'West', None, 'North']
})

print("--- Missing percentages ---")
print((sales.isna().sum() / len(sales) * 100).round(1))

# 1. Primary key missing — drop rows
print("\n--- After dropping rows with missing customer_id ---")
sales_clean = sales.dropna(subset=['customer_id'])
print(f"Rows dropped: {len(sales) - len(sales_clean)}")

# 2. Numeric with 5-30% missing — median imputation
print("\n--- Revenue: median imputation ---")
median_rev = sales['revenue'].median()
sales['revenue_filled'] = sales['revenue'].fillna(median_rev)
print(f"Median revenue: {median_rev}")

# 3. Categorical with >10% missing — placeholder + indicator
print("\n--- Region: placeholder + indicator ---")
sales['region_missing'] = sales['region'].isna()
sales['region_filled'] = sales['region'].fillna('UNKNOWN')
print(sales[['order_id', 'region', 'region_missing', 'region_filled']])

# 4. Time-series example (forward fill)
ts = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=5),
    'price': [100, None, None, 115, None]
})
ts['price_ffill'] = ts['price'].fillna(method='ffill')
print("\n--- Time-series forward fill ---")
print(ts)

Creating DataFrames From Scratch — Stop Wrapping Lists in Lists

You don't build a DataFrame by hand every day. But when you do — maybe you're bootstrapping test fixtures, parsing a weird API response, or reconstructing corrupted data — you need to know which constructor pattern won't silently break your types.

The safest path: a dictionary of lists or arrays. Keys become column names, values become columns. Pandas aligns them by index. If you pass a list of lists without column names, you get default integer columns. That's fine for throwaway analysis. It's a nightmare for production code where column names carry business meaning.

NumPy arrays work too, and they're faster for numeric data. But remember: arrays are homogeneous. Mix ints and strings in one array and NumPy coerces everything to a common dtype — usually object, which kills performance. If you need mixed types, stick with lists per column.

The shape rule is non-negotiable: every column must have the same length. Pandas won't guess. It will error or silently pad with NaN, and that NaN is a landmine waiting for a null-safe query to blow up your ETL.

// io.thecodeforge — python tutorial

import pandas as pd
import numpy as np

# This is the pattern you want: dict of lists
orders = {
    "order_id": [1001, 1002, 1003],
    "customer": ["Acme Corp", "Globex", "Initech"],
    "amount_usd": [4200.50, 1850.00, 9990.75],
    "items": [4, 2, 11]
}

df = pd.DataFrame(orders)
print(df)
print(df.dtypes)

# Bad: list of lists with no column names
bad = pd.DataFrame([
    [1001, "Acme Corp", 4200.50],
    [1002, "Globex", 1850.00]
])
print(bad)
print(bad.dtypes)  # columns are 0, 1, 2

Column Selection — The One-Liner That Makes or Breaks a Pipeline

Selecting columns looks trivial. It's not. The difference between df['col'] and df.col is the difference between a clear transaction log and a silent bug that surfaces in production three weeks later.

Rule one: use bracket notation. Always. df.col fails when the column name has spaces, starts with a number, or collides with a DataFrame method (think 'count', 'sum', 'mean'). You won't get a warning — Pandas returns the method object instead of your data. Debugging that is a special kind of waste.

To select multiple columns, pass a list: df[['col_a', 'col_b']]. This returns a DataFrame. Single bracket with a string returns a Series. If your downstream code expects a DataFrame, a single-column selection will break .shape and method chaining. Be explicit.

Filtering columns by dtype? pd.DataFrame.select_dtypes(). Production log files often have a mix of int and object columns. That method is your scalpel. Use it to isolate numeric columns for aggregation or date columns for resampling.

// io.thecodeforge — python tutorial

import pandas as pd

df = pd.DataFrame({
    "sale_id": [1, 2, 3],
    "product name": ["widget", "gadget", "doodad"],  # note the space
    "count": [10, 25, 15],
    "price": [9.99, 14.99, 7.49]
})

# Correct
print(df["product name"])
print(type(df["product name"]))

# Silent bug: df.count returns method, not column
print(df.count)
print(type(df.count))

# Select multiple columns => DataFrame
print(df[["sale_id", "price"]])
print(type(df[["sale_id", "price"]]))

# Filter by dtype
numeric = df.select_dtypes(include=["int64", "float64"])
print(numeric)

Filtering Rows — Why Loops Get You Fired and .query() Gets Promoted

You need a subset of rows. New devs write a loop with an if statement. That's how you spend CPU cycles and lose your cool. Pandas gives you Boolean indexing and the .query() method. Boolean indexing means you pass a Series of True/False values inside brackets. It's fast because NumPy handles the vectorized operation below the surface. .query() is even cleaner for complex conditions — it accepts a string expression and lets you reference columns directly without repeating the DataFrame name. Production code rarely uses loops for filtering. If you see one, it's a code smell. The why: vectorized operations run at C speed. Python loops run at, well, Python speed. Your data pipeline shouldn't wait for a for-loop to finish while your boss stares at the dashboard.

// io.thecodeforge — python tutorial

import pandas as pd

df = pd.DataFrame({
    'city': ['NYC', 'LA', 'Chicago', 'Houston', 'Phoenix'],
    'temp_f': [72, 84, 55, 90, 101],
    'humidity': [65, 45, 70, 80, 20]
})

# Boolean indexing — fast, explicit
hot_cities = df[df['temp_f'] > 85]
print(hot_cities)

# .query() — readable for complex conditions
dry_hot = df.query('temp_f > 85 and humidity < 50')
print(dry_hot)

Writing to CSV — The Hidden Ways Your Data Gets Corrupted

You built the DataFrame. You cleaned it. Now you need to ship it. .to_csv() looks innocent — one line, done. Wrong. Default parameters will silently wreck your output. Index column? It's there by default, and nobody asked for it. Encoding? Default is 'utf-8', but your production system expects 'utf-8-sig' so Excel doesn't mangle the BOM. Float precision? Default keeps 6 decimal places for every number, bloating your CSV. The WHY: CSV is a text format. Every Python object gets converted to a string. If your column contains lists, dicts, or None values, .to_csv() writes the literal Python representation, not the data you intended. Always specify index=False, encoding='utf-8-sig' for Excel compatibility, and float_format if you need precision control. Test the output by reading it back immediately — that catches 90% of corruption.

// io.thecodeforge — python tutorial

import pandas as pd

df = pd.DataFrame({
    'product': ['Widget A', 'Widget B'],
    'price': [19.9900001, 29.9900002],
    'tags': [['metal', 'blue'], ['plastic', 'red']]
})

# Wrong: includes index, float overflow, list as string
wrong_path = '/tmp/products_bad.csv'
df.to_csv(wrong_path)

# Right: clean output for downstream systems
right_path = '/tmp/products_good.csv'
df.to_csv(right_path, index=False, encoding='utf-8-sig', float_format='%.2f')

# Verify immediately
print(pd.read_csv(right_path))

Inserting and Deleting Data — Rows, Columns, and Arithmetic

You cannot build robust data pipelines without modifying DataFrame structure. Inserting or deleting rows and columns is not a hack — it is a controlled operation. For columns, use df['new_col'] = values to append; use df.drop('col', axis=1) to remove. For rows, pd.concat with new DataFrames appends rows, while df.drop(index=row_index) removes them. The loc accessor also allows row insertion via slicing. Always prefer column assignment over loops — it is vectorized and safe. Arithmetic operations like df['a'] * 2 apply element-wise, and you can broadcast across columns or rows using df.add(1, axis=0). Determine data statistics using df.describe(), df.mean(), df.corr(), and df.value_counts(). These methods reveal central tendency, distribution, and relationships. Never mutate the original without a copy unless you intend side effects; use inplace=False explicitly to avoid silent corruption. Understanding insertion, deletion, arithmetic, and statistics transforms you from a script kiddie into an engineer who controls data shape and meaning.

// io.thecodeforge — python tutorial
import pandas as pd

df = pd.DataFrame({'x': [1, 2], 'y': [3, 4]})

# Insert column
df['z'] = [5, 6]

# Delete row
df = df.drop(index=0)

# Arithmetic
df['x_plus_y'] = df['x'] + df['y']

# Statistics
stats = df.describe()
corr = df.corr()

print(stats)
print(corr)

Related Post — Building on Pandas DataFrames

DataFrame manipulation does not stop at merge and groupby. This tutorial extends the series with practical mutation patterns. If you skipped earlier posts, revisit 'Creating DataFrames From Scratch' and 'Filtering Rows' to ground yourself. Inserting rows and columns builds directly on the column selection and filtering patterns you already know. Arithmetic operations are the engine behind feature engineering — they let you derive new dimensions from existing ones. Statistics methods like df.describe() and df.corr() are your first diagnostic tools before any merge or aggregation. They reveal null counts, skew, and linear relationships that break pipelines silently. This post bridges the gap between static DataFrames and dynamic transformations. Apply these patterns after merging to clean and enrich results. Pair insertion with apply() for conditional logic. The goal is not just to code — it is to control data state transitions with precision. Treat every modification as a transaction: test, verify shape, then proceed.

// io.thecodeforge — python tutorial
import pandas as pd

df = pd.DataFrame({'a': [10, 20], 'b': [30, 40]})

# Row insert via concat
new_row = pd.DataFrame({'a': [99], 'b': [88]})
df = pd.concat([df, new_row], ignore_index=True)

# Column arithmetic
df['ratio'] = df['a'] / df['b']

# Stats
mean_vals = df.mean()

print(df)
print(mean_vals)