NumPy Shapes Not Aligned: Fix matmul Errors

Your model scores 10,000 rows fine, then dies with ValueError: shapes (128,64) and (32,10) not aligned: 64 (dim 1) != 32 (dim 0). The weights loaded. The features look right. But the inner dimensions — 64 versus 32 — refuse to join, and every retry crashes on the same @ line.

Matmul contracts over one shared axis: (n,k) @ (k,m) yields (n,m), and the two k values must be equal. Feature pipelines break this contract constantly — a dropped embedding column here, a transposed weight file there, a batch axis wandering into the wrong position. The error message states the mismatch plainly, yet developers stare at the outer dims (128 vs 10) instead of the inner pair the message names.

The deeper trap is reaching for the wrong fix: adding broadcasting where matmul belongs, or reshaping blindly until the error moves. You'll learn the inner-dim rule cold, the .shape prints that prove each operand, the broadcast-versus-matmul boundary, and reshape patterns that preserve data order. By the end, every @ in your pipeline will carry a shape assert that fails with context instead of a riddle.

The (n,k)@(k,m) Rule: Inner Dims Must Match Exactly

Matrix multiplication joins along one shared axis: the columns of the left operand (k) must equal the rows of the right (k), producing (n,m). A (4096,64) feature batch times a (64,10) weight matrix yields (4096,10) scores — 4,096 rows each mapped to 10 outputs through 64 shared features. Change either 64 without changing the other and the couplers mismatch: (4096,64) @ (32,10) raises because 64 != 32, exactly as the message states with dim numbers attached.

Read the message inside-out: 'shapes (128,64) and (32,10) not aligned: 64 (dim 1) != 32 (dim 0)' names both shapes, then pinpoints dim 1 of the left (64) against dim 0 of the right (32). Those two numbers are the entire diagnosis — the outer dims (128, 10) are fine and irrelevant. Developers who audit the batch size (128) or output count (10) chase ghosts; the fix always touches one of the two inner values.

Build the reflex: see the error, read the inner pair, ask which side violates the data contract. Features at 64 with weights at 32 means weights are stale (finish the rollout). Weights at 64 with features at 32 means the embedder lags (upgrade the stage). The message can't tell you which is stale — your pipeline's contract can, and the shape assert you'll add encodes it.

import numpy as np

X = np.zeros((128, 64))
W = np.zeros((64, 10))
print("aligned:", (X @ W).shape)  # (128, 10)

stale = np.zeros((32, 10))
try:
    X @ stale
    print("unexpectedly joined")
except ValueError as exc:
    print("misaligned:", exc)
assert X.shape[1] == W.shape[0], (X.shape, W.shape)
print("gate: inner dims", X.shape[1], "==", W.shape[0])

Print .shape First: Evidence Before Reshaping

Every shape bug starts with the same two prints: the operands' .shape values and the contract they should satisfy. Log A.shape, B.shape, and the boolean A.shape[1] == B.shape[0] before the @ — in the function, not in a notebook afterward. The log line turns 'shapes not aligned' from a riddle into a record: (4096,64) vs (32,10) with the contract (64,10) stated beside it names the stale weights file without further digging.

Trace shapes through the pipeline, not just at the crash. Embedding output width, normalization's preserved width, batching's added leading axis, weight file's stored shape — print each stage's output shape on the probe batch and the drift's location appears. The incident's drift sat between the embedder (64) and the weight file (32); stage-wise prints would have shown the embedder agreement ending exactly at the loader.

Make the prints permanent, not debugging graffiti. A shape-logging line per pipeline stage costs microseconds on a 100-row probe and pays back on every future width change. When the next embedder upgrade lands, the probe log shows the new width propagating stage by stage — or stopping dead at the stale file, with its shape quoted.

Store the probe shapes alongside the run record so post-incident review compares expected versus actual widths without rerunning anything.

import numpy as np

def scored(X, W):
    print(f"shapes: X{X.shape} W{W.shape} inner-ok={X.shape[1] == W.shape[0]}")
    assert X.shape[1] == W.shape[0], f"inner mismatch: {X.shape} vs {W.shape}"
    return X @ W

X = np.zeros((256, 64))
W = np.load("/tmp/W64.npy") if __import__("os").path.exists("/tmp/W64.npy") else np.zeros((64, 10))
print("scores:", scored(X, W).shape)
try:
    scored(X, np.zeros((32, 10)))
except AssertionError as exc:
    print("fast fail with context:", exc)

Broadcasting vs Matmul: Stretching Is Not Joining

Broadcasting (*) aligns trailing dimensions elementwise, stretching size-1 axes to match — a (4,3) frame plus a (3,) row vector adds the vector to every row. Matmul (@) contracts the shared inner axis with sums — (4,3) @ (3,2) yields (4,2) through 3-wide dot products. Same operands, different verbs: one scales and shifts, the other projects and mixes. Swapping them produces either 'shapes not aligned' (matmul given broadcast shapes) or silently wrong values (broadcast given matmul intent).

The confusion peaks on scaling ops: multiplying features by per-column weights is X w with w shaped (k,) — broadcast, no contraction. Projecting to outputs is X @ W with W shaped (k,m) — contraction to m columns. Code that writes X W for a (128,64)/(64,10) pair broadcasts nothing meaningful and errors; code that writes X @ w for scaling contracts where it should stretch. Name the intent first (scale or project), then pick the operator.

Audit by operator: grep @ lines and confirm each pair satisfies the inner rule; grep * lines on 2D operands and confirm the trailing dims broadcast legally. The two checks take minutes and separate the genuinely misaligned (fix the width) from the mis-verbed (fix the operator) — different bugs with different patches.

import numpy as np

X = np.arange(12, dtype=float).reshape(4, 3)
w = np.array([10.0, 100.0, 1000.0])
print("broadcast scale row:", (X * w)[0].tolist())  # stretch (3,) over rows
W = np.ones((3, 2))
print("matmul project:", (X @ W).shape)  # contract k=3 -> (4, 2)
try:
    X * np.ones((3, 2))  # trailing dims 4,3 vs 3,2: no broadcast
    print("unexpected broadcast")
except ValueError as exc:
    print("broadcast rejected:", str(exc)[:70])

reshape, transpose, squeeze: Change Layout Without Scrambling Data

Reshape changes the shape tuple while preserving C-order data sequence by default: arange(12).reshape(3,4) then .reshape(2,6) keeps values 0-11 marching row by row. The -1 placeholder infers one axis (X.reshape(-1, 64) keeps 64-wide rows however many batches arrived), which is ideal for variable batch sizes. Transpose swaps axes (B.T turns (32,10) into (10,32)) for orientation fixes, and squeeze/expand_dims add or drop size-1 axes that batching introduces.

Each tool answers a different diagnosis. Wrong count of elements (can't reshape 4,096x64 into 32-wide) means the data itself is stale — no layout trick fixes a width split, so finish the rollout. Right elements, wrong orientation (weights saved as (10,64)) means one .T in the loader. Extra leading batch axis on one side means squeeze or indexed selection before the @. Applying transpose to a width split just moves the error; applying reshape to an orientation issue scrambles order silently.

Verify every layout change on a tiny labeled array first: reshape arange(12) and check the first row's values survived in order, transpose a marked matrix and confirm the mark moved correctly. The 30-second micro-test on toy data prevents the order-scrambling 'fix' that passes shapes and corrupts scores.

import numpy as np

x = np.arange(12).reshape(3, 4)
print("reshape -1:", x.reshape(-1, 2).shape)  # (6, 2), order kept
print("first row kept:", x.reshape(2, -1)[0][:4].tolist())
W = np.zeros((10, 64))  # saved transposed
print("loader fix:", (np.zeros((256, 64)) @ W.T).shape)  # (256, 10)
batch = np.zeros((1, 64))
print("squeeze batch:", batch.squeeze(axis=0).shape)  # (64,)

Batched @ on 3D Tensors: Leading Axes Broadcast, Last Two Contract

For stacked batches, X @ W follows one rule: leading axes broadcast, the last two axes matmul. A (32,128,64) batch times (64,10) weights yields (32,128,10) — 32 sequences each projected independently. A (32,128,64) times (32,64,10) per-batch weights also works via leading-axis broadcast. But a (32,64,128) activation times (64,10) fails: the last two axes (64,128) vs (64,10) misalign because the sequence axis wandered into the contraction slot.

Fix orientation with transpose or moveaxis before the @, and state which axes contract in a comment: scores = X @ W # (B,T,k)@(k,m)->(B,T,m). The comment is load-bearing — the next reader sees the intended contraction without reverse-engineering axis order, and the reviewer checks the transpose against the comment instead of guessing.

Probe 3D shapes on CPU with zeros before GPU time: print the 3-tuple pair and the result tuple on a (2,4,64)/(64,10) miniature. The miniature fails in milliseconds with the same message the 1.2M-row run would produce after warmup — the cheapest possible reproduction of every batched variant of this error.

Keep one miniature per 3D call site in the test suite so axis regressions fail at merge time instead of after GPU scheduling.

import numpy as np

X = np.zeros((32, 128, 64))
W = np.zeros((64, 10))
print("batched:", (X @ W).shape)  # (32, 128, 10)
wandering = np.zeros((32, 64, 128))
try:
    wandering @ W
    print("unexpected join")
except ValueError as exc:
    print("batch in slot:", str(exc)[:80])
fixed = wandering.transpose(0, 2, 1)  # (32, 128, 64)
print("transposed:", (fixed @ W).shape)

Harden the Pipeline: Probes, Pins, and Freshness Gates

Close the class with three durable controls. First, a 100-row probe before GPU warmup: run one miniature batch through embedder-to-scores, asserting each stage's width and the final inner pair, paging on any gap. The probe fails in seconds where the full run failed after 12 minutes of warmup — same message, 1% of the cost. Second, pin the embedder and weight versions as a tested pair in the deploy manifest, verified by a script asserting W.shape against the embedder's declared width rather than comparing version strings.

Third, gate downstream on freshness: ranking tables assert row counts and max-score-date after every run, paging on emptiness instead of serving fallbacks silently for 5 hours. The homepage review that caught the incident becomes a monitor that catches it at 1:13 a.m.

Log shapes with the run record — embedder width, weight shape, batch count, rows scored — so the next mismatch arrives with its evidence attached. Width changes are normal evolution; unprobed width changes are incidents. The three controls convert every future split into a seconds-fast page with both shapes quoted.

Review the width log after each model release even when green; slow drift across releases is visible there weeks before it becomes a mismatch.

import numpy as np

def probe(embed_width, W, n=100):
    X = np.zeros((n, embed_width))
    ok = X.shape[1] == W.shape[0]
    print(f"probe: X{X.shape} W{W.shape} inner-ok={ok}")
    if not ok:
        raise SystemExit(f"width split: embed {embed_width} vs weights {W.shape}")
    return (X @ W).shape

print("paired:", probe(64, np.zeros((64, 10))))
try:
    probe(64, np.zeros((32, 10)))
except SystemExit as exc:
    print("gate paged:", exc)