Eigenvalues and Eigenvectors — Explained with Applications

Eigenvalues and eigenvectors are perhaps the most fundamental concept in applied linear algebra. Av = λv — a matrix A, when applied to its eigenvector v, produces the same vector scaled by eigenvalue λ. Finding the directions that a transformation preserves (up to scaling) reveals the essential structure of the transformation.

Google's original PageRank was the dominant eigenvector of the web's link matrix. PCA computes the eigenvectors of a covariance matrix to find principal components. Quantum states are eigenvectors of Hamiltonian operators. Structural resonant frequencies are eigenvalues of stiffness matrices. Understanding eigenvectors is understanding how to extract the 'essential directions' from a linear system.

Computing Eigenvalues and Eigenvectors

Eigenvalues: solve det(A - λI) = 0 (characteristic polynomial). Eigenvectors: for each λ, solve (A - λI)v = 0.

The characteristic polynomial of an n×n matrix is degree n. For large n, numerical methods (QR algorithm, power iteration) are used. Python's numpy.linalg.eig uses the LAPACK library, which applies the QR algorithm with shifts.

import numpy as np

A = np.array([[4,2],[1,3]], dtype=float)

# NumPy eigendecomposition
eigenvalues, eigenvectors = np.linalg.eig(A)
print('Eigenvalues:', eigenvalues)   # [5, 2]
print('Eigenvectors (columns):\n', eigenvectors)

# Verify: Av = λv
for i in range(len(eigenvalues)):
    v = eigenvectors[:, i]
    lam = eigenvalues[i]
    print(f'λ={lam:.1f}: Av={A@v}, λv={lam*v}')
    print(f'  Match: {np.allclose(A@v, lam*v)}')

Power Iteration — Finding the Dominant Eigenvector

Power iteration finds the largest eigenvalue by repeatedly multiplying by A and normalising. This is the algorithm behind original PageRank.

Algorithm: initialize random v, then loop v = A v / ‖A v‖. Convergence rate depends on the ratio |λ₂/λ₁| — closer to 1 means slower convergence. Rayleigh quotient λ = vᵀAv / vᵀv gives eigenvalue estimate.

def power_iteration(A, n_iter=100, tol=1e-10):
    """Find dominant eigenvector (largest eigenvalue)."""
    import numpy as np
    n = len(A)
    v = np.random.rand(n)
    v /= np.linalg.norm(v)
    for _ in range(n_iter):
        v_new = A @ v
        eigenvalue = np.dot(v_new, v)  # Rayleigh quotient
        v_new /= np.linalg.norm(v_new)
        if np.linalg.norm(v_new - v) < tol:
            return eigenvalue, v_new
        v = v_new
    return eigenvalue, v

A = np.array([[4,2],[1,3]], dtype=float)
lam, v = power_iteration(A)
print(f'Dominant eigenvalue: {lam:.4f}')  # 5.0
print(f'Dominant eigenvector: {v}')

Geometric Intuition: What Does Av = λv Mean?

An eigenvector points in a direction that is invariant under the transformation A — it only gets stretched (or shrunk, or reversed) by λ. This is the 'natural axis' of the matrix.

Example: A = [[2,0],[0,3]]. The eigenvectors are [1,0]ᵀ (λ=2) and [0,1]ᵀ (λ=3). Multiply any point: x-coordinate doubles, y-coordinate triples — the axes are scaled independently. If you pick any other direction, the point rotates AND scales.

For a symmetric matrix, eigenvectors are orthogonal — they form a perfect coordinate system. For a non‑symmetric matrix, eigenvectors are not orthogonal, and the transformation can shear.

import numpy as np
import matplotlib.pyplot as plt

# Diagonal matrix – eigenvectors are axes
A = np.array([[2,0],[0,3]])
v1 = np.array([1,0])
v2 = np.array([0,1])
print("Eigenvectors are standard axes:", A@v1, A@v2)  # [2,0] and [0,3]

# Shear matrix – eigenvectors not orthogonal
B = np.array([[1,1],[0,1]])
eigvals, eigvecs = np.linalg.eig(B)
print("Shear eigenvalues (both 1):", eigvals)
print("Only one eigenvector (not span):", eigvecs[:,0])

Real‑World Applications: Where Eigenvalues Matter in Production

Eigenvalues and eigenvectors are not just theoretical. They power critical algorithms:

• PCA (Principal Component Analysis): eigenvectors of the covariance matrix = principal components; eigenvalues = variance explained. Used for dimensionality reduction, noise filtering, and anomaly detection.
• PageRank: the dominant eigenvector of the web's transition matrix gives page importance. Google solved this for billions of pages using iterative methods.
• Structural Engineering: eigenvalues of the stiffness matrix = natural frequencies. If a bridge's frequency matches wind gust frequency, resonance can collapse it (Tacoma Narrows).
• Quantum Mechanics: eigenstates of the Hamiltonian operator are the possible energy levels. The Schrödinger equation is an eigenvalue problem.
• Graph Theory: eigenvalues of the adjacency matrix indicate community structure, connectivity, and node centrality (spectral clustering).

Numerical Stability and Production Gotchas

Eigendecomposition is numerically tricky. Common pitfalls:

• Near‑zero eigenvalues cause division by zero in whitening or matrix functions.
• Round‑off asymmetry: a matrix that is theoretically symmetric may have tiny asymmetry due to floating‑point ops, leading to complex eigenvalues.
• Defective matrices: not all matrices have a full set of eigenvectors — numpy.linalg.eig still returns a result, but the eigenvectors may be linearly dependent or wildly inaccurate.
• Condition number: if the condition number (max|λ|/min|λ|) is huge (e.g., > 10¹²), small changes in A cause large changes in eigenvectors. The problem is ill‑posed.

import numpy as np
from io.thecodeforge.linalg import safe_eigen # hypothetical robust wrapper

# Ill‑conditioned matrix (nearly singular)
A = np.array([[1, 1], [1, 1 + 1e-12]])
eigvals, eigvecs = np.linalg.eig(A)
print("Eigenvalues (should be ~2 and ~1e-12):", eigvals)
# Residual check
v = eigvecs[:,0]
lam = eigvals[0]
print("Residual norm:", np.linalg.norm(A@v - lam*v))
# Use safe decomposition instead
U, S, Vt = np.linalg.svd(A)
print("Singular values:", S)  # More stable

Frequently Asked Questions