Actor-Critic Methods: From Policy Gradients to Production RL

Reinforcement learning has seen a paradigm shift from tabular methods to deep neural networks, but the core challenge remains: how do you learn a policy that maximizes cumulative reward without suffering from high variance gradient estimates? Pure policy gradient methods like REINFORCE are unbiased but notoriously noisy, requiring massive sample sizes. Actor-critic methods emerged as the production-ready answer, blending the stability of value-based learning with the flexibility of policy gradients. In 2026, actor-critic variants—A2C, A3C, PPO, SAC, and TD3—power everything from robotics control to recommendation systems, yet many practitioners still struggle with the subtle implementation details that separate a working agent from a brittle one. This article dissects the theory, then goes deep into the engineering realities: gradient clipping, target network synchronization, reward normalization, and the silent bugs that kill convergence.

The Policy Gradient Problem: Variance and the Need for a Baseline

Policy gradient methods optimize the expected return J(θ) = E[Σ γ^t r_t] by ascending the gradient ∇J(θ) = E[∇log π_θ(a|s) * Ψ]. The choice of Ψ directly determines gradient estimator variance. The vanilla REINFORCE algorithm uses the full Monte Carlo return Ψ = Σ γ^k r_{t+k}, which is unbiased but suffers from extremely high variance because it accumulates noise from every future timestep. In practice, this means REINFORCE requires orders of magnitude more samples to converge—often 10x to 100x more episodes than actor-critic variants on the same task.

The core insight is that we can reduce variance without introducing bias by subtracting a baseline b(s) from the return: Ψ = (Σ γ^k r_{t+k}) - b(s). The baseline must be independent of the action at time t. The optimal baseline is the state-value function V^π(s), because it captures the expected return from state s, leaving only the advantage of the chosen action. This reduces gradient variance by roughly the variance of the returns themselves—often a factor of 2-10 in practice, depending on reward sparsity.

Why does this work? The policy gradient theorem shows that any baseline that does not depend on the action leaves the expectation unchanged: E[∇log π * b(s)] = 0. So we can freely subtract any function of state. The variance reduction comes from removing the common-mode noise shared across all actions. In high-dimensional action spaces (e.g., continuous control with 10+ DoF), this variance reduction is not optional—it's the difference between convergence and divergence.

Mathematically, the gradient estimate becomes ∇J(θ) ≈ (1/N) Σ ∇log π_θ(a_i|s_i) * (R_i - V_φ(s_i)), where V_φ is a learned baseline. This is the foundation of actor-critic: the critic learns V_φ to serve as the baseline, while the actor optimizes the policy using the reduced-variance signal. The bias-variance tradeoff is now controlled by how well V_φ approximates the true value function—a regression problem we can solve with standard supervised learning.

import numpy as np

def reinforce_gradient(log_probs, returns):
    # Vanilla REINFORCE: high variance
    grads = log_probs * returns
    return np.mean(grads, axis=0)

def reinforce_with_baseline(log_probs, returns, baseline):
    # REINFORCE with learned baseline: lower variance
    advantages = returns - baseline
    grads = log_probs * advantages
    return np.mean(grads, axis=0)

# Simulate: 1000 episodes, 10 actions, returns ~ N(5, 10)
np.random.seed(42)
log_probs = np.random.randn(1000, 10) * 0.1
returns = np.random.randn(1000) * 3 + 5
baseline = np.mean(returns)  # simple constant baseline

vanilla_grad = reinforce_gradient(log_probs, returns)
baseline_grad = reinforce_with_baseline(log_probs, returns, baseline)

print(f"Vanilla grad variance: {np.var(vanilla_grad):.4f}")
print(f"Baseline grad variance: {np.var(baseline_grad):.4f}")
print(f"Variance reduction: {100 * (1 - np.var(baseline_grad)/np.var(vanilla_grad)):.1f}%")

Actor-Critic Architecture: Policy Network and Value Function

The actor-critic architecture decouples policy optimization into two neural networks: the actor π_θ(a|s) outputs a probability distribution over actions, and the critic V_φ(s) estimates the expected return from state s. The actor is trained via policy gradient using the critic's output as a baseline (or advantage), while the critic is trained via TD learning to minimize the mean squared error between its predictions and observed returns. This dual-network design is the standard for modern deep RL.

In practice, the actor and critic often share a common encoder (e.g., convolutional layers for pixel inputs or MLP for state vectors) with separate output heads. This reduces parameter count and forces feature reuse. For example, in a continuous control task with 17-dim state and 6-dim action, a shared network might have two hidden layers of 256 units each, then split into a 256→6 linear layer for the actor (outputting mean and log_std) and a 256→1 linear layer for the critic. Total parameters ~150k, compared to ~300k if separate.

The critic is trained using the TD error δ_t = r_t + γ V_φ(s_{t+1}) - V_φ(s_t). The loss is L_critic = (1/2) δ_t^2. This is a simple regression objective, but the target r_t + γ V_φ(s_{t+1}) is non-stationary because V_φ changes during training. This bootstrapping introduces bias but drastically reduces variance compared to Monte Carlo returns. The bias-variance tradeoff is controlled by the discount factor γ (typically 0.99) and the number of steps before bootstrapping (n-step returns).

The actor update uses the critic's output as a baseline: ∇J(θ) ≈ ∇log π_θ(a|s) (Q(s,a) - V_φ(s)). Since Q(s,a) is unknown, we approximate it with the empirical return or TD target. The simplest form uses the TD error itself: ∇J(θ) ≈ ∇log π_θ(a|s) δ_t. This is the one-step actor-critic. It's biased but low-variance. For better performance, we use n-step returns or GAE (next section).

Key implementation detail: the critic must be trained on the same data distribution as the actor (on-policy). If you reuse old data, the critic's value estimates become stale and the actor's gradient becomes biased. This is why A2C and PPO are on-policy algorithms—they discard old trajectories after each update. In production, this means you need a large batch of fresh experience (e.g., 2048 steps per update) to get stable gradient estimates.

import torch
import torch.nn as nn
import torch.nn.functional as F

class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=256):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(state_dim, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU()
        )
        self.actor_mean = nn.Linear(hidden, action_dim)
        self.actor_logstd = nn.Parameter(torch.zeros(action_dim))
        self.critic = nn.Linear(hidden, 1)

    def forward(self, state):
        features = self.encoder(state)
        mean = self.actor_mean(features)
        logstd = self.actor_logstd.expand_as(mean)
        std = torch.exp(logstd)
        dist = torch.distributions.Normal(mean, std)
        value = self.critic(features)
        return dist, value

# Example usage
model = ActorCritic(state_dim=17, action_dim=6)
state = torch.randn(32, 17)  # batch of 32 states
dist, value = model(state)
action = dist.sample()
log_prob = dist.log_prob(action).sum(dim=-1)
print(f"Action shape: {action.shape}, Value shape: {value.shape}")
print(f"Log prob shape: {log_prob.shape}")

Advantage Estimation: From REINFORCE to GAE

The advantage function A(s,a) = Q(s,a) - V(s) measures how much better an action is compared to the average. Using advantage in the policy gradient gives the lowest possible variance among unbiased estimators. But we don't have Q(s,a) directly—we must estimate it. The simplest estimate is the TD error δ_t = r_t + γV(s_{t+1}) - V(s_t), which is a one-step advantage estimate. It's biased (due to bootstrapping) but low-variance. The bias comes from using an imperfect V(s_{t+1}).

To reduce bias, we can use n-step returns: A_t^{(n)} = Σ_{k=0}^{n-1} γ^k r_{t+k} + γ^n V(s_{t+n}) - V(s_t). As n increases, bias decreases (because we rely less on the critic) but variance increases (because we accumulate more reward noise). In practice, n=4 to 16 works well for many tasks. For example, in Atari games, n=5 gives a good balance; in MuJoCo continuous control, n=8-16 is common.

Generalized Advantage Estimation (GAE) elegantly interpolates between all n-step advantages using an exponential weighting with parameter λ ∈ [0,1]. The GAE advantage is A_t^{GAE(λ)} = Σ_{k=0}^{∞} (γλ)^k δ_{t+k}. When λ=0, this is the one-step TD error (high bias, low variance). When λ=1, this is the Monte Carlo return minus baseline (low bias, high variance). Typical values are λ=0.95-0.99. GAE provides a smooth bias-variance tradeoff with a single hyperparameter.

Mathematically, GAE can be computed efficiently in O(T) time by iterating backwards: A_t = δ_t + γλ * A_{t+1}. This recursive formula makes it trivial to implement in practice. The resulting advantages are then normalized (subtract mean, divide by std) before being used in the actor update. This normalization is crucial for stable training across different reward scales.

In production, GAE with λ=0.95 and γ=0.99 is the default for most on-policy algorithms. It consistently outperforms pure n-step returns on a wide range of tasks. The key insight: GAE allows you to use a biased critic (which is easier to learn) while still getting low-bias gradient estimates by tuning λ. This is why modern algorithms like PPO and A2C almost always use GAE.

import torch

def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
    """
    rewards: (T,) tensor of rewards
    values: (T+1,) tensor of values (includes bootstrap value at end)
    dones: (T,) tensor of done flags (1 if terminal, 0 otherwise)
    Returns: advantages (T,) tensor
    """
    T = rewards.shape[0]
    advantages = torch.zeros(T)
    gae = 0.0
    for t in reversed(range(T)):
        delta = rewards[t] + gamma * values[t+1] * (1 - dones[t]) - values[t]
        gae = delta + gamma * lam * (1 - dones[t]) * gae
        advantages[t] = gae
    return advantages

# Example
rewards = torch.tensor([1.0, 0.5, 0.0, -0.5, 2.0])
values = torch.tensor([0.0, 0.2, 0.1, -0.1, 0.5, 0.0])  # includes bootstrap
values[-1] = 0.0  # terminal state value is 0
dones = torch.tensor([0, 0, 0, 0, 1])
adv = compute_gae(rewards, values, dones)
print(f"Advantages: {adv}")

On-Policy Algorithms: A2C and PPO in Detail

A2C (Advantage Actor-Critic) is the synchronous version of A3C. It runs N parallel environments (typically 8-16), collects T steps from each, then computes advantages using GAE and updates the actor and critic. The update is: θ ← θ + α ∇log π_θ(a|s) * A(s,a) for the actor, and φ ← φ - β ∇(V_φ(s) - R)^2 for the critic. A2C is simple, stable, and works well for many tasks. However, it's sample-inefficient because it uses each trajectory exactly once.

PPO (Proximal Policy Optimization) improves upon A2C by allowing multiple updates per trajectory while constraining the policy change. The core idea: clip the probability ratio r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) to [1-ε, 1+ε] (typically ε=0.2). The clipped surrogate objective is L_clip = E[min(r_t A_t, clip(r_t, 1-ε, 1+ε) A_t)]. This prevents the policy from changing too much in a single update, which is the main cause of instability in A2C.

PPO also adds a value function loss (typically MSE) and an entropy bonus to encourage exploration. The total loss is L = L_clip - c1 L_value + c2 H(π_θ), where c1=0.5 and c2=0.01 are typical. The entropy bonus is crucial for preventing premature convergence to suboptimal policies. In practice, PPO with these hyperparameters works across a wide range of tasks with minimal tuning.

Implementation details matter: PPO uses mini-batch SGD over the collected trajectories (typically 4-10 epochs, mini-batch size 64-256). The advantage estimates must be computed using the old policy before any updates. After each epoch, the policy changes slightly, so the advantages become stale—but the clipping prevents this from causing collapse. In production, PPO with 2048 steps per update, 4 epochs, and mini-batch size 64 is a solid starting point.

A2C vs PPO: A2C is simpler and faster per iteration, but PPO is more sample-efficient and stable. For tasks where environment interaction is cheap (e.g., simulated robotics), A2C is fine. For expensive environments (e.g., real-world data), PPO's sample efficiency wins. Both are on-policy, meaning they discard old data—this is a fundamental limitation. Off-policy methods like SAC or DDPG can reuse data but introduce their own stability challenges.

import torch
import torch.nn as nn
import torch.optim as optim

def ppo_update(model, optimizer, states, actions, old_log_probs, advantages, returns, clip_eps=0.2, epochs=4):
    for _ in range(epochs):
        dist, values = model(states)
        log_probs = dist.log_prob(actions).sum(dim=-1)
        ratio = torch.exp(log_probs - old_log_probs)
        
        # Clipped surrogate objective
        surr1 = ratio * advantages
        surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
        actor_loss = -torch.min(surr1, surr2).mean()
        
        # Value loss (clipped or MSE)
        value_loss = nn.MSELoss()(values.squeeze(), returns)
        
        # Entropy bonus
        entropy = dist.entropy().mean()
        
        total_loss = actor_loss + 0.5 * value_loss - 0.01 * entropy
        
        optimizer.zero_grad()
        total_loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()
    return total_loss.item()

# Example usage (dummy data)
model = ActorCritic(state_dim=17, action_dim=6)
optimizer = optim.Adam(model.parameters(), lr=3e-4)
states = torch.randn(64, 17)
actions = torch.randn(64, 6)
old_log_probs = torch.randn(64) * 0.1
advantages = torch.randn(64)
returns = torch.randn(64)
loss = ppo_update(model, optimizer, states, actions, old_log_probs, advantages, returns)
print(f"PPO loss: {loss:.4f}")

Off-Policy Algorithms: SAC, DDPG, and TD3

Off-policy actor-critic methods break the on-policy shackle by learning from experience generated by a behavior policy different from the target policy. This dramatically improves sample efficiency, but introduces deadly triads: function approximation, bootstrapping, and off-policy learning. DDPG (Deep Deterministic Policy Gradient) was the first to scale this to continuous control by pairing a deterministic actor with a Q-function critic updated via clipped double Q-learning. The actor gradient is ∇_θ J ≈ E[∇_a Q(s,a) ∇_θ π_θ(s)] evaluated at a=π_θ(s). In practice, DDPG is brittle—hyperparameter sensitivity and overestimation bias plague it.

TD3 (Twin Delayed DDPG) surgically fixes DDPG's three known failure modes: (1) clipped double Q-learning uses two critics and takes the minimum Q-value for the target, reducing overestimation; (2) delayed policy updates update the actor every d steps (typically d=2) to let the critic stabilize; (3) target policy smoothing adds clipped noise to target actions, forcing the critic to be smooth in regions of low data density. The target update becomes: y = r + γ min_i Q_φ'_i(s', π_θ'(s') + ε) where ε ~ clip(N(0,σ), -c, c). These tweaks make TD3 the default choice for deterministic continuous control.

SAC (Soft Actor-Critic) takes a different philosophy: maximize both expected return and policy entropy. The objective becomes J(π) = Σ E[ r(s,a) + α H(π(·|s)) ]. The entropy term encourages exploration and prevents premature convergence. SAC uses a stochastic actor, a soft Q-function, and an automatic temperature tuning mechanism that adjusts α to hit a target entropy H_target = -dim(A). The critic loss is L_Q = E[(Q(s,a) - (r + γ (min_i Q_φ'_i(s',a') - α log π(a'|s'))))²]. SAC is the most sample-efficient off-policy method for continuous control, but its stochasticity can be a liability in latency-critical production settings where deterministic inference is preferred.

All three algorithms share a common architecture: replay buffer, target networks with Polyak averaging (τ ≈ 0.005), and gradient clipping. The choice between them depends on the problem: SAC for exploration-heavy tasks, TD3 for stable deterministic policies, DDPG only as a baseline. Never use DDPG in production without TD3's fixes.

import torch
import torch.nn as nn
import torch.nn.functional as F

class SoftQNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        super().__init__()
        self.fc1 = nn.Linear(state_dim + action_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, 1)

    def forward(self, state, action):
        x = torch.cat([state, action], dim=-1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

# SAC critic loss (single Q for brevity; double Q in practice)
def sac_critic_loss(q_net, q_target, actor, replay_buffer, gamma=0.99, alpha=0.2):
    s, a, r, s_next, done = replay_buffer.sample(256)
    with torch.no_grad():
        a_next, log_prob = actor.sample(s_next)
        q_next = q_target(s_next, a_next) - alpha * log_prob
        y = r + gamma * (1 - done.float()) * q_next
    q_current = q_net(s, a)
    return F.mse_loss(q_current, y)

Implementation Pitfalls: Gradient Clipping, Target Networks, and Reward Normalization

Actor-critic implementations are notoriously brittle. The three most common failure modes are exploding gradients, unstable target values, and reward scale sensitivity. Gradient clipping is the first line of defense: clip the global gradient norm to a max value (typically 1.0 or 0.5) before applying the optimizer step. Without it, a single outlier TD error can destabilize the entire policy. Use torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) after loss.backward() and before optimizer.step().

Target networks are essential for stabilizing bootstrapping in off-policy methods. The target network parameters φ_target are updated via Polyak averaging: φ_target ← τ φ + (1-τ) φ_target, with τ typically 0.005. Too high τ (e.g., 0.1) makes targets track the online network too quickly, reintroducing instability. Too low τ (e.g., 0.0001) slows learning. In TD3, target networks are updated only when the actor updates (every d steps). A common mistake is updating target networks every step—this wastes compute and can hurt convergence.

Reward normalization is often overlooked but critical. RL algorithms assume rewards are bounded; unbounded rewards cause Q-values to explode. The simplest fix is to normalize rewards online using a running mean and standard deviation: r_normalized = (r - running_mean) / (running_std + 1e-8). Alternatively, clip rewards to [-1, 1] if the reward scale is known. In continuous control tasks like MuJoCo, reward scales vary by orders of magnitude (e.g., HalfCheetah rewards ~1000, Humanoid ~10). Without normalization, the critic's Q-function must learn to output values spanning multiple orders of magnitude, which is hard with fixed learning rates.

Other pitfalls include: (1) not resetting the optimizer state when switching between training and evaluation; (2) using the same learning rate for actor and critic (critic typically needs lower LR, e.g., 3e-4 vs 1e-3); (3) forgetting to detach target values when computing critic loss; (4) using a replay buffer that's too small (minimum 1e5 transitions for continuous control). Always validate your implementation on a simple task like Pendulum-v1 before scaling.

import torch
import torch.nn as nn
import torch.optim as optim

class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.actor = nn.Sequential(nn.Linear(state_dim, 256), nn.ReLU(), nn.Linear(256, action_dim), nn.Tanh())
        self.critic = nn.Sequential(nn.Linear(state_dim, 256), nn.ReLU(), nn.Linear(256, 1))

    def forward(self, state):
        return self.actor(state), self.critic(state)

model = ActorCritic(3, 1)
optimizer = optim.Adam(model.parameters(), lr=3e-4)

# Training loop with proper gradient clipping and reward normalization
running_mean, running_std = 0.0, 1.0
for episode in range(1000):
    state = torch.randn(1, 3)
    action, value = model(state)
    reward = torch.randn(1) * 10  # unnormalized
    
    # Online reward normalization
    running_mean = 0.99 * running_mean + 0.01 * reward.item()
    running_std = 0.99 * running_std + 0.01 * (reward.item() - running_mean)**2
    reward_normalized = (reward - running_mean) / (running_std**0.5 + 1e-8)
    
    # Compute loss (simplified)
    td_error = reward_normalized + 0.99 * value.detach() - value
    loss = td_error**2
    
    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()

Production Deployment: Distributed Training, Monitoring, and Debugging

Production RL systems must handle scale, latency, and reliability. Distributed training is the standard approach: multiple workers collect experience in parallel, sending trajectories to a central learner. For actor-critic, the most common architecture is A3C-style (Asynchronous Advantage Actor-Critic) or IMPALA (Importance Weighted Actor-Learner Architecture). In A3C, each worker maintains its own copy of the policy and applies gradients asynchronously to a shared model. This is simple but suffers from stale gradients. IMPALA uses a single learner that processes trajectories from many actors, correcting for off-policyness with V-trace. For off-policy methods like SAC, use a distributed replay buffer (e.g., R2D2-style) where actors write to a shared buffer and the learner samples from it.

Monitoring is critical: track reward per episode, episode length, Q-values, policy entropy, and gradient norms. Set up alerts for when reward drops below a threshold or when entropy collapses (indicating policy is stuck). Use TensorBoard or Weights & Biases to log these metrics. For debugging, add a 'canary' evaluation environment that runs the current policy every N steps without exploration noise. If the canary reward diverges from training reward, your exploration is masking poor policy quality.

Debugging RL in production is harder than supervised learning because there's no ground truth. Common issues: (1) the environment is non-stationary (e.g., user behavior changes over time)—use domain randomization or periodic retraining; (2) the reward function is misspecified—add reward shaping or inverse RL; (3) the policy overfits to the training environment—use multiple random seeds and test on held-out environments. Always save checkpoints every 1000 steps and keep a rolling window of the last 10 checkpoints for rollback.

Latency is a first-class concern. For real-time systems (e.g., robotics, ad serving), the actor must run in milliseconds. Use ONNX Runtime or TensorRT to export the policy network. Batch inference is rarely possible in online settings, so optimize for single-sample latency: use smaller networks (2 layers of 256 units is often enough), quantize to FP16, and avoid Python overhead by deploying in C++ or Rust. For off-policy methods, the critic is only used during training—you can strip it from the deployment artifact.

import torch
import torch.multiprocessing as mp
import gym

def actor_worker(rank, shared_model, replay_queue, env_name='Pendulum-v1'):
    env = gym.make(env_name)
    state = env.reset()
    while True:
        with torch.no_grad():
            action = shared_model.actor(torch.FloatTensor(state).unsqueeze(0)).squeeze(0).numpy()
        next_state, reward, done, _ = env.step(action)
        replay_queue.put((state, action, reward, next_state, done))
        state = next_state if not done else env.reset()

def learner(model, replay_queue, batch_size=256):
    optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)
    replay_buffer = []
    while True:
        while len(replay_buffer) < 10000:
            replay_buffer.append(replay_queue.get())
        batch = random.sample(replay_buffer, batch_size)
        # SAC update (simplified)
        loss = compute_sac_loss(model, batch)
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

if __name__ == '__main__':
    mp.set_start_method('spawn')
    model = SACNet(3, 1)
    model.share_memory()
    replay_queue = mp.Queue(maxsize=100000)
    workers = [mp.Process(target=actor_worker, args=(i, model, replay_queue)) for i in range(4)]
    for w in workers: w.start()
    learner(model, replay_queue)

Advanced Topics: Multi-Agent Actor-Critic and Hierarchical RL

Multi-agent actor-critic (MAAC) extends the framework to environments with multiple interacting agents. The key challenge is non-stationarity: each agent's policy changes during training, making the environment appear non-stationary from any single agent's perspective. MADDPG (Multi-Agent DDPG) addresses this by using a centralized critic that observes all agents' actions and states, while each agent has a decentralized actor. The critic's Q-function is Q_i(s, a_1, ..., a_N) where s is the global state and a_i are all agents' actions. This stabilizes training because the critic sees the full picture. However, it doesn't scale to many agents because the action space grows exponentially. For large-scale multi-agent systems (e.g., traffic control), use mean-field approximations or value decomposition networks (VDN, QMIX).

Hierarchical RL (HRL) decomposes a complex task into sub-tasks at different temporal abstractions. The classic architecture is the Options framework: a high-level policy selects an 'option' (a sub-policy) that runs for multiple time steps. The actor-critic variant uses a manager (high-level actor) and a worker (low-level actor). The manager outputs a goal or sub-goal, and the worker learns to achieve it. The critic evaluates both levels: the worker's critic uses intrinsic reward (e.g., goal achievement), while the manager's critic uses extrinsic reward. HRL suffers from non-stationarity at the high level because the low-level policy changes. Solutions include off-policy corrections (HIRO) or using a fixed low-level policy during high-level training.

Another advanced topic is the combination of actor-critic with model-based RL. The actor learns a policy, the critic learns a value function, and a learned world model predicts next states and rewards. This enables planning: the actor can simulate trajectories in the model and use the critic to evaluate them. Dreamer and MuZero are prominent examples. MuZero learns a model that predicts reward, value, and policy without requiring the true environment dynamics—it's a fully learned world model. The actor-critic update then uses both real and imagined trajectories. This is the state of the art for sample efficiency in board games and video games.

Finally, consider meta-learning for actor-critic: learning to learn. MAML (Model-Agnostic Meta-Learning) can be applied to actor-critic by learning initial parameters that can quickly adapt to new tasks. The meta-objective is to minimize the loss after a few gradient steps on a new task. This is useful for robotics where the same robot must adapt to different environments. The challenge is that the inner loop (adaptation) requires computing second-order gradients, which is memory-intensive. First-order approximations (Reptile, FOMAML) work well in practice.

import torch
import torch.nn as nn

class CentralizedCritic(nn.Module):
    def __init__(self, num_agents, state_dim, action_dim, hidden_dim=256):
        super().__init__()
        # Concatenate all states and actions
        input_dim = num_agents * (state_dim + action_dim)
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, states, actions):
        # states: (batch, num_agents, state_dim), actions: (batch, num_agents, action_dim)
        batch = states.shape[0]
        x = torch.cat([states.view(batch, -1), actions.view(batch, -1)], dim=-1)
        return self.net(x)

# Usage: critic = CentralizedCritic(3, 10, 2)
# q_value = critic(all_states, all_actions)  # single Q-value for all agents