Sigmoid Hidden Layers Cause NaN Loss — Activation Fix

Your neural network is dead on arrival without the right activation function. It's that simple. Pick wrong, and your model won't train. It'll just sit there, stuck. The math collapses.

Activation functions aren't just math. They're the decision engine. They take a raw input signal and decide how much of it gets passed on. No activation function means your multi-layer network is mathematically identical to a single layer. It can only draw straight lines. Real data isn't linear.

You need non-linearity. That's what these functions inject. They let networks model curves, complex boundaries, and the messy patterns of real-world data. We'll cut through the theory and show you exactly which one to use, where, and why. You'll see the code, understand the trade-offs, and fix the common pitfalls that kill models in production.

Why Sigmoid Hidden Layers Cause NaN Loss

Activation functions in neural networks are the non-linear transformations applied to each neuron's output. Without them, a network collapses into a single linear transformation, regardless of depth. The core mechanic: an activation function introduces non-linearity, enabling the network to learn complex patterns by deciding which signals pass forward. In practice, the sigmoid function squashes any input to a value between 0 and 1, but its derivative peaks at 0.25 and vanishes near the tails. This causes gradients to shrink exponentially with depth — the vanishing gradient problem — making deep networks untrainable. For hidden layers, sigmoid is a poor choice because its output is not zero-centered, leading to zigzagging updates and slower convergence. Worse, when outputs saturate near 0 or 1, gradients become effectively zero, and loss can explode to NaN as weights oscillate or diverge. Use ReLU or its variants for hidden layers; reserve sigmoid for binary classification output layers where probability interpretation is needed.

The Big Three: ReLU, Sigmoid, and Tanh

ReLU's the king for a reason—we've watched models using Sigmoid in hidden layers stall for days. Its zero gradient for negatives isn't a bug, it's the feature that finally lets deep networks learn. You'll see training loss plummet once you swap those S-shaped functions out.

Don't get me wrong, Sigmoid and Tanh still have their place in output layers. But put them anywhere else and you'll be debugging vanishing gradients all night. Those tiny slopes multiply across layers until your early weights barely budge.

We learned this the hard way when our LSTM's first layer refused to update. The fix? Swapping Tanh for ReLU in the hidden states gave us 3x faster convergence. Your team should treat anything but ReLU in hidden layers as a performance red flag.

# Package: io.thecodeforge.ml.core
import numpy as np
import torch
import torch.nn as nn

class ActivationForge:
    @staticmethod
    def manual_relu(input_tensor):
        """Standard ReLU: f(x) = max(0, x)"""
        return np.maximum(0, input_tensor)

    @staticmethod
    def manual_sigmoid(input_tensor):
        """Sigmoid: 1 / (1 + exp(-x))"""
        return 1 / (1 + np.exp(-input_tensor))

# Production usage with PyTorch
model_layer = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),           # Use ReLU for hidden layers
    nn.Linear(128, 10),
    nn.Softmax(dim=1)    # Use Softmax for multi-class output
)

Softmax — The Probability Architect

Softmax doesn't work on one neuron — it sees the whole output layer at once. It takes every raw score (logit) and squashes them into a probability distribution that sums to exactly 1.0. That's what makes it the right call for multi-class classification. Your model isn't just picking a winner — it's telling you how confident it is across every possible class.

Here's the problem you'll hit in production: Softmax amplifies small logit differences into near-certainty. A model that's barely leaning toward 'Cat' will report 94% confidence. We've seen this burn teams using Softmax outputs directly in risk-scoring pipelines. It's great for ranking, terrible for calibration.

Watch out for extreme logits during training too. When logit values get large, Softmax saturates — gradients vanish and the model stops learning. That's why you'll see temperature scaling and label smoothing in real production systems. They're not optional polish — they're fixes for Softmax's core overconfidence problem.

# Package: io.thecodeforge.ml.output
import torch

def calculate_probabilities(logits):
    """
    Convert raw model scores (logits) into probabilities.
    Formula: exp(i) / sum(exp(j))
    """
    # Use torch.softmax for numerical stability (prevents overflow)
    probabilities = torch.softmax(logits, dim=0)
    return probabilities

# Example output for 3 classes: [Dog, Cat, Bird]
raw_scores = torch.tensor([2.0, 1.0, 0.1])
probs = calculate_probabilities(raw_scores)
print(f"Probabilities: {probs.tolist()}")

The Leaky ReLU Fix: What Your Gradients Are Begging For

Dead neurons happen. You train a network, loss plateaus, and half your ReLU units sit at zero forever, contributing nothing. That's the 'dying ReLU' problem — negative inputs get clamped to zero, gradients stop flowing, and those neurons are effectively dead. A ReLU with a slope of exactly zero for negative values means no gradient update, ever. Enter Leaky ReLU. It replaces that flat zero with a small positive slope — typically 0.01 — allowing a tiny gradient to flow even when the input is negative. That small change keeps gradients alive and neurons trainable. Parametric ReLU goes further by making that slope a learnable parameter, so the network can decide how much to leak. You don't have to guess α. The optimizer will learn it. Why does this matter? Because in deep networks with many layers, dead ReLUs compound. One dead neuron in early layers can kill signal for the entire downstream path. Use Leaky ReLU as your default for hidden layers in regression and classification tasks. The performance gain is small but real, and it eliminates a silent failure mode.

// io.thecodeforge
import torch
import torch.nn as nn

class LeakyNet(nn.Module):
    def __init__(self, alpha=0.01):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)
        self.leaky = nn.LeakyReLU(negative_slope=alpha)

    def forward(self, x):
        x = self.leaky(self.fc1(x))
        x = self.leaky(self.fc2(x))
        return self.fc3(x)

model = LeakyNet()
x = torch.randn(32, 784)
print(f"Output shape: {model(x).shape}")
# Gradients flow even for negative activations

Swish: The Activation That Just Works (And Why It's Not a Gimmick)

Google's Swish activation — x * sigmoid(x) — isn't just a trendy ReLU replacement. It's a smooth, non-monotonic function that empirically outperforms ReLU on deep nets, especially with batch normalization. Why? Because it doesn't zero out negative values completely. Instead, it allows a small negative output that can help regularize the network and smooth the loss landscape. The non-monotonic dip near zero is the secret sauce: it provides a gentle 'off ramp' for negative inputs rather than a sharp cutoff. This means gradients are more stable and training is less sensitive to initialization. In practice, Swish often matches or beats ReLU on ImageNet-scale tasks with no hyperparameter tuning. The trade-off? It's computationally heavier — sigmoid is expensive. But with modern GPU hardware, the cost is negligible for most architectures. Replace ReLU with Swish in your next model and watch validation loss drop. If you're worried about compute, use its close cousin, Hard Swish, which replaces sigmoid with a piecewise linear approximation. Hard Swish is quantized-friendly and mobile-optimized.

// io.thecodeforge
import torch
import torch.nn as nn

class Swish(nn.Module):
    def forward(self, x):
        return x * torch.sigmoid(x)

class HardSwish(nn.Module):
    def forward(self, x):
        # From MobileNetV3: x * ReLU6(x+3) / 6
        return x * nn.functional.relu6(x + 3) / 6

model_swish = nn.Sequential(
    nn.Linear(784, 256),
    Swish(),
    nn.Linear(256, 10)
)

x = torch.randn(1, 784)
print(f"Swish output: {model_swish(x).tolist()[0][:3]}...")

# Hard Swish is ~15% faster on mobile CPUs