ARP Cache Timeout — Why 300s Default Breaks HA Failover

Every time you load a webpage, send a Slack message, or ping a server, your operating system has to solve a puzzle before a single byte leaves your machine: it knows the destination's IP address, but your network hardware — your Ethernet card, your Wi-Fi adapter — doesn't understand IP addresses. It only speaks in MAC addresses, those 48-bit hardware identifiers burned into every network interface at the factory. Without a way to bridge that gap, your packets go nowhere.

This is the exact problem ARP was designed to solve back in 1982 (RFC 826), and it's still doing that job on virtually every LAN on the planet. It sits at the boundary between Layer 2 (Data Link) and Layer 3 (Network) of the OSI model, acting as a live translation service that maps 'logical' IP addresses to 'physical' MAC addresses. When it works, it's invisible. When it breaks — or gets exploited — things get interesting fast.

By the end you'll understand exactly how ARP request and reply packets are constructed, why the ARP cache exists and what happens when it goes stale, how ARP spoofing works at a packet level so you can reason about network security, and how to inspect and manipulate ARP behavior on a real Linux or macOS machine. This is the kind of depth that separates engineers who just use networks from engineers who actually understand them.

What is ARP — Address Resolution Protocol?

ARP — Address Resolution Protocol is a core networking mechanism that bridges Layer 2 (MAC) and Layer 3 (IP). Instead of a dry definition, let's see it in action. When your machine wants to send a packet to another machine on the same Ethernet segment, it needs the destination's MAC address. It broadcasts an ARP request: 'who-has 192.168.1.42? Tell 192.168.1.1'. The target unicasts back its MAC. Your OS caches that mapping so future packets don't need to broadcast again.

That's the entire protocol in one paragraph. The details — packet format, cache behavior, timeouts — are where production issues live.

#!/bin/bash
# Inspect ARP cache and watch ARP traffic on Linux

echo "=== Current ARP cache ==="
ip neigh show

echo ""
echo "=== Watch ARP packets for 5 seconds ==="
sudo timeout 5 tcpdump -i eth0 -n arp -c 10 2>/dev/null || echo "No ARP traffic"

echo ""
echo "=== Clear ARP entry for a specific IP ==="
# Uncomment to clear: sudo arp -d 192.168.1.1
# Or: sudo ip neigh del 192.168.1.1 dev eth0

echo ""
echo "=== Force re-resolution via arping ==="
# arping -c 2 -I eth0 192.168.1.1

ARP Spoofing: The Attack That Redirects Traffic Without Routes

ARP spoofing (ARP cache poisoning) exploits the fact that ARP has no authentication. An attacker sends unsolicited ARP replies (gratuitous ARP) claiming to own the IP address of the default gateway or another host. The victim's ARP cache updates with the attacker's MAC, and all traffic destined for that IP is sent to the attacker instead.

How it works: attacker sends "192.168.1.1 is at aa:bb:cc:dd:ee:ff" (where aa:bb:cc:dd:ee:ff is attacker's MAC). The target believes this unsolicited update and forwards all traffic. The attacker can then inspect, modify, or block the traffic — a classic man-in-the-middle attack.

Mitigations: dynamic ARP inspection (DAI) on switches validates ARP packets against DHCP snooping bindings. Port security limits MAC addresses per port. Static ARP entries for critical IPs (gateway, DNS, NTP) prevent poisoning but are administratively heavy. Use arp_filter and arp_ignore sysctl on Linux to reject unsolicited ARP on some interfaces.

Detection: use arpwatch (logs ARP changes) or arp-scan to detect duplicate IP claims. On Linux, arp -a may show the same IP with different MACs over time. Anomaly detection can alert when gateway MAC changes outside maintenance windows.

Gratuitous ARP: The Double-Edged Sword

Gratuitous ARP (GARP) is an ARP announcement sent without a corresponding request. It's used for IP address takeover (failover), MAC address updates, and duplicate address detection (DAD).

In gratuitous ARP, the sender puts its own IP in the 'target IP' field (not the usual request format). The message says 'this IP is now at this MAC'. Recipients may update their ARP cache immediately, even though they didn't ask.

Common uses: - HA failover (VRRP, CARP): Standby server sends GARP to update switch MAC tables and client caches when VIP moves. - MAC address change: If a NIC MAC changes (rare, but possible with virtual machines), GARP can notify the network. - Duplicate IP detection: A node that receives GARP claiming an IP it already owns can detect conflict.

Why GARP fails in production: - Many switches and client OSes ignore unsolicited ARP updates (security hardening). They only update cache in response to requests. - Even when accepted, some implementations only update if the entry doesn't already exist or is stale. - The solution is to send a series of ARP requests for the same IP, forcing a cache refresh via reply.

#!/bin/bash
# Demonstrating gratuitous ARP and testing cache update behavior

# Using arping to send gratuitous ARP (unsolicited)
# This tells the network that IP address 192.168.1.100 is at this interface's MAC

echo "=== Sending gratuitous ARP from this machine ==="
# -U means unsolicited (gratuitous)
# -c 3 sends 3 packets to ensure delivery
# -I specifies the network interface (eth0, wlan0, etc.)
arping -U -c 3 -I eth0 192.168.1.100

echo ""
echo "=== Forcing ARP cache update via request to target ==="
# Alternative: send a request for the IP (as if we're looking for it).
# The target will reply, and we learn its MAC. This is guaranteed to work.
arping -c 2 -I eth0 192.168.1.100

echo ""
echo "=== Check current ARP cache entry for the IP ==="
arp -a 192.168.1.100

echo ""
echo "=== To test GARP acceptance on another machine ==="
# On a Linux destination, check if GARP updated cache:
# sudo tcpdump -i eth0 arp
# ip neigh show
# The sysctl 'arp_accept' controls whether unsolicited ARP updates are accepted.

ARP Cache Internals: Aging, GC and Production Tuning

The ARP cache is a simple key-value store: IP → MAC. But its behavior is governed by several timers and thresholds that directly impact production reliability.

Key Linux sysctl parameters: - gc_stale_time (default 60s): how long an entry can be stale before it's considered for garbage collection. A stale entry means the MAC hasn't been verified recently, but the entry still exists. - gc_thresh1 (default 128): if the cache has fewer entries than this, GC doesn't run. - gc_thresh2 (default 512): if cache exceeds this, GC runs more aggressively. - gc_thresh3 (default 1024): hard limit. Once reached, new ARP resolutions fail with "neighbour table overflow". - base_reachable_time (default 30s): base time for an entry to be considered reachable; actual reachable time = base_reachable_time + random(0, gc_stale_time/2). - delay_first_probe_time (default 5s): time to wait before first probe after an entry becomes stale.

Windows ARP cache: netsh interface ip delete arpcache flushes. Default timeout is 300 seconds (ARP cache timeout = 60 seconds for neighbor unreachability detection actually). Windows uses a different mechanism (NUD).

Switch MAC aging: Layer-2 switches have a MAC address table that maps MACs to ports. Aging time default is often 300 seconds. When a GARP arrives, the switch may update the MAC table if the entry is not the same MAC on different port? Actually, MAC learning updates on any frame with source MAC. If the frame comes from a different port than the current entry, the switch updates immediately (MAC flapping). GARP triggers this. However, ARP cache on the switch (if it's a layer-3 switch) is separate and may not update from GARP.

Tuning for HA: - Reduce gc_stale_time to 15-30 seconds for faster failover. - Increase gc_thresh3 if you have many neighbors (e.g., container hosts). - Set arp_accept=1 if you trust GARP from your failover script. - Always test: send GARP and verify cache update on target with ip neigh show.

Proxy ARP and ARP in Virtualized/Cloud Environments

Proxy ARP is a technique where a device (usually a router) answers ARP requests on behalf of another host. It's used in scenarios like VPNs, virtual IPs, and transparent bridging. The router sees an ARP request for an IP that belongs to a host behind it, and it replies with its own MAC address. This tricks the sender into forwarding traffic to the router, which then forwards the packet to the real destination.

When to use Proxy ARP: - VPN clients on a subnet need to appear as local hosts. - Load balancers that proxy connections to backend servers. - Containers in host-networking mode where the host answers for container IPs.

Production pitfalls: - Proxy ARP can cause routing loops if misconfigured. The router answers for an IP that is on the same subnet but behind itself, leading to a cycle. - It hides the true topology, making debugging harder. - Many security teams disable proxy ARP to prevent spoofing.

ARP in cloud environments (AWS, GCP, Azure): - Cloud providers use Software-Defined Networking (SDN) that replaces ARP entirely. Instances do not send ARP requests to other instances. - The hypervisor handles MAC-to-IP mapping. Even if you see MACs in arp -a, they are virtual MACs assigned by the cloud controller. - Gratuitous ARP is ignored. Failover must use cloud-specific mechanisms: health checks, load balancers, Elastic IPs (AWS), etc. - In AWS, if you move an Elastic IP to another instance, the network mapping updates in seconds — but it's not ARP-based. It's a control plane update. - Key rule: In the cloud, forget everything you know about ARP. It doesn't work the same way.

#!/bin/bash
# Configure Proxy ARP on Linux

echo "=== Enable Proxy ARP on an interface ==="
sysctl -w net.ipv4.conf.eth0.proxy_arp=1

echo ""
echo "=== Add a proxy ARP entry for a remote IP ==="
# Respond to ARP requests for 10.0.0.5 as if it's on the local subnet
sudo ip neigh add proxy 10.0.0.5 dev eth0

echo ""
echo "=== Verify proxy entries ==="
ip neigh show proxy

echo ""
echo "=== Check if proxying is active (from another host) ==="
# arping -I eth0 10.0.0.5
# If proxy works, this returns a reply from the router's MAC.

How ARP Actually Resolves an IP — The Two-Message Dance

When your machine needs to send a packet to 192.168.1.105, it checks the ARP cache. No entry? It broadcasts an ARP Request: "Who has 192.168.1.105? Tell 00:1A:2B:3C:4D:5E." Every host on the broadcast domain sees this. The owner of that IP unicasts back an ARP Reply containing its MAC address. Your kernel now inserts this mapping into the ARP cache and sends the frame. This entire exchange happens before a single TCP handshake or UDP datagram leaves the interface. Timeout on a flooded network? You lose. The request is broadcast, meaning every idle NIC on the subnet wakes up, processes it, and drops it. In a /24 with 250 hosts, ARP storms can spike CPU on older switch ASICs. Design your network so that ARP traffic is the exception, not the heartbeat.

// io.thecodeforge
#include <stdio.h>
#include <string.h>

struct arp_entry {
    unsigned char ip[4];
    unsigned char mac[6];
    int valid;
};

int arp_lookup(struct arp_entry *cache, int size, unsigned char *target_ip) {
    for (int i = 0; i < size; i++) {
        if (cache[i].valid &&
            memcmp(cache[i].ip, target_ip, 4) == 0) {
            return i;
        }
    }
    return -1; // Cache miss; must broadcast ARP request
}

ARP Message Format — What the Wire Actually Carries

The ARP packet is tiny — 28 bytes for IPv4 over Ethernet. It sits inside a Layer 2 frame with EtherType 0x0806. The format is dead simple: hardware type (1 for Ethernet), protocol type (0x0800 for IPv4), hardware size (6 for MAC), protocol size (4 for IPv4), and an opcode (1 for request, 2 for reply). Then come four addresses: sender MAC, sender IP, target MAC (zeroed in requests), target IP. That's it. No headers. No checksum. No validation. Why does this matter? Because a malformed ARP packet with a spoofed sender MAC can corrupt your cache faster than any ACL can stop it. When debugging "intermittent connectivity" on a VLAN, dump the ARP frames with tcpdump -i eth0 arp. If you see duplicate IPs with different MACs, you've found your gremlin.

# io.thecodeforge
import struct

def parse_arp(packet):
    # Skip Ethernet header (14 bytes)
    arp = packet[14:42]
    htype, ptype, hlen, plen, opcode = struct.unpack('!HHBBH', arp[:8])
    sender_mac = ':'.join(f'{b:02x}' for b in arp[8:14])
    sender_ip = '.'.join(str(b) for b in arp[14:18])
    target_mac = ':'.join(f'{b:02x}' for b in arp[18:24])
    target_ip = '.'.join(str(b) for b in arp[24:28])
    return {
        'opcode': 'REQUEST' if opcode == 1 else 'REPLY',
        'sender': f'{sender_ip} @ {sender_mac}',
        'target': f'{target_ip} @ {target_mac}'
    }

Production ARP Attacks You Will Actually See

Users don't complain about "ARP cache poisoning." They complain about "random disconnects" and "my bank login page shows a certificate warning." Here are the three attacks that survive past certs and firewalls. First, ARP spoofing: attacker sends forged ARP replies associating the gateway's IP with its own MAC. Traffic flows through the attacker, allowing MITM sniffing. Second, ARP denial: attacker floods the network with fake MAC claims for a target IP. The target's cache flips to an invalid MAC, dropping traffic until timeout. Third, gratuitous ARP floods: a misconfigured VM or container announces its MAC at startup thousands of times per second. The switch's MAC table fills, causing packet punting to the CPU. In production, mitigate with Dynamic ARP Inspection (DAI) on managed switches. DAI intercepts and validates ARP packets against a trusted DHCP snooping database. Without it, your network is flat and vulnerable.

# io.thecodeforge
#!/bin/bash
# Detect ARP spoofing by tracking MAC changes
# Requires arpwatch or manual arp cache scanning

GATEWAY="192.168.1.1"
KNOWN_MAC="00:11:22:33:44:55"

while true; do
  CUR_MAC=$(arp -n $GATEWAY | tail -1 | awk '{print $3}')
  if [ "$CUR_MAC" != "$KNOWN_MAC" ]; then
    echo "ALERT: Gateway MAC changed from $KNOWN_MAC to $CUR_MAC" | logger -t arp_watch
  fi
  sleep 5
done