14.2 Million ConcurrentHashMap Entries — JVM OOM Debug

SYMPTOM                          | WHAT TO CHECK                  | TOOL                           | IMMEDIATE ACTION
---------------------------------+--------------------------------+--------------------------------+-------------------------------------------
Exit code 137 (no JVM error)     | Container memory vs heap       | kubectl top + jcmd VM.native   | Increase container to 1.43x heap
OOM: Java heap space             | Heap contents (what is big?)   | jmap -histo + Eclipse MAT      | Find leak via dominator tree
OOM: Metaspace                   | Classloader count              | jcmd VM.classloader_stats      | Restart JVM, fix classloader leak
OOM: Direct buffer memory        | Direct buffer allocation       | jcmd VM.native_memory summary  | Fix buffer leak, set MaxDirectMemorySize
OOM: GC overhead limit           | GC frequency + old gen usage   | jstat -gcutil + GC logs        | Fix memory leak (GC cannot free enough)
StackOverflowError               | Call stack depth               | jstack <pid>                   | Convert recursion to iteration
Latency spikes (no OOM)          | GC pause times                 | GC logs (-Xlog:gc)             | Tune GC or switch collector
CPU high + slow response         | GC time percentage             | jstat -gcutil <pid> 1s         | If GC time > 5%, fix leak or increase heap
Memory grows over hours          | Old gen trend (post-GC)        | jstat -gcutil + jmap -histo    | Compare histograms, find growing types
OOM only at high traffic         | Allocation rate                | JFR (settings=profile)         | Reduce allocation rate or increase heap
OOM only in production           | Object count comparison        | jmap -histo (prod vs staging)  | Find data-dependent leak
OOM after code deploy            | Code diff (new caches/threads) | git diff + heap dump           | Check for removed eviction logic

#!/bin/bash
# ============================================================
# JVM Debug Commands — Production Reference
# Run from inside the container or on the host with JVM access
# ============================================================

PID=$(pgrep -f 'java.*-Xmx')  # Find JVM PID

# ============================================================
# JCMD — Swiss Army Knife (JDK 11+)
# ============================================================

# List all JVM processes
jcmd

# JVM summary (uptime, arguments, heap config)
jcmd $PID VM.info

# Native memory breakdown (heap, thread, class, GC, direct)
jcmd $PID VM.native_memory summary
jcmd $PID VM.native_memory summary.diff  # Since last baseline
jcmd $PID VM.native_memory baseline      # Set baseline for diff

# Classloader statistics (class count, classloader count)
jcmd $PID VM.classloader_stats

# GC class statistics (instance count and size by class)
jcmd $PID GC.class_stats | head -20

# Force full GC
jcmd $PID GC.run

# Print all VM flags
jcmd $PID VM.flags -all | grep -E '(HeapDump|GC|Metaspace|DirectMemory|ThreadStackSize)'

# Print command line flags (shows effective GC settings)
jcmd $PID VM.command_line

# Thread dump (replaces jstack)
jcmd $PID Thread.print

# Heap dump
jcmd $PID GC.heap_dump /tmp/heap.hprof

# Heap histogram (live objects only, forces GC)
jcmd $PID GC.class_histogram | head -30

# JFR: start recording
jcmd $PID JFR.start name=debug settings=profile maxsize=100M maxage=1h

# JFR: dump recording
jcmd $PID JFR.dump name=debug filename=/tmp/recording.jfr

# JFR: stop recording
jcmd $PID JFR.stop name=debug

# ============================================================
# JMAP — Heap Dump and Histogram
# ============================================================

# Histogram of live objects (top 30 by count)
jmap -histo:live $PID | head -30

# Histogram of all objects (including unreachable — faster, no GC)
jmap -histo $PID | head -30

# Full heap dump (live objects only — forces GC first)
jmap -dump:live,format=b,file=/tmp/heap.hprof $PID

# Full heap dump (all objects — faster but larger file)
jmap -dump:format=b,file=/tmp/heap_all.hprof $PID

# ============================================================
# JSTAT — Real-Time GC Monitoring
# ============================================================

# GC utilization every 1 second, 10 samples
jstat -gcutil $PID 1000 10

# Output columns:
# S0  — Survivor 0 usage %
# S1  — Survivor 1 usage %
# E   — Eden usage %
# O   — Old gen usage %  ← KEY METRIC for leak detection
# M   — Metaspace usage %
# CCS — Compressed class space usage %
# YGC — Young GC count
# YGCT — Young GC total time (seconds)
# FGC — Full GC count         ← SHOULD BE 0 in healthy service
# FGCT — Full GC total time (seconds)
# GCT — Total GC time (seconds)

# Key diagnostic:
# If O (old gen) keeps growing after GC → memory leak
# If FGC > 0 and increasing → old gen pressure
# If GCT/uptime > 5% → GC overhead problem

# ============================================================
# JSTACK — Thread Dump
# ============================================================

# Full thread dump
jstack $PID > /tmp/threads.txt

# Thread dump with lock information
jstack -l $PID > /tmp/threads_locked.txt

# Count threads by state (useful for thread leak detection)
jstack $PID | grep "java.lang.Thread.State" | sort | uniq -c | sort -rn

# ============================================================
# KUBERNETES / CONTAINER COMMANDS
# ============================================================

# Pod memory usage
kubectl top pod <pod-name> --containers

# Pod memory limits and usage
kubectl describe pod <pod-name> | grep -A 10 "Limits\|Requests"

# Container OOM kill events
kubectl get events --field-selector reason=OOMKilling

# Exec into running container
kubectl exec -it <pod-name> -- /bin/bash

# Check container memory limit from inside container
cat /sys/fs/cgroup/memory/memory.limit_in_bytes  # cgroup v1
cat /sys/fs/cgroup/memory.max                     # cgroup v2

# Check container memory usage from inside container
cat /sys/fs/cgroup/memory/memory.usage_in_bytes   # cgroup v1
cat /sys/fs/cgroup/memory.current                  # cgroup v2

# ============================================================
# QUICK DIAGNOSTIC SEQUENCE (Run this for any OOM)
# ============================================================

echo "=== Quick JVM Memory Diagnostic ==="
echo "PID: $PID"
echo ""
echo "--- 1. JVM Flags ---"
jcmd $PID VM.flags -all | grep -E '(MaxHeap|MaxMetaspace|MaxDirect|ThreadStack|GC)'
echo ""
echo "--- 2. Native Memory Summary ---"
jcmd $PID VM.native_memory summary
echo ""
echo "--- 3. Heap Histogram (top 15) ---"
jmap -histo:live $PID | head -15
echo ""
echo "--- 4. GC Status ---"
jstat -gcutil $PID 1000 5
echo ""
echo "--- 5. Thread Count ---"
jcmd $PID Thread.print | grep "java.lang.Thread.State" | wc -l
echo ""
echo "=== Diagnostic Complete ==="

package io.thecodeforge.monitoring;

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.lang.management.MemoryUsage;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.GarbageCollectorMXBean;
import java.util.List;
import java.util.Map;
import java.util.HashMap;

/**
 * OOM Type Detector — identifies which memory region is at risk
 * before an OOM occurs.
 */
public class OomTypeDetector {

    private static final double HEAP_WARNING_THRESHOLD = 0.80;
    private static final double HEAP_CRITICAL_THRESHOLD = 0.90;
    private static final double METASPACE_WARNING_THRESHOLD = 0.80;

    public enum RiskLevel {
        HEALTHY, WARNING, CRITICAL, IMMINENT
    }

    public enum OomType {
        HEAP_SPACE,
        METASPACE,
        DIRECT_BUFFER,
        GC_OVERHEAD,
        STACK_OVERFLOW,
        CONTAINER_LIMIT
    }

    public static class MemoryRiskReport {
        public RiskLevel heapRisk;
        public RiskLevel metaspaceRisk;
        public RiskLevel gcOverheadRisk;
        public Map<OomType, String> recommendations;
        public long heapUsedMB;
        public long heapMaxMB;
        public long metaspaceUsedMB;
        public long metaspaceMaxMB;
        public double gcTimePercent;

        public MemoryRiskReport() {
            recommendations = new HashMap<>();
        }
    }

    public static MemoryRiskReport analyze() {
        MemoryRiskReport report = new MemoryRiskReport();
        MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();

        // Heap analysis
        MemoryUsage heapUsage = memBean.getHeapMemoryUsage();
        report.heapUsedMB = heapUsage.getUsed() / (1024 * 1024);
        report.heapMaxMB = heapUsage.getMax() / (1024 * 1024);
        double heapPercent = (double) heapUsage.getUsed() / heapUsage.getMax();

        if (heapPercent >= HEAP_CRITICAL_THRESHOLD) {
            report.heapRisk = RiskLevel.CRITICAL;
            report.recommendations.put(OomType.HEAP_SPACE,
                "Heap at " + (int)(heapPercent * 100) + "% — capture heap dump and analyze dominator tree.");
        } else if (heapPercent >= HEAP_WARNING_THRESHOLD) {
            report.heapRisk = RiskLevel.WARNING;
            report.recommendations.put(OomType.HEAP_SPACE,
                "Heap at " + (int)(heapPercent * 100) + "% — monitor growth rate.");
        } else {
            report.heapRisk = RiskLevel.HEALTHY;
        }

        // Metaspace analysis
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            if (pool.getName().contains("Metaspace")) {
                MemoryUsage usage = pool.getUsage();
                report.metaspaceUsedMB = usage.getUsed() / (1024 * 1024);
                report.metaspaceMaxMB = usage.getMax() > 0
                    ? usage.getMax() / (1024 * 1024) : -1;
                if (report.metaspaceMaxMB > 0) {
                    double metaPercent = (double) usage.getUsed() / usage.getMax();
                    if (metaPercent >= METASPACE_WARNING_THRESHOLD) {
                        report.metaspaceRisk = RiskLevel.WARNING;
                        report.recommendations.put(OomType.METASPACE,
                            "Metaspace at " + (int)(metaPercent * 100)
                            + "% — check for classloader leaks.");
                    } else {
                        report.metaspaceRisk = RiskLevel.HEALTHY;
                    }
                }
            }
        }

        // GC overhead analysis
        long totalGcTimeMs = 0;
        long totalGcCount = 0;
        for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) {
            totalGcTimeMs += gc.getCollectionTime();
            totalGcCount += gc.getCollectionCount();
        }
        long uptimeMs = ManagementFactory.getRuntimeMXBean().getUptime();
        report.gcTimePercent = (double) totalGcTimeMs / uptimeMs * 100;

        if (report.gcTimePercent > 5.0) {
            report.gcOverheadRisk = RiskLevel.CRITICAL;
            report.recommendations.put(OomType.GC_OVERHEAD,
                "GC consuming " + String.format("%.1f", report.gcTimePercent)
                + "% of uptime — likely memory leak. Capture heap dump.");
        } else if (report.gcTimePercent > 2.0) {
            report.gcOverheadRisk = RiskLevel.WARNING;
        } else {
            report.gcOverheadRisk = RiskLevel.HEALTHY;
        }

        return report;
    }
}

package io.thecodeforge.diagnostics;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.lang.management.MemoryUsage;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

/**
 * Proactive heap monitor that captures dumps when memory
 * growth rate indicates a leak — before OOM occurs.
 */
public class ProactiveHeapMonitor {

    private final ScheduledExecutorService scheduler;
    private final List<Snapshot> history;
    private final long maxHeapMB;
    private final double growthRateThresholdMBPerHour;
    private final String dumpDirectory;

    public ProactiveHeapMonitor(
            long maxHeapMB,
            double growthRateThresholdMBPerHour,
            String dumpDirectory
    ) {
        this.maxHeapMB = maxHeapMB;
        this.growthRateThresholdMBPerHour = growthRateThresholdMBPerHour;
        this.dumpDirectory = dumpDirectory;
        this.history = new ArrayList<>();
        this.scheduler = Executors.newSingleThreadScheduledExecutor(
            r -> {
                Thread t = new Thread(r, "heap-monitor");
                t.setDaemon(true);
                return t;
            }
        );
    }

    public void start(long intervalSeconds) {
        scheduler.scheduleAtFixedRate(
            this::checkMemory,
            intervalSeconds,
            intervalSeconds,
            TimeUnit.SECONDS
        );
    }

    private void checkMemory() {
        MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();
        MemoryUsage heapUsage = memBean.getHeapMemoryUsage();
        long usedMB = heapUsage.getUsed() / (1024 * 1024);
        Instant now = Instant.now();

        history.add(new Snapshot(now, usedMB));

        // Keep only last 24 hours of snapshots
        Instant cutoff = now.minusSeconds(86400);
        history.removeIf(s -> s.timestamp.isBefore(cutoff));

        // Check absolute threshold
        double usagePercent = (double) usedMB / maxHeapMB;
        if (usagePercent > 0.85) {
            logWarning("Heap usage at " + (int)(usagePercent * 100)
                + "% (" + usedMB + "MB / " + maxHeapMB + "MB)");
            if (usagePercent > 0.90) {
                captureHeapDump("high-usage-" + now.getEpochSecond());
            }
        }

        // Check growth rate (leak detection)
        if (history.size() >= 2) {
            Snapshot oldest = history.get(0);
            Snapshot newest = history.get(history.size() - 1);
            double hoursElapsed = (newest.timestamp.toEpochMilli()
                - oldest.timestamp.toEpochMilli()) / 3_600_000.0;
            if (hoursElapsed > 0.5) {
                double growthRateMBPerHour = (newest.usedMB - oldest.usedMB)
                    / hoursElapsed;
                if (growthRateMBPerHour > growthRateThresholdMBPerHour) {
                    logWarning("Heap growth rate: " + growthRateMBPerHour
                        + " MB/hour — possible leak");
                    captureHeapDump("leak-suspect-" + now.getEpochSecond());
                }
            }
        }
    }

    private void captureHeapDump(String label) {
        String filename = dumpDirectory + "/heap-" + label + ".hprof";
        try {
            String pid = ManagementFactory.getRuntimeMXBean().getName()
                .split("@")[0];
            ProcessBuilder pb = new ProcessBuilder(
                "jmap", "-dump:live,format=b,file=" + filename, pid
            );
            pb.redirectErrorStream(true);
            Process p = pb.start();
            int exitCode = p.waitFor();
            if (exitCode == 0) {
                logWarning("Heap dump captured: " + filename);
            } else {
                logWarning("Heap dump failed with exit code: " + exitCode);
            }
        } catch (Exception e) {
            logWarning("Heap dump failed: " + e.getMessage());
        }
    }

    private void logWarning(String message) {
        System.err.println("[HeapMonitor] " + Instant.now() + " " + message);
    }

    private static class Snapshot {
        final Instant timestamp;
        final long usedMB;
        Snapshot(Instant timestamp, long usedMB) {
            this.timestamp = timestamp;
            this.usedMB = usedMB;
        }
    }
}

package io.thecodeforge.monitoring;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * GC Log Analyzer — parses GC logs and extracts key metrics
 * for production tuning decisions.
 */
public class GcLogAnalyzer {

    // JDK 11+ unified GC log format
    private static final Pattern GC_PAUSE_PATTERN = Pattern.compile(
        "\\[(?<timestamp>[\\d-T:.]+)\\]\\[(?<uptime>[\\d.]+)s\\]\\[(?<level>\\w+)\\]"
        + ".*GC\\((?<gcId>\\d+)\\) Pause (?<type>Young|Full|Mixed)"
        + ".*?(?<durationMs>[\\d.]+)ms"
    );

    private static final Pattern HEAP_PATTERN = Pattern.compile(
        "(?<used>\\d+)K->(?<after>\\d+)K\\((?<total>\\d+)K\\)"
    );

    public static class GcMetrics {
        public int totalGcPauses;
        public int youngGcCount;
        public int fullGcCount;
        public int mixedGcCount;
        public double maxPauseMs;
        public double p99PauseMs;
        public double p95PauseMs;
        public double avgPauseMs;
        public double totalPauseMs;
        public double gcTimePercent;
        public long maxHeapUsedKB;
        public long minHeapAfterGcKB;
        public List<Double> pauseTimes = new ArrayList<>();
    }

    public static GcMetrics analyze(String gcLogFile) throws IOException {
        GcMetrics metrics = new GcMetrics();

        try (BufferedReader reader = new BufferedReader(
                new FileReader(gcLogFile))) {
            String line;
            while ((line = reader.readLine()) != null) {
                Matcher pauseMatcher = GC_PAUSE_PATTERN.matcher(line);
                if (pauseMatcher.find()) {
                    double duration = Double.parseDouble(
                        pauseMatcher.group("durationMs"));
                    String type = pauseMatcher.group("type");

                    metrics.totalGcPauses++;
                    metrics.pauseTimes.add(duration);
                    metrics.totalPauseMs += duration;

                    switch (type) {
                        case "Young": metrics.youngGcCount++; break;
                        case "Full":  metrics.fullGcCount++; break;
                        case "Mixed": metrics.mixedGcCount++; break;
                    }

                    if (duration > metrics.maxPauseMs) {
                        metrics.maxPauseMs = duration;
                    }
                }

                Matcher heapMatcher = HEAP_PATTERN.matcher(line);
                if (heapMatcher.find()) {
                    long used = Long.parseLong(heapMatcher.group("used"));
                    long after = Long.parseLong(heapMatcher.group("after"));
                    if (used > metrics.maxHeapUsedKB) {
                        metrics.maxHeapUsedKB = used;
                    }
                    if (metrics.minHeapAfterGcKB == 0
                            || after < metrics.minHeapAfterGcKB) {
                        metrics.minHeapAfterGcKB = after;
                    }
                }
            }
        }

        // Calculate percentiles
        if (!metrics.pauseTimes.isEmpty()) {
            metrics.pauseTimes.sort(Double::compareTo);
            int size = metrics.pauseTimes.size();
            metrics.avgPauseMs = metrics.totalPauseMs / size;
            metrics.p95PauseMs = metrics.pauseTimes.get((int)(size * 0.95));
            metrics.p99PauseMs = metrics.pauseTimes.get((int)(size * 0.99));
        }

        return metrics;
    }

    public static String generateReport(GcMetrics m) {
        StringBuilder sb = new StringBuilder();
        sb.append("=== GC Analysis Report ===\n");
        sb.append("Total GC pauses: ").append(m.totalGcPauses).append("\n");
        sb.append("Young GC: ").append(m.youngGcCount).append("\n");
        sb.append("Full GC: ").append(m.fullGcCount).append("\n");
        sb.append("Mixed GC: ").append(m.mixedGcCount).append("\n");
        sb.append("Max pause: ").append(m.maxPauseMs).append(" ms\n");
        sb.append("P99 pause: ").append(m.p99PauseMs).append(" ms\n");
        sb.append("P95 pause: ").append(m.p95PauseMs).append(" ms\n");
        sb.append("Avg pause: ").append(String.format("%.2f", m.avgPauseMs)).append(" ms\n");
        sb.append("Max heap used: ").append(m.maxHeapUsedKB / 1024).append(" MB\n");
        sb.append("Min heap after GC: ").append(m.minHeapAfterGcKB / 1024).append(" MB\n");

        // Warnings
        if (m.fullGcCount > 0) {
            sb.append("WARNING: Full GC detected — investigate old gen pressure\n");
        }
        if (m.p99PauseMs > 200) {
            sb.append("WARNING: P99 pause > 200ms — consider ZGC or Shenandoah\n");
        }
        if (m.minHeapAfterGcKB > 0) {
            long liveDataMB = m.minHeapAfterGcKB / 1024;
            sb.append("INFO: Live data set ~").append(liveDataMB).append(" MB\n");
            sb.append("INFO: Recommended heap (2x live data): ")
                .append(liveDataMB * 2).append(" MB\n");
        }

        return sb.toString();
    }
}

package io.thecodeforge.diagnostics;

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryUsage;
import java.lang.management.GarbageCollectorMXBean;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

/**
 * Memory Leak Detector — monitors old gen growth rate
 * to detect leaks before OOM occurs.
 *
 * Core insight: a leak shows as increasing old gen usage
 * after each full GC. The post-GC baseline is the key metric.
 */
public class MemoryLeakDetector {

    private final ScheduledExecutorService scheduler;
    private final List<OldGenSnapshot> snapshots;
    private final double alertThresholdMBPerHour;
    private final LeakAlertHandler alertHandler;

    public interface LeakAlertHandler {
        void onLeakDetected(double growthRateMBPerHour,
                           long currentOldGenMB,
                           String recommendation);
    }

    public MemoryLeakDetector(
            double alertThresholdMBPerHour,
            LeakAlertHandler alertHandler
    ) {
        this.alertThresholdMBPerHour = alertThresholdMBPerHour;
        this.alertHandler = alertHandler;
        this.snapshots = new ArrayList<>();
        this.scheduler = Executors.newSingleThreadScheduledExecutor(
            r -> {
                Thread t = new Thread(r, "leak-detector");
                t.setDaemon(true);
                return t;
            }
        );
    }

    public void start(long intervalSeconds) {
        scheduler.scheduleAtFixedRate(
            this::sampleOldGen,
            intervalSeconds,
            intervalSeconds,
            TimeUnit.SECONDS
        );
    }

    private void sampleOldGen() {
        long oldGenUsedMB = getOldGenUsedMB();
        Instant now = Instant.now();

        snapshots.add(new OldGenSnapshot(now, oldGenUsedMB));

        // Keep only last 6 hours
        Instant cutoff = now.minusSeconds(21600);
        snapshots.removeIf(s -> s.timestamp.isBefore(cutoff));

        // Need at least 30 minutes of data
        if (snapshots.size() < 6) return;

        // Calculate growth rate
        OldGenSnapshot oldest = snapshots.get(0);
        OldGenSnapshot newest = snapshots.get(snapshots.size() - 1);
        double hoursElapsed = (newest.timestamp.toEpochMilli()
            - oldest.timestamp.toEpochMilli()) / 3_600_000.0;

        if (hoursElapsed < 0.5) return;

        double growthRateMBPerHour = (newest.usedMB - oldest.usedMB)
            / hoursElapsed;

        if (growthRateMBPerHour > alertThresholdMBPerHour) {
            String recommendation = buildRecommendation(
                growthRateMBPerHour, newest.usedMB);
            alertHandler.onLeakDetected(
                growthRateMBPerHour, newest.usedMB, recommendation);
        }
    }

    private long getOldGenUsedMB() {
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            String name = pool.getName();
            if (name.contains("Old") || name.contains("Tenured")) {
                return pool.getUsage().getUsed() / (1024 * 1024);
            }
        }
        // Fallback: use heap usage
        MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();
        return memBean.getHeapMemoryUsage().getUsed() / (1024 * 1024);
    }

    private String buildRecommendation(
            double growthRateMBPerHour, long currentOldGenMB
    ) {
        StringBuilder sb = new StringBuilder();
        sb.append("Memory leak detected. ");
        sb.append("Growth rate: ").append(String.format("%.1f", growthRateMBPerHour));
        sb.append(" MB/hour. ");
        sb.append("Current old gen: ").append(currentOldGenMB).append(" MB. ");
        sb.append("Actions: ");
        sb.append("1) Capture heap dump (jmap -dump:live,format=b,file=heap.hprof). ");
        sb.append("2) Analyze with MAT — check dominator tree and histogram. ");
        sb.append("3) Compare with previous histogram to find growing object types.");
        return sb.toString();
    }

    private static class OldGenSnapshot {
        final Instant timestamp;
        final long usedMB;
        OldGenSnapshot(Instant timestamp, long usedMB) {
            this.timestamp = timestamp;
            this.usedMB = usedMB;
        }
    }
}

#!/bin/bash
# Production JVM flags for containerized Java services
# Tested on JDK 17 with G1GC and ZGC configurations

# ============================================================
# BASELINE CONFIGURATION (G1GC — suitable for most services)
# ============================================================

JVM_BASE_FLAGS="
  # Memory
  -XX:MaxRAMPercentage=70.0
  -XX:InitialRAMPercentage=50.0
  -XX:MaxMetaspaceSize=256m
  -Xss512k

  # GC — G1GC
  -XX:+UseG1GC
  -XX:MaxGCPauseMillis=200
  -XX:G1HeapRegionSize=4m
  -XX:InitiatingHeapOccupancyPercent=45

  # Diagnostics (non-negotiable)
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=/var/log/jvm/heapdump.hprof
  -XX:+ExitOnOutOfMemoryError
  -XX:+CrashOnOutOfMemoryError

  # GC Logging (JDK 11+)
  -Xlog:gc*:file=/var/log/jvm/gc.log:time,uptime,level,tags:filecount=5,filesize=100m

  # JFR continuous recording
  -XX:StartFlightRecording=settings=default,maxsize=100M,maxage=1h,name=continuous,filename=/var/log/jvm/recording.jfr

  # Compressed oops (auto-enabled <32GB heap)
  -XX:+UseCompressedOops
  -XX:+UseCompressedClassPointers
"

# ============================================================
# LOW-LATENCY CONFIGURATION (ZGC — for p99 < 10ms services)
# ============================================================

JVM_ZGC_FLAGS="
  # Memory
  -XX:MaxRAMPercentage=70.0
  -XX:MaxMetaspaceSize=256m
  -Xss512k

  # GC — ZGC
  -XX:+UseZGC
  -XX:+ZGenerational          # JDK 21+ generational ZGC
  -XX:ConcGCThreads=4
  -XX:ParallelGCThreads=8

  # Diagnostics
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=/var/log/jvm/heapdump.hprof
  -XX:+ExitOnOutOfMemoryError
  -Xlog:gc*:file=/var/log/jvm/gc.log:time,uptime,level,tags:filecount=5,filesize=100m
  -XX:StartFlightRecording=settings=default,maxsize=100M,maxage=1h,name=continuous
"

# ============================================================
# CONTAINER MEMORY CALCULATION
# ============================================================
#
# For a 4GB heap (-XX:MaxRAMPercentage=70.0 on a 5.7GB container):
#
#   Heap:           4000 MB  (70% of 5700MB)
#   Metaspace:       256 MB  (MaxMetaspaceSize)
#   Thread stacks:   200 MB  (400 threads × 512KB)
#   Direct memory:   256 MB  (default = Xmx)
#   GC overhead:     200 MB  (G1GC bookkeeping)
#   Native/JNI:      300 MB  (JNI libraries, socket buffers)
#   OS overhead:     500 MB  (page cache, file descriptors)
#   ----------------------------------------
#   Total:          5712 MB  (container limit: 5.7GB)
#
# Formula: Container = Xmx × 1.43 (round up to nearest 256MB)
# ============================================================

echo "JVM flags configured for production deployment"
echo "Container memory recommendation: Xmx × 1.43"

package io.thecodeforge.monitoring;

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryUsage;
import java.lang.management.ThreadMXBean;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.Map;

/**
 * Off-Heap Memory Monitor — tracks memory usage outside
 * the JVM heap that contributes to container OOM kills.
 */
public class OffHeapMonitor {

    public static class OffHeapReport {
        public long metaspaceUsedMB;
        public long metaspaceMaxMB;
        public long threadStackMB;
        public int threadCount;
        public long directMemoryMaxMB;
        public long compressedClassSpaceMB;
        public long codeCacheMB;
        public Map<String, String> recommendations = new HashMap<>();

        public long totalOffHeapMB() {
            return metaspaceUsedMB + threadStackMB
                + compressedClassSpaceMB + codeCacheMB;
        }
    }

    public static OffHeapReport analyze() {
        OffHeapReport report = new OffHeapReport();

        // Metaspace
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            String name = pool.getName();
            MemoryUsage usage = pool.getUsage();

            if (name.contains("Metaspace")) {
                report.metaspaceUsedMB = usage.getUsed() / (1024 * 1024);
                report.metaspaceMaxMB = usage.getMax() > 0
                    ? usage.getMax() / (1024 * 1024) : -1;
            } else if (name.contains("Compressed Class Space")) {
                report.compressedClassSpaceMB = usage.getUsed() / (1024 * 1024);
            } else if (name.contains("Code Cache")) {
                report.codeCacheMB = usage.getUsed() / (1024 * 1024);
            }
        }

        // Thread stacks
        ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
        report.threadCount = threadBean.getThreadCount();
        // Estimate: each thread uses -Xss (default ~1MB)
        // More accurate: check -XX:ThreadStackSize via VM flags
        report.threadStackMB = report.threadCount; // Rough estimate: 1MB per thread

        // Direct memory limit
        try {
            long maxDirectMemory = sun.misc.VM.maxDirectMemory();
            report.directMemoryMaxMB = maxDirectMemory / (1024 * 1024);
        } catch (Exception e) {
            report.directMemoryMaxMB = -1;
        }

        // Recommendations
        if (report.metaspaceUsedMB > 200) {
            report.recommendations.put("metaspace",
                "Metaspace using " + report.metaspaceUsedMB
                + "MB — check for classloader leaks");
        }
        if (report.threadCount > 300) {
            report.recommendations.put("threads",
                report.threadCount + " threads active — "
                + report.threadStackMB + "MB in stacks. "
                + "Consider reducing thread pool size or -Xss.");
        }
        long totalOffHeap = report.totalOffHeapMB();
        if (totalOffHeap > 1024) {
            report.recommendations.put("total",
                "Total off-heap: " + totalOffHeap + "MB. "
                + "Ensure container memory limit accounts for this.");
        }

        return report;
    }

    /**
     * Monitor direct buffer allocation rate.
     * Call this periodically to detect direct memory pressure.
     */
    public static long getDirectMemoryUsedEstimate() {
        // NMT is more accurate, but this gives a quick estimate
        // by attempting a small allocation and checking if it succeeds
        try {
            ByteBuffer test = ByteBuffer.allocateDirect(1024);
            test = null;
            return -1; // Allocation succeeded — no pressure
        } catch (OutOfMemoryError e) {
            return 0; // Direct memory exhausted
        }
    }
}

package io.thecodeforge.monitoring;

import com.sun.management.OperatingSystemMXBean;
import java.lang.management.*;
import java.util.HashMap;
import java.util.Map;

/**
 * Memory Metrics Exporter — exposes JVM memory metrics
 * for Prometheus/monitoring integration.
 *
 * Lightweight alternative to heavy APM agents.
 * Estimated overhead: <5MB heap, <0.1% CPU.
 */
public class MemoryMetricsExporter {

    public static class MemoryMetrics {
        // Heap
        public long heapUsedMB;
        public long heapMaxMB;
        public long heapCommittedMB;
        public double heapUsagePercent;

        // Young gen
        public long youngGenUsedMB;
        public long youngGenMaxMB;

        // Old gen
        public long oldGenUsedMB;
        public long oldGenMaxMB;
        public double oldGenUsagePercent;

        // Off-heap
        public long metaspaceUsedMB;
        public long metaspaceMaxMB;
        public long threadCount;
        public long threadStackEstimateMB;
        public long directMemoryMaxMB;

        // GC
        public long youngGcCount;
        public long youngGcTimeMs;
        public long fullGcCount;
        public long fullGcTimeMs;
        public double gcTimePercent;

        // Container
        public long containerMemoryLimitMB;
        public long processPhysicalMemoryMB;
        public double containerUsagePercent;
    }

    public static MemoryMetrics collect() {
        MemoryMetrics m = new MemoryMetrics();
        MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();

        // Heap
        MemoryUsage heap = memBean.getHeapMemoryUsage();
        m.heapUsedMB = heap.getUsed() / (1024 * 1024);
        m.heapMaxMB = heap.getMax() / (1024 * 1024);
        m.heapCommittedMB = heap.getCommitted() / (1024 * 1024);
        m.heapUsagePercent = (double) heap.getUsed() / heap.getMax() * 100;

        // Memory pools
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            String name = pool.getName();
            MemoryUsage usage = pool.getUsage();
            if (name.contains("Eden") || name.contains("Survivor")) {
                m.youngGenUsedMB += usage.getUsed() / (1024 * 1024);
                if (usage.getMax() > 0) {
                    m.youngGenMaxMB += usage.getMax() / (1024 * 1024);
                }
            } else if (name.contains("Old") || name.contains("Tenured")) {
                m.oldGenUsedMB = usage.getUsed() / (1024 * 1024);
                m.oldGenMaxMB = usage.getMax() > 0
                    ? usage.getMax() / (1024 * 1024) : 0;
                m.oldGenUsagePercent = m.oldGenMaxMB > 0
                    ? (double) m.oldGenUsedMB / m.oldGenMaxMB * 100 : 0;
            } else if (name.contains("Metaspace")) {
                m.metaspaceUsedMB = usage.getUsed() / (1024 * 1024);
                m.metaspaceMaxMB = usage.getMax() > 0
                    ? usage.getMax() / (1024 * 1024) : -1;
            }
        }

        // GC stats
        long uptimeMs = ManagementFactory.getRuntimeMXBean().getUptime();
        for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) {
            String name = gc.getName();
            if (name.contains("Young") || name.contains("Scavenge")
                    || name.contains("G1 Young")) {
                m.youngGcCount = gc.getCollectionCount();
                m.youngGcTimeMs = gc.getCollectionTime();
            } else if (name.contains("Old") || name.contains("MarkSweep")
                    || name.contains("G1 Old")) {
                m.fullGcCount = gc.getCollectionCount();
                m.fullGcTimeMs = gc.getCollectionTime();
            }
        }
        m.gcTimePercent = (double)(m.youngGcTimeMs + m.fullGcTimeMs)
            / uptimeMs * 100;

        // Threads
        ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
        m.threadCount = threadBean.getThreadCount();
        m.threadStackEstimateMB = m.threadCount; // ~1MB per thread estimate

        // Container / OS memory
        try {
            OperatingSystemMXBean osBean = (OperatingSystemMXBean)
                ManagementFactory.getOperatingSystemMXBean();
            long totalPhysical = osBean.getTotalPhysicalMemorySize();
            long freePhysical = osBean.getFreePhysicalMemorySize();
            m.processPhysicalMemoryMB = (totalPhysical - freePhysical)
                / (1024 * 1024);
            m.containerMemoryLimitMB = totalPhysical / (1024 * 1024);
            m.containerUsagePercent = (double) m.processPhysicalMemoryMB
                / m.containerMemoryLimitMB * 100;
        } catch (Exception e) {
            // Not available on all JVMs
        }

        return m;
    }

    public static String toPrometheusFormat(MemoryMetrics m) {
        StringBuilder sb = new StringBuilder();
        sb.append("# HELP jvm_memory_heap_used_bytes JVM heap used\n");
        sb.append("# TYPE jvm_memory_heap_used_bytes gauge\n");
        sb.append("jvm_memory_heap_used_bytes ")
            .append(m.heapUsedMB * 1024 * 1024).append("\n\n");

        sb.append("# HELP jvm_memory_old_gen_usage_percent Old gen usage\n");
        sb.append("# TYPE jvm_memory_old_gen_usage_percent gauge\n");
        sb.append("jvm_memory_old_gen_usage_percent ")
            .append(String.format("%.2f", m.oldGenUsagePercent)).append("\n\n");

        sb.append("# HELP jvm_gc_full_count Full GC count\n");
        sb.append("# TYPE jvm_gc_full_count counter\n");
        sb.append("jvm_gc_full_count ").append(m.fullGcCount).append("\n\n");

        sb.append("# HELP jvm_memory_container_usage_percent Container memory usage\n");
        sb.append("# TYPE jvm_memory_container_usage_percent gauge\n");
        sb.append("jvm_memory_container_usage_percent ")
            .append(String.format("%.2f", m.containerUsagePercent)).append("\n");

        return sb.toString();
    }
}