HTTP 500 Internal Server Error — Pool Exhaustion No Timeout

// io.thecodeforge — System Design tutorial
// What actually happens during an HTTP 500 — request/response lifecycle

// === CLIENT SIDE (what the browser or API consumer sees) ===

REQUEST:
  POST /api/checkout/complete HTTP/1.1
  Host: shop.example.com
  Content-Type: application/json
  Body: { "cart_id": "abc123", "payment_token": "tok_xyz" }

RESPONSE (what the client receives — almost useless for debugging):
  HTTP/1.1 500 Internal Server Error
  Content-Type: application/json
  Body: { "error": "Something went wrong. Please try again." }

// Notice: the client gets ZERO useful information.
// This is intentional — leaking stack traces to clients is a security risk.
// The real information lives in the SERVER LOGS, not the response.

// === SERVER SIDE (what actually happened — where you debug) ===

[2024-11-29 02:47:13] ERROR CheckoutService - Unhandled exception during payment processing
  java.lang.NullPointerException: Cannot invoke method getBalance() on null object reference
    at io.thecodeforge.checkout.PaymentProcessor.validateFunds(PaymentProcessor.java:112)
    at io.thecodeforge.checkout.CheckoutService.completeOrder(CheckoutService.java:87)
    at io.thecodeforge.checkout.CheckoutController.handleCheckout(CheckoutController.java:45)
  Caused by: UserAccount object was null — user session expired mid-checkout

// === INFRASTRUCTURE LAYER (may be the real root cause) ===

[2024-11-29 02:47:13] WARN DatabasePool - Connection pool exhausted (max=10, active=10, pending=47)
// 47 requests waiting for a DB connection that never comes free.
// The NullPointerException above is a SYMPTOM.
// The DB pool exhaustion is the ROOT CAUSE.
// Fixing only the NPE would not fix the 500s — they'd keep coming.

// === THE THREE LAYERS — always check all three ===
// Layer 1: HTTP response  → tells you a 500 happened
// Layer 2: App logs       → tells you WHAT failed (stack trace)
// Layer 3: Infra metrics  → tells you WHY it failed (root cause)

// io.thecodeforge — System Design tutorial
// Decision tree: diagnosing which category of 500 you're dealing with
// Run these checks IN ORDER — each one narrows the field

==========================================================================
STEP 1 — Did this just start? Or has it always happened on this endpoint?
==========================================================================

  Always happened on this endpoint:
    → Likely: Unhandled exception OR misconfiguration
    → Go to STEP 3

  Just started after a deployment:
    → Likely: Bad deployment (syntax error, missing env var, schema mismatch)
    → IMMEDIATE ACTION: Check deploy logs and consider rollback
    → Go to STEP 2

  Started gradually under load:
    → Likely: Resource exhaustion or DB connection pool saturation
    → Go to STEP 4

==========================================================================
STEP 2 — Bad Deployment Checklist
==========================================================================

  [ ] Check application startup logs — did the process even start cleanly?
      Red flag: "Error: Cannot find module './config/database'"
      Red flag: "SyntaxError: Unexpected token }" (runtime parse error)

  [ ] Check environment variables are set in the NEW environment
      Red flag: process.env.DATABASE_URL is undefined
      Fix: Re-run your secrets injection / config sync before redeploying

  [ ] Check for database schema mismatches
      Red flag: "column 'user_tier' does not exist" (code expects column, migration didn't run)
      Fix: Run pending migrations BEFORE deploying code that depends on them

  [ ] If nothing obvious — ROLL BACK first, investigate second
      Rule: Production stability > root cause analysis. Rollback. Then debug.

==========================================================================
STEP 3 — Unhandled Exception / Misconfiguration Checklist
==========================================================================

  [ ] Pull the server application log for the exact timestamp of the 500
      Look for: stack trace, exception class name, file + line number

  [ ] Most common exception types that cause 500s:
      NullPointerException / TypeError   → object was null/undefined when you accessed it
      FileNotFoundException              → config file path is wrong or file not deployed
      ClassNotFoundException             → dependency jar/package missing in production
      OperationalError: no such table   → database migration never ran

  [ ] Search for the error message verbatim in your codebase
      This tells you exactly which line threw — and whether it has error handling

==========================================================================
STEP 4 — Resource Exhaustion Checklist
==========================================================================

  [ ] Database connection pool
      Check: SELECT count(*) FROM pg_stat_activity; (PostgreSQL)
      Red flag: active connections near or at max_connections limit
      Quick fix: Kill idle connections; longer fix: tune pool size + add timeouts

  [ ] Memory
      Check: `free -h` (Linux) or your cloud provider's memory metric
      Red flag: available memory near zero, OOMKiller in system logs
      Fix: Increase instance size OR fix the memory leak (heap dump required)

  [ ] Disk space
      Check: `df -h`
      Red flag: filesystem at 100% — logs often fill disks silently
      Quick fix: Clear old logs; permanent fix: log rotation + disk alerts

  [ ] File descriptors
      Check: `ulimit -n` vs `lsof | wc -l`
      Red flag: open files near system limit
      Fix: Increase ulimit; check for connection/file handle leaks in code

==========================================================================
DECISION OUTPUT — what to do with your finding
==========================================================================

  Bad Deployment    → Rollback → Fix → Redeploy with proper migration order
  Unhandled Exception → Add try/catch → return meaningful error response → fix root cause
  Misconfiguration  → Set the missing config → restart service → add config validation at startup
  Resource Exhaustion → Immediate: scale or kill idle connections → Long term: fix the leak

// io.thecodeforge — System Design tutorial
// Production error handling pattern for an e-commerce checkout service
// Framework: Express.js — patterns apply to any Node.js web framework

const express = require('express');
const app = express();

// ─────────────────────────────────────────────────────────────────
// CIRCUIT BREAKER — fail fast when a dependency is known to be down
// Without this: every request hangs for 30s waiting for a timeout,
// threads pile up, memory spikes, the whole service goes 500.
// ─────────────────────────────────────────────────────────────────
class CircuitBreaker {
  constructor(failureThreshold = 5, recoveryTimeoutMs = 30000) {
    this.failureCount = 0;
    this.failureThreshold = failureThreshold; // open circuit after 5 consecutive failures
    this.state = 'CLOSED'; // CLOSED = normal, OPEN = failing fast, HALF_OPEN = testing recovery
    this.nextAttemptAt = null;
    this.recoveryTimeoutMs = recoveryTimeoutMs;
  }

  async call(operationFn) {
    if (this.state === 'OPEN') {
      if (Date.now() < this.nextAttemptAt) {
        // Still in recovery window — reject immediately without calling the dependency
        throw new Error('CircuitBreaker:OPEN — dependency unavailable, failing fast');
      }
      // Recovery window expired — allow one probe request through
      this.state = 'HALF_OPEN';
    }

    try {
      const result = await operationFn();
      this._onSuccess();
      return result;
    } catch (err) {
      this._onFailure();
      throw err; // re-throw so the caller handles it — don't swallow
    }
  }

  _onSuccess() {
    this.failureCount = 0;
    this.state = 'CLOSED'; // dependency is healthy again
  }

  _onFailure() {
    this.failureCount += 1;
    if (this.failureCount >= this.failureThreshold) {
      this.state = 'OPEN';
      // Schedule the recovery probe — don't hammer a sick dependency
      this.nextAttemptAt = Date.now() + this.recoveryTimeoutMs;
    }
  }
}

// One circuit breaker per external dependency — never share them
const paymentGatewayBreaker = new CircuitBreaker(5, 30000);
const inventoryServiceBreaker = new CircuitBreaker(3, 15000);

// ─────────────────────────────────────────────────────────────────
// CHECKOUT ROUTE — specific error handling close to the source
// Each failure type gets its own response — no generic catch-all
// ─────────────────────────────────────────────────────────────────
app.post('/api/checkout/complete', async (req, res, next) => {
  const { cartId, paymentToken, userId } = req.body;

  // INPUT VALIDATION — catch bad requests before any business logic runs
  // These are 400s, not 500s — the client sent bad data, not our fault
  if (!cartId || !paymentToken || !userId) {
    return res.status(400).json({
      error: 'MISSING_REQUIRED_FIELDS',
      message: 'cartId, paymentToken, and userId are all required'
    });
  }

  try {
    // STEP 1: Check inventory via circuit-breaker-protected call
    const inventoryAvailable = await inventoryServiceBreaker.call(() =>
      checkInventoryAvailability(cartId)
    );

    if (!inventoryAvailable) {
      // This is an expected business failure — not a 500, it's a 409 Conflict
      return res.status(409).json({
        error: 'INVENTORY_CONFLICT',
        message: 'One or more items in your cart are no longer available'
      });
    }

    // STEP 2: Process payment via circuit-breaker-protected call
    const paymentResult = await paymentGatewayBreaker.call(() =>
      chargePaymentToken(paymentToken, calculateCartTotal(cartId))
    );

    // STEP 3: Persist the order — wrap in try/catch for DB-specific errors
    const order = await persistOrder(userId, cartId, paymentResult.transactionId);

    return res.status(201).json({
      orderId: order.id,
      transactionId: paymentResult.transactionId,
      status: 'CONFIRMED'
    });

  } catch (err) {
    // SPECIFIC KNOWN ERRORS — handle gracefully without a 500
    if (err.message.includes('CircuitBreaker:OPEN')) {
      // Dependency is known-down — tell the client, don't pretend it's our fault
      return res.status(503).json({
        error: 'SERVICE_TEMPORARILY_UNAVAILABLE',
        message: 'Payment processing is temporarily unavailable. Please try again in 30 seconds.',
        retryAfterSeconds: 30
      });
    }

    if (err.code === 'PAYMENT_DECLINED') {
      // Payment gateway explicitly declined — this is a 402, client needs to act
      return res.status(402).json({
        error: 'PAYMENT_DECLINED',
        message: 'Your payment was declined. Please check your card details and try again.'
      });
    }

    // UNEXPECTED ERROR — pass to the global error handler via next()
    // DO NOT return a 500 here — let the central handler do it.
    // DO NOT log here — the central handler does that too.
    // This keeps logging consistent and prevents double-logging.
    next(err);
  }
});

// ─────────────────────────────────────────────────────────────────
// GLOBAL ERROR HANDLER — the last line of defence
// Express recognises this as an error handler because it has 4 params
// This runs for any error that reaches next(err) from any route
// ─────────────────────────────────────────────────────────────────
app.use((err, req, res, next) => {
  // Generate a unique ID so you can correlate the user's report with your logs
  const errorId = `ERR-${Date.now()}-${Math.random().toString(36).substr(2, 6).toUpperCase()}`;

  // ALWAYS log the full stack trace server-side — never swallow it
  // Include request context so you can reproduce the failure
  console.error({
    errorId,
    message: err.message,
    stack: err.stack,
    request: {
      method: req.method,
      url: req.url,
      userId: req.body?.userId,    // log who was affected
      cartId: req.body?.cartId,    // log what they were doing
      userAgent: req.headers['user-agent']
    },
    timestamp: new Date().toISOString()
  });

  // Trigger your alerting pipeline here (PagerDuty, Sentry, etc.)
  // notifyOnCallEngineer(err, errorId); ← wire this up in production

  // Return the error ID to the client — they can quote it in a support ticket
  // NEVER return the stack trace or internal error message to the client
  return res.status(500).json({
    error: 'INTERNAL_SERVER_ERROR',
    message: 'An unexpected error occurred. Please try again or contact support.',
    errorId  // lets your support team look this up in logs instantly
  });
});

// Placeholder stubs — these would be real service calls in production
async function checkInventoryAvailability(cartId) { return true; }
async function chargePaymentToken(token, amount) { return { transactionId: 'txn_abc123' }; }
async function calculateCartTotal(cartId) { return 99.99; }
async function persistOrder(userId, cartId, txnId) { return { id: 'order_xyz789' }; }

app.listen(3000, () => console.log('Checkout service running on port 3000'));

// io.thecodeforge — System Design tutorial
// Production readiness checklist: preventing and catching 500s before users do

==========================================================================
TIER 1 — STARTUP VALIDATION (catch misconfigs before the service accepts traffic)
==========================================================================

At service boot, BEFORE binding to a port:

  [ ] Validate all required environment variables exist and are non-empty
      Pattern: fail-fast with a clear message
      Example:
        const required = ['DATABASE_URL', 'PAYMENT_API_KEY', 'JWT_SECRET'];
        required.forEach(key => {
          if (!process.env[key]) {
            throw new Error(`STARTUP_FAILURE: Required environment variable '${key}' is not set.`);
            // Process exits. Load balancer sees the instance never became healthy.
            // No 500s served. Clean failure.
          }
        });

  [ ] Test database connectivity at startup
      Pattern: ping the DB, confirm connection pool initialises successfully
      If DB is unreachable at startup: crash loudly, do not serve traffic

  [ ] Verify critical config file paths exist
      Pattern: fs.accessSync(configPath) — throws if file missing, crashes cleanly

==========================================================================
TIER 2 — STRUCTURED LOGGING (make logs searchable when it matters most)
==========================================================================

  Bad log (plain text — useless under pressure):
    [ERROR] Something failed during checkout for user abc at 2024-11-29 02:47:13

  Good log (structured JSON — filterable in 10 seconds):
    {
      "timestamp": "2024-11-29T02:47:13.000Z",
      "level": "ERROR",
      "service": "checkout-service",
      "errorId": "ERR-1732845600000-K7X2MN",
      "userId": "usr_456",
      "cartId": "cart_789",
      "endpoint": "POST /api/checkout/complete",
      "errorClass": "NullPointerException",
      "message": "Cannot invoke getBalance() on null UserAccount",
      "durationMs": 234
    }

  Why this matters: grep '"userId": "usr_456"' | jq '.errorId'
  Gets you the exact error ID in one command. Without structure: read every line manually.

==========================================================================
TIER 3 — ALERTING THRESHOLDS (know before your users do)
==========================================================================

  Metric                        | Alert threshold          | Severity
  ─────────────────────────────────────────────────────────────────────
  5xx error rate (critical path)| > 1% over 5 min window   | PAGE
  5xx error rate (non-critical) | > 5% over 5 min window   | SLACK ALERT
  DB connection pool usage      | > 80% of max             | SLACK ALERT
  DB connection pool usage      | > 95% of max             | PAGE
  Available memory              | < 20% of total           | SLACK ALERT
  Disk usage                    | > 85% of total           | SLACK ALERT
  Disk usage                    | > 95% of total           | PAGE
  P99 response latency          | > 5x normal baseline     | SLACK ALERT

  Key rule: alert on RATE, not raw count.
  10 errors in 1 minute during 10 req/min traffic = 100% error rate. PAGE.
  10 errors in 1 minute during 100,000 req/min traffic = 0.01% error rate. Ignore.

==========================================================================
TIER 4 — HEALTH CHECK ENDPOINT (let your load balancer save you)
==========================================================================

  GET /health → should check:
    [ ] Database is reachable (run a lightweight SELECT 1 query)
    [ ] Memory usage is below critical threshold
    [ ] All required config is loaded
    [ ] Any circuit breakers are not permanently open

  Return 200 only when ALL checks pass.
  Return 503 (not 500) when any dependency is unhealthy.

  Your load balancer polls /health every 10-30 seconds.
  If it gets a non-200, it stops routing traffic to that instance.
  This means a sick instance stops serving 500s automatically — 
  without anyone waking up at 3am to restart it manually.

  Health check response time must be < 500ms.
  If your health check itself times out, it causes cascading failures.