Eureka UP, Gateway 502 Drops: Debugging Spring Boot Microservices in Production

If you've ever deployed a Spring Boot microservices architecture using Spring Cloud Eureka and Spring Cloud Gateway, you've likely seen this ghost: Eureka says all services are UP, yet your Gateway starts vomiting 502 Bad Gateway errors like a broken coffee machine. The Gateway is the single entry point for all client requests. It uses Eureka to discover available service instances and then forwards requests to one of them via load balancing. When a 502 happens, the Gateway successfully picked a service instance from the registry, but the actual HTTP connection to that instance failed — either because the instance crashed, is in the middle of a graceful shutdown, or its health endpoint responds but the main application thread pool is exhausted. The most painful part is that Eureka's default eviction interval is 60 seconds. That means a dead instance can sit in the registry for a full minute, and the Gateway will happily keep routing traffic to it. In high-throughput systems with auto-scaling, this gets worse: instances that are killed during scale-in events can cause 502 spikes that last 30-90 seconds. This article walks through a real incident I debugged in a payment-processing system handling 5k requests/second, and gives you the exact code changes, configuration tweaks, and debugging commands to fix it.

Understanding the Gateway-Eureka Dance

Spring Cloud Gateway acts as a reverse proxy that routes incoming requests to downstream microservices. It uses Spring Cloud LoadBalancer (or Netflix Ribbon in older versions) to select a healthy instance from the list provided by Eureka Client. The flow is: 1) Gateway receives a request. 2) It extracts the service name from the route configuration (e.g., lb://payment-service). 3) It queries the LoadBalancer, which either hits its cache or calls Eureka Client to get the list of instances. 4) It picks one instance via a round-robin or random strategy. 5) It opens an HTTP connection to that instance's host:port. If step 5 fails (connection refused, timeout, or the instance returns a non-2xx response), the Gateway returns a 502. The critical detail: Eureka's heartbeat mechanism is a push model — the client sends heartbeats every 30 seconds (default). If the server doesn't receive a heartbeat for 90 seconds (default), it marks the instance as DOWN, but doesn't evict it immediately. The eviction thread runs every 60 seconds. So a dead instance can live in the registry for up to 150 seconds (90s expiry + 60s eviction). During that window, the Gateway's LoadBalancer cache (default 30s) may still hold the stale instance. This is the root cause of intermittent 502 spikes.

import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class GatewayRouteConfig {

    @Bean
    public RouteLocator customRoutes(RouteLocatorBuilder builder) {
        return builder.routes()
            .route("payment-service", r -> r
                .path("/api/payments/**")
                .filters(f -> f
                    .circuitBreaker(config -> config
                        .setName("paymentCircuitBreaker")
                        .setFallbackUri("forward:/fallback/payment")
                    )
                    .retry(3)
                )
                .uri("lb://payment-service")
            )
            .build();
    }
}

What the Official Docs Won't Tell You

The Spring Cloud Gateway reference documentation explains how to configure routes and filters, but it doesn't warn you about the default timeout values that cause production outages. Here are three undocumented traps: First, spring.cloud.gateway.httpclient.connect-timeout defaults to 45 seconds. That's an eternity in microservices. If a downstream instance is dead, the Gateway will wait 45 seconds before giving up and returning a 502. During that time, the thread is blocked. Second, the LoadBalancer cache (spring.cloud.loadbalancer.cache.ttl) defaults to 30 seconds in Spring Cloud 2022.0.x. But if you're using the old Ribbon-based approach (deprecated since 2020), the cache TTL is 10 seconds. Many teams upgraded from Ribbon to Spring Cloud LoadBalancer and saw their 502 spikes triple because the cache TTL increased. Third, there's a hidden property spring.cloud.gateway.loadbalancer.use-blocking which defaults to false (reactive). If you accidentally set it to true (e.g., by copying an old config), the Gateway will use a blocking HTTP client that can exhaust the thread pool under load, causing cascading 502s. I've seen a team spend 3 days debugging a 502 issue that was caused by a single line in their application.properties: spring.cloud.gateway.loadbalancer.use-blocking=true. Remove that and the 502s vanished.

spring:
  cloud:
    gateway:
      httpclient:
        connect-timeout: 2000
        response-timeout: 5s
      loadbalancer:
        use-blocking: false  # NEVER set to true
    loadbalancer:
      cache:
        ttl: 5s  # Match your Eureka eviction interval
        capacity: 1024

eureka:
  client:
    registry-fetch-interval-seconds: 5
  instance:
    lease-renewal-interval-in-seconds: 10
    lease-expiration-duration-in-seconds: 20

Step-by-Step Debugging: From Symptom to Root Cause

When you see a 502 error, don't panic and restart the Gateway. Follow this structured approach. Step 1: Check if the downstream service is actually healthy. Use curl -v http://:/actuator/health directly. If it returns 200, the issue is likely in the Gateway's routing or load balancing. Step 2: Enable Gateway debug logging. Add logging.level.org.springframework.cloud.gateway=DEBUG and logging.level.org.springframework.cloud.loadbalancer=TRACE to your application.properties. Look for lines like 'LoadBalancer cache hit' or 'Selected service instance'. You'll see the exact instance being used. Step 3: Check Eureka's registry for that instance. Call GET /eureka/apps/ on the Eureka server. Look at the status field for each instance. If it says UP but the instance is dead, you have a stale entry. Step 4: Verify the LoadBalancer cache. Spring Cloud LoadBalancer uses a Caffeine cache by default. You can expose cache metrics via Actuator. Add management.endpoints.web.exposure.include=loadbalancer-cache. Then call GET /actuator/loadbalancer-cache to see cache hits, misses, and evictions. A high hit rate with stale entries means your TTL is too long. Step 5: Check the Gateway's thread pool. If you're using the reactive stack (WebFlux), there's no thread pool per se, but you can monitor the event loop group. Use management.metrics.export.prometheus.enabled=true and check reactor_netty_http_client_connections_active. If active connections are maxed out, your downstream is slow.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.filter.CommonsRequestLoggingFilter;

@Configuration
public class DebugLoggingConfig {

    @Bean
    public CommonsRequestLoggingFilter requestLoggingFilter() {
        CommonsRequestLoggingFilter filter = new CommonsRequestLoggingFilter();
        filter.setIncludeClientInfo(true);
        filter.setIncludeQueryString(true);
        filter.setIncludePayload(true);
        filter.setMaxPayloadLength(1000);
        filter.setAfterMessagePrefix("GATEWAY REQUEST: ");
        return filter;
    }
}

# In application.properties add:
# logging.level.org.springframework.cloud.gateway=DEBUG
# logging.level.org.springframework.cloud.loadbalancer=TRACE

Configuring Eureka for Fast Eviction

Eureka's default configuration is designed for long-lived, stable instances. In a Kubernetes environment where pods are created and destroyed every few minutes, you need to tune Eureka for fast eviction. The key properties are on the Eureka server: eureka.server.eviction-interval-timer-in-ms (default 60000 ms) and eureka.server.response-cache-update-interval-ms (default 30000 ms). Set eviction interval to 5000 ms (5 seconds). On the Eureka client (your microservices), set eureka.instance.lease-renewal-interval-in-seconds to 5 and eureka.instance.lease-expiration-duration-in-seconds to 15. This means the client sends a heartbeat every 5 seconds, and if the server doesn't receive one for 15 seconds, it marks the instance as DOWN. With a 5-second eviction interval, the dead instance will be removed within 20 seconds. But be careful: too aggressive eviction can cause flapping. If a network hiccup causes a missed heartbeat, the instance might be evicted prematurely. In a payment system, we use 10/20/5 (renewal/expiration/eviction) as a balance. Also, enable self-preservation mode on the Eureka server: eureka.server.enable-self-preservation=true. This prevents Eureka from evicting all instances if it loses network connectivity to the clients. Without self-preservation, a network partition can cause a complete registry wipe.

import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableEurekaServer
public class EurekaServerConfig {
    // application.yml for the Eureka server:
    // eureka:
    //   server:
    //     enable-self-preservation: true
    //     eviction-interval-timer-in-ms: 5000
    //     response-cache-update-interval-ms: 5000
    //     renewal-percent-threshold: 0.85
}

// Client-side config (in each microservice):
// eureka:
//   instance:
//     lease-renewal-interval-in-seconds: 10
//     lease-expiration-duration-in-seconds: 20
//     prefer-ip-address: true
//   client:
//     registry-fetch-interval-seconds: 5

Gateway Circuit Breaker and Timeout Configuration

Even with fast Eureka eviction, there will be brief windows where the Gateway tries to connect to a dying instance. The solution is a circuit breaker pattern with resilience4j. Add the spring-cloud-starter-circuitbreaker-reactor-resilience4j dependency. Then configure a circuit breaker on the Gateway route. The circuit breaker will wrap the downstream call and if it fails (timeout, connection refused, 5xx), it will open the circuit and return a fallback response instead of a 502. Configure the circuit breaker with a sliding window of 10 calls, a failure rate threshold of 50%, and a wait duration of 10 seconds before half-open. Also set a timeout on the circuit breaker itself — resilience4j's default timeout is 1 second, which is good. But you also need to set the Gateway's HTTP client timeout to be slightly shorter than the circuit breaker timeout. For example, set spring.cloud.gateway.httpclient.connect-timeout=1500 (1.5s) and the circuit breaker timeout to 2 seconds. This way, the HTTP client times out first (1.5s), the circuit breaker catches it and opens the circuit, and the client gets a fallback response in under 2 seconds. Without the circuit breaker, the Gateway would wait 45 seconds (default connect timeout) and then return a 502. The fallback can be a simple static response or a call to a cached data service.

import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.timelimiter.TimeLimiterConfig;
import org.springframework.cloud.circuitbreaker.resilience4j.ReactiveResilience4JCircuitBreakerFactory;
import org.springframework.cloud.circuitbreaker.resilience4j.Resilience4JConfigBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.time.Duration;

@Configuration
public class Resilience4jConfig {

    @Bean
    public ReactiveResilience4JCircuitBreakerFactory circuitBreakerFactory() {
        ReactiveResilience4JCircuitBreakerFactory factory = new ReactiveResilience4JCircuitBreakerFactory();
        factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
            .circuitBreakerConfig(CircuitBreakerConfig.custom()
                .slidingWindowSize(10)
                .failureRateThreshold(50)
                .waitDurationInOpenState(Duration.ofSeconds(10))
                .permittedNumberOfCallsInHalfOpenState(3)
                .build())
            .timeLimiterConfig(TimeLimiterConfig.custom()
                .timeoutDuration(Duration.ofSeconds(2))
                .build())
            .build());
        return factory;
    }
}

Load Balancer Cache: The Hidden Culprit

Spring Cloud LoadBalancer uses a Caffeine cache to store the list of service instances fetched from Eureka. The cache key is the service name (e.g., payment-service). The cache value is a list of ServiceInstance objects with host, port, and metadata. By default, this cache has a TTL of 30 seconds and a maximum size of 256 entries. The problem: even if Eureka evicts a dead instance, the LoadBalancer cache may still hold the old list for up to 30 seconds. During that time, the Gateway will continue to route traffic to the dead instance. The fix is to set spring.cloud.loadbalancer.cache.ttl to a value lower than your Eureka eviction interval. I recommend 5 seconds. You also need to set spring.cloud.loadbalancer.cache.capacity to a value that can hold all your service instances. If you have 50 services with 10 instances each, set capacity to 1000. Additionally, you can enable eager loading of the cache by setting spring.cloud.loadbalancer.eager-load.enabled=true. This will populate the cache on startup instead of lazily on the first request. Without eager loading, the first request to a new service will incur a cache miss and a call to Eureka, adding latency. In high-throughput systems, this can cause a thundering herd problem where multiple Gateway instances all hit Eureka simultaneously.

import org.springframework.cloud.loadbalancer.annotation.LoadBalancerClient;
import org.springframework.cloud.loadbalancer.annotation.LoadBalancerClients;
import org.springframework.cloud.loadbalancer.core.ServiceInstanceListSupplier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@LoadBalancerClients({
    @LoadBalancerClient(name = "payment-service", configuration = PaymentServiceConfig.class),
    @LoadBalancerClient(name = "order-service", configuration = OrderServiceConfig.class)
})
public class LoadBalancerConfig {

    // application.yml additions:
    // spring:
    //   cloud:
    //     loadbalancer:
    //       cache:
    //         ttl: 5s
    //         capacity: 1000
    //       eager-load:
    //         enabled: true
    //       health-check:
    //         interval: 10s
    //         path: /actuator/health
}

// Custom ServiceInstanceListSupplier for health checks
class PaymentServiceConfig {
    @Bean
    public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier() {
        return ServiceInstanceListSupplier.builder()
            .withDiscoveryClient()
            .withHealthChecks()
            .build();
    }
}

Kubernetes Graceful Shutdown and PreStop Hooks

In Kubernetes, when a pod is terminated (e.g., during a rolling update or scale-in), the kubelet sends a SIGTERM signal to the main process (PID 1). By default, Spring Boot handles SIGTERM by initiating a graceful shutdown: it stops accepting new requests, closes the ApplicationContext, and deregisters from Eureka. However, Kubernetes also sends a SIGKILL after a grace period (default 30 seconds). If the Eureka deregistration takes longer than 30 seconds, the pod is killed before it can deregister. The result: a stale entry in Eureka. The fix is to use a preStop hook in your Kubernetes deployment. The preStop hook runs before the SIGTERM is sent. In the hook, you can call a script that sends a request to the Spring Boot Actuator's shutdown endpoint (/actuator/shutdown) or directly calls the Eureka REST API to deregister. But the most reliable approach is to increase the terminationGracePeriodSeconds to allow enough time for deregistration. Set it to 60 seconds. Also, configure Spring Boot's graceful shutdown timeout with server.shutdown=graceful and spring.lifecycle.timeout-per-shutdown-phase=45s. This gives the application 45 seconds to finish in-flight requests and deregister from Eureka before the SIGKILL arrives. On the Eureka client side, set eureka.instance.lease-expiration-duration-in-seconds to a low value (e.g., 15) so that even if deregistration fails, Eureka will quickly evict the dead instance.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: payment-service
        image: payment-service:1.0.0
        ports:
        - containerPort: 8080
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "curl -X POST http://localhost:8080/actuator/shutdown || true"]
        env:
        - name: SPRING_APPLICATION_JSON
          value: |
            {
              "server.shutdown": "graceful",
              "spring.lifecycle.timeout-per-shutdown-phase": "45s",
              "eureka.instance.lease-expiration-duration-in-seconds": 15
            }

Monitoring and Alerting for 502 Errors

You can't fix what you don't measure. Set up Prometheus metrics and Grafana dashboards to track Gateway 502 errors in real time. Spring Cloud Gateway exposes metrics via Micrometer. The key metrics are: spring.cloud.gateway.requests with tags outcome (SUCCESS, FAILURE) and status (500, 502, etc.). Add a Prometheus alert for when the 502 rate exceeds 1% of total requests for more than 1 minute. Also monitor the Eureka server's eureka.server.evicted-instances metric to see how many instances are being evicted. A sudden spike in evictions often precedes a 502 spike. Additionally, monitor the LoadBalancer cache hit rate via loadbalancer.cache.hit.ratio. If the hit rate drops below 90%, the cache is stale and needs tuning. Finally, set up synthetic monitoring (e.g., with a cron job or a tool like Pingdom) that sends a request through the Gateway every 30 seconds and alerts if it gets a 502. This catches issues before users report them. In our payment system, we have a Grafana dashboard with three panels: Gateway 502 rate (per minute), Eureka evicted instances (per minute), and LoadBalancer cache hit ratio. When the 502 rate spikes, we look at the other two panels to determine if the root cause is Eureka eviction or cache staleness.

groups:
- name: gateway-502-alerts
  rules:
  - alert: HighGateway502Rate
    expr: |
      (rate(spring_cloud_gateway_requests_total{outcome="FAILURE", status="502"}[1m])
      /
      rate(spring_cloud_gateway_requests_total[1m])) > 0.01
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Gateway 502 rate above 1% for 1 minute"
      description: "Current 502 rate: {{ $value | humanizePercentage }}. Check Eureka and LoadBalancer cache."

  - alert: EurekaEvictionSpike
    expr: rate(eureka_server_evicted_instances_total[1m]) > 10
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "Eureka eviction rate spike"
      description: "More than 10 instances evicted per minute. Possible network issue or aggressive eviction config."