Missing Health Check: DevOps Interview Gotcha Broke CI/CD

DevOps interviews are brutal if you walk in memorising buzzwords. Interviewers at companies like Netflix, Spotify, and Stripe don't want you to recite a Wikipedia definition of CI/CD — they want to know if you've felt the pain of a 3am production outage and understand why the practices exist. The difference between a candidate who gets the offer and one who doesn't usually isn't technical depth alone — it's the ability to connect a tool or practice back to a real business problem it solves.

DevOps exists because the old model was broken. Developers would spend weeks writing code, hand a giant batch over a metaphorical wall to operations, and then watch chaos unfold — mismatched environments, undocumented configs, surprise dependencies. DevOps isn't a job title, it's a cultural and technical philosophy: automate everything that can be automated, deliver in small increments, and make feedback loops as short as possible.

By the end of this article you'll be able to answer the questions that trip most candidates up — not by reciting definitions, but by explaining the WHY behind Docker, Kubernetes, CI/CD pipelines, Infrastructure as Code, and monitoring. You'll also know the common traps interviewers set and how to sidestep them with confident, experience-flavoured answers.

Why DevOps Interview Questions Are a Filter for Real-World CI/CD Understanding

Top DevOps interview questions test whether you understand the integration and delivery pipeline as a system, not just a set of tools. They probe your grasp of automation, observability, and failure modes — especially the subtle ones like a missing health check that silently breaks CI/CD. The core mechanic is that a health check is a probe (HTTP, TCP, or command) that validates a service is ready to serve traffic; without it, a deployment can appear successful while the application is actually dead. In practice, a missing health check means the orchestrator (Kubernetes, Nomad, or a load balancer) never detects a crashed or stuck process. The deployment proceeds, the old pods are terminated, and traffic is routed to a non-responsive container. This leads to cascading failures: monitoring alerts fire, rollbacks are manual, and the deployment pipeline reports success despite zero uptime. You must use health checks in every deployment — readiness probes for traffic routing, liveness probes for automatic restarts. They are not optional; they are the difference between a self-healing system and a silent outage. In production, a missing health check is the #1 cause of 'deployment succeeded, app is down' incidents.

Infrastructure as Code (IaC) and Automation

One of the most frequent questions is: 'Why do we need Infrastructure as Code?' In the past, servers were hand-crafted 'pets'—if a production server crashed, no one knew exactly how it was configured. IaC turns infrastructure into 'cattle.' By defining your servers, networks, and databases in code (using tools like Terraform or Ansible), you ensure that your environments are reproducible, version-controlled, and immune to 'configuration drift.' This allows a DevOps engineer to spin up a mirror image of production in minutes for testing purposes.

Interviewers want to see that you understand the pain IaC solves: the 'it works on my machine' syndrome, the cost of manual patching, and the compliance nightmare of snowflake servers. Mentioning the principle of immutability—destroy and rebuild rather than patch—shows you've lived the trade-off between operational overhead and speed.

# io.thecodeforge: Standard AWS Infrastructure Provisioning
# Declaring infrastructure as code ensures consistency across Dev, Staging, and Prod
provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "forge_server" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"

  # In an interview, highlight how tags help in cost tracking and environment isolation
  tags = {
    Name        = "TheCodeForge-Production-Node"
    Environment = "Production"
    ManagedBy   = "Terraform"
    Project     = "ForgeCore"
  }

  # Ensure security groups are also handled via code, not manual console clicks
  vpc_security_group_ids = [aws_security_group.forge_sg.id]
}

Containerization and Orchestration: Docker vs. Kubernetes

Interviewers often ask to explain the relationship between Docker and Kubernetes. Think of Docker as the standardized shipping container: it packages the application and its dependencies so it runs the same anywhere. Kubernetes (K8s) is the crane and the cargo ship: it manages thousands of these containers, handling scaling, self-healing (restarting crashed containers), and load balancing across a cluster of machines.

The real depth comes from explaining the WHY: Docker solves environment consistency (no more 'works on my machine'). Kubernetes solves orchestration at scale. When an interviewer asks 'Should we use Docker or Kubernetes?' the correct answer is 'Both — they solve different problems.' If you're running a single service, Docker is enough. If you have multiple services that need to scale independently, you need K8s. Senior engineers also talk about readiness probes, resource limits, and network policies — because those are the things that actually break in production.

# io.thecodeforge: Optimized Multi-stage Build for Spring Boot
# Stage 1: Build - keeps the final image small and secure
FROM eclipse-temurin:17-jdk-alpine as build
WORKDIR /workspace/app
COPY . .
RUN ./gradlew build -x test

# Stage 2: Runtime - only includes the JRE and the JAR
FROM eclipse-temurin:17-jre-alpine
WORKDIR /app
VOLUME /tmp

# Best Practice: Run as non-root user for production security
RUN addgroup -S forgegroup && adduser -S forgeuser -G forgegroup
USER forgeuser

COPY --from=build /workspace/app/build/libs/*.jar forge-app.jar

EXPOSE 8080
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app/forge-app.jar"]

CI/CD Pipelines: The Automation Heartbeat

CI/CD is the engine that makes DevOps tick. Interviewers want to see you understand the difference between Continuous Integration (merge often, test automatically) and Continuous Delivery (every commit is deployable). The real power comes from the feedback loop: a good pipeline tells you the moment something breaks, so you fix it before it reaches production.

When asked about CI/CD, avoid reciting tools. Instead, talk about pipeline stages: lint → unit test → build → integration test → security scan → deploy to staging → smoke test → deploy to production. Explain why each stage exists and what happens if it fails. Mention that a well-designed pipeline is idempotent: running it twice on the same commit should produce the same result. Also, high-performing teams have less than 1 hour lead time for changes — that's the metric you want to optimise.

# io.thecodeforge: Production CI/CD Pipeline with Quality Gates
name: Forge CI/CD Pipeline
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up JDK 17
        uses: actions/setup-java@v4
        with:
          java-version: '17'
          distribution: 'temurin'
      - name: Run unit tests with coverage
        run: |
          ./gradlew test jacocoTestReport
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        if: success()

  security-scan:
    runs-on: ubuntu-latest
    needs: build-and-test
    steps:
      - uses: actions/checkout@v4
      - name: Run Snyk to check for vulnerabilities
        uses: snyk/actions/gradle-jdk17@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

  deploy-staging:
    runs-on: ubuntu-latest
    needs: [build-and-test, security-scan]
    if: github.ref == 'refs/heads/main'
    environment: staging
    steps:
      - name: Deploy to staging using Helm
        run: |
          helm upgrade --install forge-api ./charts/forge-api \
            --namespace staging \
            --set image.tag=${{ github.sha }} \
            --wait

  health-check:
    runs-on: ubuntu-latest
    needs: deploy-staging
    steps:
      - name: Run smoke tests against staging
        run: |
          forge-health-check --endpoint https://staging.forge.io/health

  deploy-production:
    runs-on: ubuntu-latest
    needs: health-check
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
      - name: Deploy to production with canary
        run: |
          # Use Flux or ArgoCD for GitOps; this is a simplified example
          kubectl set image deployment/forge-api forge-api=forge.io/forge-api:${{ github.sha }} --record

Monitoring and Observability: You Can't Improve What You Can't Measure

DevOps interviews often include questions about monitoring. The key distinction they're looking for is between monitoring (checking known metrics) and observability (the ability to infer unknown states from logs, metrics, and traces). Senior engineers know that dashboards are nice but debugging requires the three pillars: logs (what happened), metrics (how many times it happened), and traces (where it happened in a request's journey).

Interviewers want to hear that you don't just rely on dashboards — you build alerting with actionable thresholds, not noise. For example, alerting on CPU at 90% is useless if your app is IO-bound. The golden signals of monitoring (latency, traffic, errors, saturation) are a good start. Also, mention SLOs, SLIs, and error budgets to show you understand the business side — DevOps is about balancing reliability with velocity.

# io.thecodeforge: Production-grade Prometheus alerting rules
groups:
  - name: forge-production
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 2 minutes"
          description: "Instance {{ $labels.instance }} has error rate {{ $value }}"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 1 second for 5 minutes"

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"

Incident Management and Blameless Post-Mortems

This is the part of DevOps that most candidates ignore. Interviewers at senior levels want to know how you handle incidents, not just how you set up CI/CD. They ask: 'Tell me about a time you handled a production outage.' The structure they expect: detection → containment → root cause analysis → fix → prevention.

Key principles: blameless culture (assume good intent), write a post-mortem within 48 hours, and follow up on action items. The goal is to improve the system, not to find a scapegoat. Senior engineers also talk about incident severity levels (SEV1, SEV2), escalation paths, and how they communicate during an outage. They mention that a good post-mortem has a timeline, a root cause analysis, and action items with owners and due dates.

# Post-Mortem: April 22, 2026 - Payment Gateway Outage

## Severity
SEV1 (payment service down, 12% of users affected)

## Timeline (UTC)
- 14:23 - Alert: error rate spike > 10%
- 14:25 - On-call engineer acknowledges
- 14:30 - Identify that a recent config change removed the retry logic for payment api
- 14:32 - Roll back the config change
- 14:45 - Service restored

## Root Cause
A config change to the payment service accidentally removed the retry logic. The change was committed without code review and deployed without testing.

## Action Items
| Action | Owner | Due |
|--------|-------|-----|
| Add validation to config changes | Alice | 2026-04-29 |
| Enforce mandatory code review for all config changes | Bob | 2026-04-26 |
| Add integration test that simulates payment api timeout | Carol | 2026-05-06 |

Configuration Drift: Why Your IaC Will Lie to You

You'll deploy Terraform. It'll say 'No changes.' But your production server has a config file that doesn't match. That's configuration drift. It happens when someone SSH's in and 'just fixes something.' Or an emergency patch gets applied manually. The infrastructure code thinks it's running version X. Reality is version Y. Next deploy, Terraform reverts the fix. Now you're down. The WHY is simple: humans bypass automation under pressure. The HOW is preventive: immutable infrastructure. Don't patch running servers. Deploy new ones. Use baking AMIs or container images. Treat servers like cattle, not pets. And enforce golden images with Packer. Your CI/CD pipeline should be the only path to production. If someone SSH's in after deployment, that's a policy violation, not a workaround. Interviewers will ask: 'How do you detect drift?' Answer with automated compliance checks. Tools like Cloud Custodian, Chef Inspec, or even a scheduled Terraform plan that alerts on changes. If your IaC and your running environment don't match, you have a compliance incident, not a deploy.

// io.thecodeforge.drift_alert
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"

  # Immutable: user_data never runs on existing instances
  user_data = filebase64("${path.module}/init.sh")

  lifecycle {
    # Prevent manual changes from persisting
    ignore_changes = [
      user_data,
      tags["PatchDate"],
      security_groups
    ]
  }
}

# Drift detection: run this in CI every hour
data "aws_instance" "web_current" {
  instance_id = aws_instance.web.id
}

output "drift_warning" {
  value = "Check instance ${aws_instance.web.id}. If tags differ, drift detected."
}

Git Workflows That Won't Make You Cry at 3 AM

A senior DevOps engineer doesn't just know git commands. They know which workflow prevents merge hell during a hotfix. The WHY: production doesn't care about your feature branch strategy. It cares about getting a fix out in 10 minutes. The HOW: trunk-based development. Main branch is always deployable. Short-lived feature branches (under 2 days). No long-running release branches unless you're pinned to a compliance calendar. When a Sev1 hits, you revert the last commit, not cherry-pick through 12 branches. For feature flags, use LaunchDarkly, not git branches. Your CI/CD should run on every push to main, not only on PR merge. That's how you catch integration failures before they reach production. Avoid the 'git flow' cargo cult. It was written in 2010 and assumes you have a quarterly release cycle, not daily deploys. If you hear a candidate describe a 6-branch workflow for a microservice, they've never been paged at midnight. The simplest test: can you roll back a single commit in under 60 seconds? If not, your git process is a liability.

#!/bin/bash
# io.thecodeforge.rollback
# Production rollback: revert single commit, fast

BRANCH="main"
# Find the bad commit
BAD_SHA=$(git log --oneline -1 --grep="hotfix/vuln")
if [ -z "$BAD_SHA" ]; then
  echo "No hotfix commit found. Exiting safely."
  exit 0
fi

echo "Reverting: $BAD_SHA"
git checkout $BRANCH
git revert $BAD_SHA --no-edit
git push origin $BRANCH

# Verify deploy works
if ! curl -s --fail https://api.example.com/health; then
  echo "Rollback failed! Pinging on-call."
  # Insert PagerDuty call here
  exit 1
fi

echo "Rollback successful."