Falco Runtime Security for Docker: Catch Container Attacks Before They Spread

You wake up to a PagerDuty alert at 3 AM. CPU on your Kubernetes node is pegged at 100%, and a crypto miner is running inside a container that's supposed to be a stateless Node.js API. The image was scanned in CI. The base image was pinned by digest. The CVE scan passed. But nobody was watching what the container did at runtime. That's the gap Falco fills.

Container security has traditionally focused on the supply chain: scan images, sign artifacts, enforce admission policies. But once a container starts running, all that hardening can be undone by a single exploited vulnerability — a Log4Shell, an RCE in a web framework, or a compromised dependency that drops a reverse shell. Runtime security detects the attack in progress, not after the fact.

Falco is the CNCF standard for this. It's been adopted by major cloud providers, integrated into Kubernetes security stacks (CIS benchmarks, NSA Kubernetes Hardening Guide), and ships with over 1400 rules mapped to the MITRE ATT&CK framework. In this guide, you'll learn how Falco works under the hood, deploy it in production with Helm, write custom rules, route alerts to your incident response pipeline, and avoid the production pitfalls that burn teams new to runtime security.

How Falco Intercepts Syscalls: Kernel Module vs eBPF vs Modern eBPF

Falco has three driver backends for intercepting syscalls: kernel module, classic eBPF, and modern eBPF. Each has different performance characteristics, security implications, and compatibility requirements.

The kernel module (falco-probe.ko) is the oldest and most complete driver. It hooks into the kernel's syscall table directly using kprobes and tracepoints. It has the lowest overhead (1-3% CPU) but requires kernel headers to build, is tied to specific kernel versions, and must be loaded with insmod. If a kernel update happens without rebuilding the module, Falco stops working. Many organizations reject kernel modules on principle — they run in kernel space with full access to memory.

Classic eBPF uses the kernel's BPF subsystem, compiling Falco's probe into BPF bytecode that runs in a sandboxed BPF virtual machine. No kernel module loading is required — just CONFIG_BPF=y and CONFIG_BPF_EVENTS=y. Performance is slightly higher (3-5% CPU) because BPF programs are verified by the kernel before execution. This is the recommended driver for most production deployments. It survives kernel upgrades because BPF is part of the kernel tree.

Modern eBPF (introduced in Falco 0.36+) uses BPF CO-RE (Compile Once, Run Everywhere) with BTF (BPF Type Format) information. It eliminates the need for kernel headers entirely. The probe is compiled against a generic BTF file and adapts to the running kernel at load time. This is the future of Falco instrumentation — one binary that works across all kernel versions from 5.4+. Performance is comparable to classic eBPF. The tradeoff: it requires a relatively modern kernel (5.4+ for BTF support) and a BTF-enabled kernel config.

Which one to pick? If your kernel is 5.4+, use modern eBPF. Otherwise, use classic eBPF. Only fall back to the kernel module if your kernel lacks BPF support entirely (e.g., RHEL 7 or older enterprise kernels). Modern eBPF is now the default in the Falco Helm chart since version 0.37.

// io.thecodeforge — DevOps tutorial

# Check kernel version and BTF support
uname -r
# Example: 6.2.0-35-generic (supports modern eBPF)

# Check if BTF is available
ls /sys/kernel/btf/vmlinux
# If this file exists, modern eBPF works

# Install Falco with modern eBPF driver (Docker run)
docker run --rm -i --privileged \
  -v /var/run/docker.sock:/host/var/run/docker.sock \
  -v /proc:/host/proc:ro \
  -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro \
  -v /usr:/host/usr:ro \
  -v /etc:/host/etc:ro \
  falcosecurity/falco:latest \
  falco --modern-bpf

# Helm chart (recommended)
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
  --set driver.kind=modern_ebpf \
  --set falco.grpc.enabled=true \
  --set falco.grpc.unixSocketPath=/var/run/falco/falco.sock

Falco Rules Deep Dive: Writing Custom Rules for Your Workload

Falco's rule engine is what makes it powerful. Each rule consists of a condition (a boolean expression over syscall fields), an output (a format string for the alert message), a priority, and a set of exceptions. Rules are written in YAML and loaded via -r /path/to/rules.yaml.

The condition macro is the heart of a rule. It can reference syscall fields (evt.type for syscall name, proc.name for process name, fd.name for file descriptor), container metadata (container.name, container.image), and Kubernetes labels (k8s.ns.name, k8s.pod.name). Falco supports over 200 fields across system, container, and orchestration contexts.

Falco ships with default rules organized by MITRE ATT&CK technique. You can see them at /etc/falco/rules.d/. The default rules catch container escapes (mount namespace escape, privileged container), reverse shells, crypto miners (process spawning with high CPU, network connections to mining pools), fileless execution (memfd_create), and unexpected process execution (shell inside a distroless container).

Custom rules follow a standard pattern. For example, a rule to detect any process accessing the Kubernetes service account token would use proc.name != (kubelet, kube-proxy) and fd.name startswith '/var/run/secrets/kubernetes.io/serviceaccount'. Use the append: true attribute to add exceptions to existing rules without modifying the original file — this is the cleanest way to handle false positives.

Write your custom rules in a separate file (e.g., /etc/falco/rules.d/custom-rules.yaml) and load them after the default rules. Falco merges rules alphabetically within the rules directory, so name your files to get the desired load order (e.g., 01-override.yaml for overrides, 99-custom.yaml for additions).

# io.thecodeforge — DevOps tutorial

# Custom Falco rules for workload-specific detection

- macro: app_container
  condition: container.image.repository in (nginx, node, python)

- rule: Shell in App Container
  desc: A shell was spawned in an application container
  condition: >
    spawned_process and container and app_container
    and proc.name in (bash, zsh, sh, dash, ash)
  output: >
    Shell spawned in app container (user=%user.name container=%container.info
    image=%container.image.repository proc=%proc.name cmdline=%proc.cmdline)
  priority: CRITICAL
  tags: [container, mitre_execution, T1059]

- rule: Shell in App Container
  append: true
  condition: and not env.HOSTNAME startswith debug-

- rule: Service Account Token Access
  desc: Detects access to K8s service account token by non-standard processes
  condition: >
    open_read and container
    and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount
    and not proc.name in (kubelet, kube-proxy)
  output: >
    SA token accessed (proc=%proc.name pid=%proc.pid container=%container.info)
  priority: WARNING
  tags: [container, kubernetes, mitre_credential_access, T1528]

Alert Routing with Falcosidekick: Slack, PagerDuty, Webhook, and OTel

Falco itself outputs alerts to stdout, a file, or a gRPC endpoint. For production alert routing, you need falcosidekick — a companion component that takes Falco events and forwards them to 50+ destinations: Slack, PagerDuty, Opsgenie, Teams, Discord, webhooks, S3, GCS, Kafka, OTel, and more.

Falcosidekick runs as a sidecar or separate deployment. Falco sends events to falcosidekick via a Unix socket or TCP. Falcosidekick then applies routing rules: you can send CRITICAL alerts to PagerDuty, WARNING to Slack, INFO to a log file. It also supports deduplication, rate limiting, and muting windows.

For OTel-native observability, falcosidekick can export Falco events as OpenTelemetry logs. This integrates Falco directly into your OTel pipeline alongside application traces and metrics. Set otlp.http.endpoint or otlp.grpc.endpoint to your OTel Collector. Event severity maps to OTel log severity. Container and Kubernetes metadata are attached as OTel resource attributes.

The deployment pattern in Kubernetes: Falco as a DaemonSet with a falcosidekick sidecar per pod. Falcosidekick reads Falco events from the Unix socket and forwards them. For high-volume deployments, use a central falcosidekick service that receives events from all Falco DaemonSet pods via TCP — this gives you a single point for deduplication and routing configuration.

# io.thecodeforge — DevOps tutorial

slack:
  webhookurl: "https://hooks.slack.com/services/T00/B00/XXXX"
  footer: "Falco Runtime Security"
  minimumpriority: "warning"

pagerduty:
  routingkey: "your-pagerduty-routing-key"
  minimumpriority: "critical"

webhook:
  address: "http://ops-alerthandler:8080/falco"
  minimumpriority: "warning"
  customheaders: "Authorization: Bearer your-token"

otlp:
  http:
    endpoint: "http://otel-collector:4318/v1/logs"
  grpc:
    endpoint: "otel-collector:4317"
  minimumpriority: "info"

dedup:
  enabled: true
  timeout: 1m

ratelimit:
  enabled: true
  maxpersecond: 10

Container Escape Detection: What Falco Catches That Admission Controllers Miss

Kubernetes admission controllers (PodSecurity Admission, OPA/Kyverno) validate pod spec against policy before a pod runs. They check: no privileged containers, no hostPID sharing, no hostNetwork access, no volume mount of /var/run/docker.sock. But admission controllers are static — they evaluate a YAML spec. They cannot detect what a running process does.

Falco catches the dynamic attacks that admission controllers miss. The classic container escape: a process inside the container mounts the host's filesystem via mount --bind / /host, then writes an SSH key or a cron job. Admission controllers don't see this because the pod spec was compliant. But the process escalates privileges after starting.

Falco rules that catch escapes: Mount below /host (triggers on mount after a process enters the host mount namespace), ptrace inside container (process tries to trace another container's process), namespace (process creates a new namespace via unshare or clone), write below /dev (process writes to a device file), and system-user interactive (non-system user runs an interactive shell).

The most dangerous escape vector: a container that mounts the Docker socket (/var/run/docker.sock). With the socket, a process can run new containers on the host, including a privileged container that mounts the root filesystem. Falco's Launch Privileged Container rule catches this.

# io.thecodeforge — DevOps tutorial

- rule: Mount Below /host
  desc: a process mounted to a path below /host, indicating container escape
  condition: >
    mount and container and
    (fd.name startswith /host or fd.name startswith /hostroot)
  output: >
    Mount to host filesystem detected (proc=%proc.name container=%container.info
    target=%fd.name)
  priority: CRITICAL
  tags: [container, mitre_privilege_escalation, T1611]

- rule: Debugfs Access Inside Container
  desc: debugfs is mounted inside a container
  condition: >
    mount and container and fd.name startswith /sys/kernel/debug
  output: >
    debugfs accessed from container (proc=%proc.name container=%container.info)
  priority: CRITICAL
  tags: [container, mitre_discovery, T1087]

- rule: Write Below /etc/cron
  desc: Writing to cron files from container — persistence mechanism
  condition: >
    open_write and container and
    (fd.name startswith /etc/cron or fd.name endswith crontab)
  output: >
    Cron write detected (proc=%proc.name file=%fd.name container=%container.info)
  priority: WARNING
  tags: [container, mitre_persistence, T1053]

Falco in Kubernetes: DaemonSet Configuration with Helm

The recommended way to deploy Falco in production is the official Helm chart. It configures Falco as a DaemonSet (one pod per node), manages the driver selection, and configures falcosidekick as a sidecar. The Helm chart supports all three drivers, gRPC output, and integration with the Kubernetes API for enriching alerts with namespace and pod labels.

Critical Helm values: driver.kind (modern_ebpf, ebpf, or kernel_module), falco.grpc.enabled (true for falcosidekick integration), falco.grpc.unixSocketPath (path shared with sidecar), falco.dryRun (true for initial validation), falco.rulesFile (paths to custom rules), and resources (Falco needs 200-500m CPU and 512Mi-1Gi memory per node).

For the falcosidekick sidecar: falcosidekick.enabled: true, falcosidekick.config with your destination config. For high-volume clusters (100+ nodes), consider deploying falcosidekick as a separate Deployment with a TCP listener instead of a sidecar per DaemonSet pod.

Testing your deployment: kubectl exec -it -n falco <falco-pod> -- falco --list-events shows live syscall events. Trigger a test alert: docker run --rm -it --privileged alpine:latest sh -c 'cat /etc/shadow' on a node — Falco should emit a 'Read sensitive file untrusted' alert within seconds.

# io.thecodeforge — DevOps tutorial

driver:
  kind: modern_ebpf

falco:
  grpc:
    enabled: true
    unixSocketPath: /var/run/falco/falco.sock
  rulesFile:
    - /etc/falco/rules.d/custom-rules.yaml
  dryRun: false

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/T00/B00/XXXX"
      minimumpriority: "warning"
    pagerduty:
      routingkey: "key-here"
      minimumpriority: "critical"

resources:
  requests:
    cpu: 200m
    memory: 512Mi
  limits:
    cpu: 500m
    memory: 1Gi

tolerations:
  - operator: Exists

nodeSelector:
  kubernetes.io/os: linux

Incident Response with Falco: From Alert to Containment

Falco alerts are just the beginning. An effective incident response workflow turns those alerts into actions. The lifecycle: Falco detects anomalous behavior → alert routed via falcosidekick → responder receives notification → initial triage → containment → investigation → remediation.

Containment options: automatically cordon the node (safe, disruptive), annotate the pod (soft isolation), or kill the pod (fastest). For confirmed container escapes: immediately cordon the node, kill the offending container, and capture a forensic snapshot. For suspicious-but-not-confirmed alerts: add a network policy that restricts the pod's egress to only known endpoints, then investigate.

Falco integrates with notification platforms for the IR workflow. PagerDuty: CRITICAL alerts create incidents with the Falco output as the incident body. Slack: WARNING and INFO alerts go to #ops or #falco-alerts. Webhooks: send alerts to a custom IR tool (TheHive, Shuffle, Tines) for automated playbooks.

Automation: use falcosidekick's webhook output to trigger a serverless function that kills pods matching specific rules. For example: if Falco detects a crypto miner, automatically kill the pod and post in #security-incidents. This reduces mean-time-to-containment from minutes to seconds. But be careful — automated kill actions can become a DoS vector if an attacker triggers Falco alerts intentionally.

#!/bin/bash
# Auto-response script triggered by falcosidekick webhook

alert_payload=$(cat)

POD_NAME=$(echo "$alert_payload" | jq -r '.output_fields.k8s.pod.name')
NAMESPACE=$(echo "$alert_payload" | jq -r '.output_fields.k8s.ns.name')
RULE=$(echo "$alert_payload" | jq -r '.rule')
PRIORITY=$(echo "$alert_payload" | jq -r '.priority')

# Auto-kill for CRITICAL crypto miner or reverse shell
if [[ "$PRIORITY" == "CRITICAL" ]] && \
   ([[ "$RULE" == *"Crypto Miner"* ]] || [[ "$RULE" == *"Reverse Shell"* ]]); then
    echo "[AUTO-KILL] Killing pod $POD_NAME in $NAMESPACE (rule: $RULE)"
    kubectl delete pod "$POD_NAME" -n "$NAMESPACE" --grace-period=30

    curl -X POST -H 'Content-type: application/json' \
      --data "{\"text\":\"Auto-killed pod $POD_NAME in $NAMESPACE for $RULE\"}" \
      https://hooks.slack.com/services/T00/B00/YYYY
fi

# Soft isolation for WARNING
if [[ "$PRIORITY" == "WARNING" ]]; then
    kubectl annotate pod "$POD_NAME" -n "$NAMESPACE" \
      "falco.incident=true" "falco.rule=$RULE" --overwrite
fi