Kubernetes Security — Pod Security Standards and PSA

Welcome to Kubernetes Security — Pod Security Standards and PSA. We'll break this down from first principles.

Why Pod Security Standards Replaced PodSecurityPolicy

PodSecurityPolicy (PSP) was notoriously complex and easy to misconfigure. Many teams either left PSP disabled or created overly permissive policies that provided no real security. The Kubernetes community deprecated PSP in v1.21 and removed it in v1.25. Pod Security Standards (PSS) and Pod Security Admission (PSA) were designed as a simpler, more enforceable replacement. PSS defines three clear policy levels: Privileged (no restrictions), Baseline (minimal restrictions for common workloads), and Restricted (hardened for security-sensitive workloads). PSA is the admission controller that enforces these levels per namespace. The key improvement is that PSA is built into kube-apiserver, requires no webhook, and is configured via namespace labels. This reduces operational overhead and eliminates the 'PSP sprawl' problem where hundreds of policies became unmanageable.

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

The Three Policy Levels: Privileged, Baseline, Restricted

Privileged is the least restrictive level. It allows all capabilities, host networking, host PID, host IPC, and privileged containers. Use it only for system-level pods like kube-proxy, CNI plugins, or monitoring agents that require host access. Baseline applies minimal restrictions to prevent known privilege escalations. It disallows hostPID, hostIPC, hostNetwork, privileged containers, and most capabilities except NET_BIND_SERVICE. It also requires that seccomp is unconfined or runtime/default. Baseline is the default recommendation for most workloads. Restricted is the most restrictive level. It enforces all Baseline restrictions plus: seccomp must be RuntimeDefault or Localhost, containers must run as non-root (runAsNonRoot: true), and must drop ALL capabilities. It also forbids hostPath volumes and requires that the container's root filesystem is read-only (readOnlyRootFilesystem: true). Restricted is for security-critical workloads like payment processing or handling PII. Choosing the right level requires understanding your workload's needs. Over-restricting can break applications; under-restricting leaves you vulnerable.

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault
      readOnlyRootFilesystem: true

Configuring PSA via Namespace Labels

PSA is configured by adding labels to namespaces. There are three label keys: pod-security.kubernetes.io/enforce, pod-security.kubernetes.io/audit, and pod-security.kubernetes.io/warn. The enforce label blocks pods that violate the policy. The audit label logs violations but does not block. The warn label returns a warning to the user but does not block. Each label's value is the policy level: privileged, baseline, or restricted. You can also use versioned values like restricted-v1.25 to pin to a specific Kubernetes version's policy definition. This is important because policy definitions can change between versions. For example, in v1.25, the restricted level added a requirement for seccomp. If you use restricted without a version, it resolves to the latest version supported by the cluster. To avoid surprises, always pin to a version. You can also use the latest version to always get the latest policy. However, this can break workloads during upgrades. Best practice: pin to the version you've tested against.

apiVersion: v1
kind: Namespace
metadata:
  name: staging
  labels:
    pod-security.kubernetes.io/enforce: baseline-v1.25
    pod-security.kubernetes.io/audit: restricted-v1.25
    pod-security.kubernetes.io/warn: restricted-v1.25

Testing Pod Compliance with kubectl and Dry-Run

Before enforcing a policy, you can test pod compliance using kubectl with dry-run or by checking the audit logs. The kubectl dry-run flag (--dry-run=server) sends the request to the API server but does not persist it. The API server will evaluate the request against the enforce policy and return an error if it violates. However, dry-run does not evaluate warn or audit labels. To see warnings, you must actually create the resource (or use --validate=warn which is not supported for PSA). Instead, use the kubectl apply --server-side --dry-run=server to get enforcement errors. For warnings, you can temporarily set the warn label and create the pod, then delete it. Alternatively, use the kubectl audit command to view audit logs for violations. Another approach is to use the psa-check tool (part of the kubernetes-sigs/security-profiles-operator) to validate manifests offline. This tool can check a set of YAML files against a specified policy level without needing a cluster.

# Test if a pod manifest violates the enforce policy
kubectl apply --dry-run=server -f pod.yaml -n staging

# If violation, you'll see:
# Error from server (Forbidden): pods "my-pod" is forbidden: violates PodSecurity "baseline-v1.25": ...

# To check warnings, create the pod and delete it:
kubectl create -f pod.yaml -n staging && kubectl delete pod my-pod -n staging

Handling Exceptions: Exemptions and Exceptions

Not all pods can comply with a policy. System pods like kube-proxy, CNI plugins, and ingress controllers often require privileged access. PSA provides two mechanisms for exceptions: exemptions and exceptions. Exemptions are configured at the API server level via the --admission-control-config-file flag. They allow you to exempt specific users, groups, or service accounts from PSA enforcement. This is useful for cluster administrators or system components. Exceptions are namespace-level and are configured via the pod-security.kubernetes.io/enforce-version label? Actually, exceptions are not a built-in feature. Instead, you can create a separate namespace with a lower policy level for workloads that need exceptions. For example, create a system namespace with privileged policy for system pods. For user workloads that need exceptions, you can use a baseline namespace instead of restricted. If you truly need to allow a specific pod to violate the policy, you can use the pod-security.kubernetes.io/exempt label? No, that doesn't exist. The correct approach is to use a separate namespace or use the securityContext with seccompProfile: Unconfined? That would violate the policy. The only way to bypass is to exempt the service account at the API server level. This is a deliberate design: PSA forces you to think about namespace boundaries.

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "restricted-v1.25"
      enforce-version: "v1.25"
      audit: "restricted-v1.25"
      audit-version: "v1.25"
      warn: "restricted-v1.25"
      warn-version: "v1.25"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: ["kube-system", "gatekeeper-system"]

Integrating PSA with CI/CD Pipelines

To enforce security early, integrate PSA validation into your CI/CD pipeline. Use kubectl dry-run or psa-check to validate manifests before they reach the cluster. For example, in a GitHub Actions workflow, you can run a step that checks all YAML files against the target namespace's policy. If validation fails, the pipeline fails. This prevents non-compliant pods from being deployed. You can also use tools like kubeconform or kubeval with custom schemas, but they don't understand PSA. The best approach is to use psa-check which directly implements the PSA logic. Another option is to use kubectl apply --dry-run=server against a real cluster, but that requires cluster access in CI. For offline validation, psa-check is ideal. You can also use kustomize with a PSA validator plugin. However, the simplest approach is to run psa-check on all manifests in a PR.

name: PSA Check
on: [pull_request]
jobs:
  psa-check:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Install psa-check
      run: go install sigs.k8s.io/security-profiles-operator/cmd/psa-check@latest
    - name: Validate manifests
      run: |
        for file in $(find . -name '*.yaml' -path '*/deploy/*'); do
          psa-check -policy restricted-v1.25 -file $file
        done

Monitoring PSA Violations with Audit Logs

PSA audit logs are written to the Kubernetes API server audit log. To capture them, you must enable audit logging on the API server. The audit log contains entries with annotations like pod-security.kubernetes.io/audit-violations. You can parse these logs to monitor violations. For example, using kubectl logs on the API server pod (if self-managed) or using cloud provider's log service (e.g., CloudWatch, Stackdriver). You can also use tools like kubectl audit to view recent audit events. However, the audit log can be noisy. To reduce noise, filter by namespace or policy level. You can also set up alerts for specific violations, e.g., when a pod in a 'restricted' namespace violates the policy. This helps you detect misconfigurations or malicious attempts. Another approach is to use the kube-state-metrics with custom metrics for PSA violations, but that requires additional setup.

# Search for PSA violations in audit log (example using jq)
kubectl logs -n kube-system kube-apiserver-<node> --tail=1000 | \
  jq 'select(.annotations."pod-security.kubernetes.io/audit-violations" != null) | {user: .user.username, pod: .objectRef.name, violations: .annotations."pod-security.kubernetes.io/audit-violations"}'

Migrating from PodSecurityPolicy to PSA

Migrating from PSP to PSA requires careful planning. First, audit your existing PSPs to understand which permissions they grant. Use the pss-psp-migration tool to map PSPs to PSS levels. Then, choose a target policy level for each namespace. Start by setting warn and audit labels to see which pods would be affected. Fix any violations by updating pod specs. Once all pods are compliant, set the enforce label. During migration, you can run both PSP and PSA simultaneously (PSP is still available in v1.24). This allows you to gradually transition. However, be aware that if both enforce, the more restrictive one wins. To avoid conflicts, disable PSP after PSA is enforced. The migration should be done namespace by namespace. Start with non-critical namespaces, then move to production. Document the policy level for each namespace.

# 1. Audit existing PSPs
kubectl get psp -o yaml > psp-dump.yaml

# 2. Use pss-psp-migration tool (from kubernetes-sigs/security-profiles-operator)
pss-psp-migration analyze -f psp-dump.yaml

# 3. Add warn and audit labels to namespace
kubectl label ns production pod-security.kubernetes.io/warn=restricted-v1.25
kubectl label ns production pod-security.kubernetes.io/audit=restricted-v1.25

# 4. Check for violations (via audit logs or kubectl)
kubectl get pods -n production -o yaml | grep -i "pod-security"

# 5. Fix violations, then enforce
kubectl label ns production pod-security.kubernetes.io/enforce=restricted-v1.25 --overwrite

Common Pitfalls and How to Avoid Them

One common pitfall is forgetting to set runAsNonRoot: true in the pod spec. Many base images run as root by default. Always specify a non-root user in your Dockerfile or pod securityContext. Another pitfall is using capabilities: add without dropping all capabilities first. The restricted policy requires dropping ALL capabilities. If you add a capability, you must still drop all others. A third pitfall is using hostPath volumes. Restricted policy forbids hostPath. Use emptyDir, configMap, secret, or PVC instead. A fourth pitfall is not setting seccomp profile. Restricted requires RuntimeDefault or Localhost. Many developers forget this. A fifth pitfall is using privileged containers for debugging. Instead, use kubectl debug with ephemeral containers that can have different security contexts. Finally, a common mistake is setting the policy level too high for workloads that don't need it. This causes unnecessary friction. Use Baseline for most workloads, Restricted only for sensitive ones.

# Violation: missing runAsNonRoot
apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
spec:
  containers:
  - name: app
    image: myapp
    # missing securityContext

# Violation: not dropping all capabilities
    securityContext:
      capabilities:
        add: ["NET_ADMIN"]
        # missing drop: ["ALL"]

# Violation: hostPath volume
volumes:
- name: data
  hostPath:
    path: /data

# Fix: add securityContext and remove hostPath

Advanced: Custom Policies with OPA/Gatekeeper

While PSA covers the three standard levels, you may need custom policies. For example, you might want to enforce that all images come from a trusted registry, or that resource limits are set. For these, use OPA/Gatekeeper or Kyverno. Gatekeeper is a Kubernetes admission webhook that uses the Open Policy Agent (OPA) to enforce custom policies. You can write Rego rules to check any aspect of a pod spec. For example, you can enforce that all containers have readOnlyRootFilesystem: true even in baseline namespaces. However, be careful not to duplicate PSA. Use Gatekeeper for policies that go beyond PSS. For example, you can create a constraint that requires all pods to have a label team. Or require that all images are from a specific registry. Gatekeeper can also be used to enforce PSA policies more granularly, but that's redundant. Best practice: use PSA for the three standard levels, and Gatekeeper for custom policies. This keeps your security stack simple and maintainable.

package k8srequiredlabels

violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  not container.securityContext.readOnlyRootFilesystem
  msg := sprintf("Container %v must have readOnlyRootFilesystem: true", [container.name])
}

Performance Impact of PSA

PSA is a built-in admission plugin that runs in the API server. It has minimal performance impact because it only evaluates pod creation and update requests. The evaluation is a simple check against the policy level. Unlike webhooks, there is no network call. The overhead is negligible. However, if you have many namespaces with different policies, the API server must evaluate the labels each time. This is still very fast. In practice, PSA adds less than 1ms to admission latency. The main performance consideration is audit logging. If you enable audit logging for all requests, it can increase disk I/O and storage costs. To mitigate, use a sampling rate or only audit violations. You can configure audit policy to log only requests that match certain criteria. For example, log only requests that violate the audit policy. This reduces log volume significantly.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods"]
  # Only log if violation
  annotations:
    pod-security.kubernetes.io/audit-violations: ".*"
  # Omit users
  omitStages:
  - RequestReceived

Future of Pod Security: What's Next?

The Kubernetes community is working on further improvements to pod security. One area is the integration of PSA with the new SecurityContext fields like seccompProfile and appArmorProfile. Another is the potential for dynamic policy levels based on workload identity. There is also discussion about making PSA more flexible without requiring Gatekeeper. For example, allowing custom profiles that extend the standard levels. However, the current design intentionally limits flexibility to keep it simple. The future may bring a more granular policy model, but for now, PSA is the standard. Another trend is the use of eBPF-based security tools like Cilium Network Policies and Tetragon for runtime security. These complement PSA by enforcing security at the kernel level. PSA is about admission control; runtime security is about monitoring and enforcement during execution. Both are needed for a defense-in-depth strategy.

# Hypothetical future PSA with custom profiles
apiVersion: pod-security.admission.config.k8s.io/v2
kind: PodSecurityConfiguration
profiles:
- name: my-custom
  # ... custom rules