Kubernetes RBAC — Service Accounts, Roles, and Bindings

Welcome to Kubernetes RBAC — Service Accounts, Roles, and Bindings. We'll break this down from first principles.

Why RBAC Matters in Production

In production Kubernetes clusters, security is non-negotiable. Without RBAC, any user or service account with network access to the API server can perform any action — including deleting namespaces, reading secrets, or modifying deployments. RBAC provides granular control by binding roles (sets of permissions) to subjects (users, groups, or service accounts). This prevents lateral movement and limits blast radius. For example, a CI/CD pipeline should only be able to deploy to its own namespace, not touch production secrets. RBAC is the primary defense against insider threats and compromised credentials. Always enable RBAC from day one; retrofitting it is painful and risky.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-apiserver-config
  namespace: kube-system
data:
  kube-apiserver.yaml: |
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - command:
        - kube-apiserver
        - --authorization-mode=RBAC,Node
        - --enable-admission-plugins=NodeRestriction
        image: registry.k8s.io/kube-apiserver:v1.28.0

Core Concepts: Subjects, Roles, and Bindings

RBAC has three core components: Subjects (who), Roles (what), and RoleBindings (binding). Subjects can be users (external), groups, or ServiceAccounts (internal to the cluster). Roles define a set of rules — verbs (get, list, watch, create, update, patch, delete) on resources (pods, secrets, deployments, etc.). RoleBindings associate a Role with a Subject within a namespace. ClusterRoles and ClusterRoleBindings work cluster-wide. Understanding these building blocks is essential. A Role is scoped to a namespace; a ClusterRole is not. Use Roles for namespace-scoped permissions and ClusterRoles for cluster-scoped resources (nodes, PVs, namespaces) or for reuse across namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

ServiceAccounts: Identity for Pods

ServiceAccounts are Kubernetes identities for pods. Every namespace has a default ServiceAccount, but it's best practice to create dedicated ones. Pods authenticate to the API server using a token mounted as a volume (usually at /var/run/secrets/kubernetes.io/serviceaccount/token). ServiceAccounts are subjects in RBAC bindings. Use them to give pods specific permissions — e.g., a pod that needs to list pods should have a ServiceAccount bound to a role with list pods permission. Never use the default ServiceAccount for production workloads; it often has no permissions or too many. Create a ServiceAccount per application or microservice.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  namespace: my-app
---
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  namespace: my-app
spec:
  serviceAccountName: my-app-sa
  containers:
  - name: my-app
    image: my-app:latest

Creating Roles and RoleBindings

Roles and RoleBindings are namespace-scoped. To create a Role, define rules with apiGroups, resources, and verbs. For core resources (pods, services), apiGroups is empty string "". For apps (deployments, statefulsets), apiGroups is "apps". RoleBindings link a Role to a subject (User, Group, or ServiceAccount). The subject must include the kind and name. For ServiceAccounts, the namespace must match the ServiceAccount's namespace. You can bind multiple subjects to one Role, but it's cleaner to create separate bindings for clarity.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: my-app-sa
  namespace: my-app
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

ClusterRoles and ClusterRoleBindings

ClusterRoles are not bound to a namespace. They can grant permissions to cluster-scoped resources (nodes, PVs, namespaces) or be reused across namespaces via RoleBindings. ClusterRoleBindings bind subjects to ClusterRoles cluster-wide. Use ClusterRoles for: (1) permissions on cluster-scoped resources, (2) permissions that should be available in all namespaces (e.g., a 'view' ClusterRole). Be careful with cluster-admin — it grants full access. Never bind a ServiceAccount to cluster-admin unless absolutely necessary. Prefer using ClusterRoles with specific permissions and binding them via RoleBindings in specific namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-reader
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: namespace-reader-binding
subjects:
- kind: User
  name: alice
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: namespace-reader
  apiGroup: rbac.authorization.k8s.io

Verbs and Resources: The Permission Grammar

RBAC rules consist of apiGroups, resources, and verbs. Verbs: get, list, watch, create, update, patch, delete, deletecollection. Resources: pods, services, deployments, secrets, configmaps, etc. You can also use subresources like pods/log, pods/exec. Wildcards: '' for all verbs or resources. Use wildcards sparingly. For apiGroups, '' matches all API groups. Be explicit to avoid over-permission. For example, to allow reading pods and logs: resources: ["pods", "pods/log"], verbs: ["get", "list"].

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: dev-operator
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["batch"]
  resources: ["jobs", "cronjobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Aggregation and Default ClusterRoles

Kubernetes ships with default ClusterRoles: view, edit, admin, cluster-admin. These are aggregated — they combine multiple rules. For example, 'view' grants read-only access to most resources, 'edit' adds write access, 'admin' adds full control within a namespace (except ResourceQuota and namespace itself), and 'cluster-admin' is superuser. You can create custom aggregated ClusterRoles by adding labels to your ClusterRoles. This is useful for building role hierarchies. However, be aware that default roles may change across versions. For production, define your own roles rather than relying solely on defaults.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-view
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
- apiGroups: ["stable.example.com"]
  resources: ["crontabs"]
  verbs: ["get", "list", "watch"]

Testing RBAC with kubectl auth can-i

Before deploying, test RBAC permissions using 'kubectl auth can-i'. This command checks if a user or ServiceAccount can perform an action. Use it to validate your roles. Example: 'kubectl auth can-i get pods --as=system:serviceaccount:my-app:my-app-sa'. You can also use '--list' to see all permissions. This is invaluable for debugging permission issues. Always test with the exact subject and namespace. Remember that 'can-i' checks against the current kubeconfig context; use '--as' to impersonate.

# Check if ServiceAccount can list pods in namespace default
kubectl auth can-i list pods \
  --as=system:serviceaccount:my-app:my-app-sa \
  --namespace=default

# List all permissions for a user
kubectl auth can-i --list \
  --as=alice \
  --namespace=default

Auditing RBAC with kubectl describe and get

Audit your RBAC configuration regularly. Use 'kubectl describe rolebinding' and 'kubectl describe clusterrolebinding' to see subjects and roleRefs. Use 'kubectl get roles --all-namespaces' to list all roles. For a comprehensive view, use tools like 'rbac-tool' or 'kubectl rbac'. Look for overly permissive roles, unused bindings, and subjects with cluster-admin. Remove stale ServiceAccounts and bindings. Audit logs from the API server can also show denied requests due to RBAC — monitor them to detect misconfigurations or attacks.

# List all ClusterRoleBindings
kubectl get clusterrolebindings -o wide

# Describe a specific RoleBinding
kubectl describe rolebinding pod-reader-binding -n default

# List all roles in all namespaces
kubectl get roles --all-namespaces

# Check for subjects with cluster-admin
kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name=="cluster-admin") | .subjects'

Common Pitfalls and How to Avoid Them

Pitfall 1: Using the default ServiceAccount. Always create dedicated ones. Pitfall 2: Overusing cluster-admin. It's rarely needed. Pitfall 3: Forgetting to specify apiGroups. For core resources, apiGroups is "". Pitfall 4: Using '*' for verbs or resources without understanding the blast radius. Pitfall 5: Binding a ClusterRole to a ServiceAccount via ClusterRoleBinding when you only need namespace-scoped access. Use RoleBinding instead. Pitfall 6: Not testing permissions before deployment. Use 'can-i'. Pitfall 7: Ignoring subresource permissions. For example, 'pods/exec' is separate from 'pods'. Pitfall 8: Not cleaning up old bindings. They accumulate and become a security risk.

# BAD: Using default ServiceAccount with cluster-admin
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-cluster-admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

RBAC Best Practices for Production

1. Enable RBAC and the Node authorizer. 2. Use dedicated ServiceAccounts per application. 3. Apply least privilege: start with minimal permissions and add as needed. 4. Use namespaces to isolate environments (dev, staging, prod) and apply different roles. 5. Use ClusterRoles for reusable permissions but bind them via RoleBindings. 6. Regularly audit RBAC resources. 7. Use external authentication (OIDC, LDAP) for users, not static tokens. 8. Rotate ServiceAccount tokens periodically. 9. Monitor API server audit logs for RBAC failures. 10. Document your RBAC design and review it with your team.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: production-reader
rules:
- apiGroups: [""]
  resources: ["pods", "services", "endpoints"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitoring-sa
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: monitoring-reader
  namespace: production
subjects:
- kind: ServiceAccount
  name: monitoring-sa
  namespace: production
roleRef:
  kind: Role
  name: production-reader
  apiGroup: rbac.authorization.k8s.io

Troubleshooting RBAC Issues

Common symptoms: 'Forbidden' errors when accessing resources. Steps: 1. Verify the subject (ServiceAccount or user) is correct. 2. Check if the Role exists and has the right rules. 3. Ensure the RoleBinding exists and references the correct Role and subject. 4. Use 'kubectl auth can-i' to test. 5. Check if the resource is cluster-scoped or namespace-scoped. 6. Look for typos in apiGroups, resources, or verbs. 7. Check if the ServiceAccount token is valid and mounted. 8. Review API server logs for RBAC decisions. 9. If using OIDC, verify group mappings. 10. Remember that RBAC changes take effect immediately; no restart needed.

# Step 1: Check if ServiceAccount exists
kubectl get sa my-app-sa -n my-app

# Step 2: Check Role
kubectl get role pod-reader -n default

# Step 3: Check RoleBinding
kubectl get rolebinding pod-reader-binding -n default

# Step 4: Test permission
kubectl auth can-i get pods \
  --as=system:serviceaccount:my-app:my-app-sa \
  --namespace=default

# Step 5: Check API server logs (if accessible)
kubectl logs -n kube-system kube-apiserver-<node> | grep -i rbac