AWS AccessDenied — Decode and Fix IAM Fast

Your deploy fails with AccessDenied, the encoded message is a wall of base64, and the engineer's first instinct is to attach AdministratorAccess 'just to unblock' — which works, ships to production, and becomes a $47K incident six months later when those credentials leak. The error names what was denied but never which policy is responsible, so teams guess: they add S3FullAccess for a KMS problem, or loosen a bucket policy when an SCP is the actual wall.

AccessDenied strikes at the intersection of five policy layers: identity policies, resource policies, SCPs, permissions boundaries, and session policies. Any single explicit deny anywhere beats every allow everywhere — which is why adding permissions to a denied principal changes nothing, and why the fix requires finding the deny before granting the allow.

The trap is treating all denials as missing allows. Half of production AccessDenied cases involve an explicit deny (often an SCP nobody on the team remembers), where the correct fix is scoping an exception — not stapling on broader permissions that silently expand blast radius.

By the end of this article you'll decode authorization messages into exact actions and context keys, trace denials through CloudTrail, evaluate the five policy layers in order, and write least-privilege fixes you can verify by replaying the denied call.

The Evaluation Algorithm: Deny Beats Allow, Always

AWS evaluates every request against up to five policy layers and applies three rules in strict order: an explicit deny in any layer wins immediately; otherwise a single explicit allow anywhere permits; otherwise an implicit deny refuses. This ordering is the entire debugging strategy in one paragraph — it means adding allows can only fix implicit denials (missing permissions), never explicit ones (SCPs, boundaries, deny statements). The teams that internalize this stop escalating privileges on day one and start hunting denies instead.

Map each layer to who owns it and what it can do. Identity policies (attached to users, groups, roles) grant allows and occasional denies — your team's normal surface. Resource policies (bucket policies, key policies, trust policies) grant cross-account access and can also deny. SCPs (organization level) can only restrict — they never grant, only filter the maximum available. Permissions boundaries cap a single principal's maximum. Session policies narrow assumed-role sessions. A request needs an allow surviving every filter with no deny anywhere.

The practical consequence: when generous identity policies fail, the answer is never 'more allows' — it's a deny in a layer the team forgot. Check SCPs, boundaries, KMS key policies (S3 writes to KMS-encrypted buckets need key-policy allows too), and trust-policy conditions in that order. Each is a five-minute check that saves days of allow-stacking. Save every simulation command with its verdict in the incident ticket — the allow/deny table becomes the audit trail proving the fix was scoped, and the next engineer inherits evidence instead of folklore.

# Evaluate all layers at once — the verdict names deny vs allow explicitly
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/backup-role \
  --action-names s3:PutObject kms:GenerateDataKey \
  --resource-arns arn:aws:s3:::reports/q3.pdf arn:aws:kms:us-east-1:123456789012:key/abcd \
  --query 'EvaluationResults[*].{action:EvalActionName,decision:EvalDecision,matched:MatchedStatements[*].SourcePolicyId}' \
  --output table
# decision=explicitDeny -> hunt the matched deny (SCP/boundary), stops here
# decision=implicitDeny -> genuinely missing allow, grant least privilege
# decision=allowed      -> policy fine, check conditions/context or propagation delay

# List the deny-capable layers most teams forget
aws organizations list-policies-for-target --target-id 123456789012 --filter SERVICE_CONTROL_POLICY
aws iam get-role --role-name backup-role --query 'Role.PermissionsBoundary'

Decode the Message: From Base64 to Actionable Sentence

The encoded authorization message looks like noise and reads like a verdict. It's a base64 (sometimes gzipped) JSON blob carrying the evaluation context: principal ARN, action, resource, and the condition keys AWS considered. Decoding needs only STS access with the same credentials — no IAM read permissions, no admin, no ticket to the security team. The engineer on call can run it in the first minute of any AccessDenied page.

Read the decoded output for three fields: the action-plus-resource pair (what was attempted), the context block (which condition keys were present or absent — the missing kms:ViaService in this article's incident was the whole diagnosis), and any matched-statement hints. Absent context keys are the classic reveal: policies demanding encryption, VPC endpoint, MFA, or tag conditions fail when callers simply don't send those attributes.

Make decoding muscle memory, not archaeology. Paste the message, decode, pretty-print, and read it before opening the IAM console. Half of all AccessDenied investigations end at this step with a sentence like 'PutObject denied because the call didn't specify KMS encryption' — a caller fix, not a policy fix. Store the decoded JSON in the incident ticket alongside the encoded original — future auditors and the next on-call inherit the full decision context instead of a base64 blob nobody can read.

# Decode in the first minute — needs only STS, no admin, no tickets
aws sts decode-authorization-message \
  --encoded-message 'ENCODED_MESSAGE_PASTE_HERE' \
  --query DecodedMessage --output text | python3 -m json.tool

# Read three things in the output:
# 1. action + resource  -> 's3:PutObject on arn:aws:s3:::reports/q3.pdf'
# 2. context keys       -> which conditions were present vs ABSENT
#    e.g. missing 's3:x-amz-server-side-encryption: aws:kms' = caller fix
# 3. principal          -> confirm WHO was evaluated (often a surprise role)

# One-liner for pipelines: decode straight from a failed command's output
# aws s3api put-object ... 2>&1 | grep -o 'Encoded[^ ]*' > /tmp/enc.txt
# aws sts decode-authorization-message --encoded-message "$(cat /tmp/enc.txt)" --query DecodedMessage --output text

SCPs and Boundaries: The Denies Nobody Remembers

Service Control Policies and permissions boundaries share one trait: they're set once by a security project, then forgotten by everyone who debugs daily IAM. SCPs attach to OUs and accounts, filtering the maximum permissions for everything inside — they grant nothing, only deny-shaped guardrails like 'no unencrypted S3 writes' or 'no leaving these regions.' Boundaries cap individual roles, commonly applied to delegated developer roles. Both produce explicit denies that survive AdministratorAccess, which is exactly why escalation 'mysteriously fails' and teams spiral.

SCP evaluation has quirks worth knowing. FullAWSAccess (the default) allows everything unless another SCP denies — removing it accidentally denies everything, a spectacular self-inflicted outage. Deny statements in any attached SCP apply; allows merely carve the ceiling. And SCP changes propagate in seconds but cached sessions can confuse — always test with fresh credentials after SCP edits.

Treat these layers as inventory, not mystery. List attached SCPs per OU quarterly, document each guardrail's intent and owner, and require security-team review for exceptions rather than silent allow-stacking elsewhere. When the simulator names an SCP deny, the correct fix is usually conforming the caller (encrypt with KMS, stay in-region) — the guardrail is working as designed.

# Inventory the forgotten layers (quarterly review + incident use)
# SCPs affecting this account (walk up: account -> OU -> root)
aws organizations list-policies-for-target --target-id 123456789012 --filter SERVICE_CONTROL_POLICY
aws organizations describe-policy --policy-id p-deny-unencrypted 2>/dev/null

# Permissions boundary on the failing role
aws iam get-role --role-name backup-role --query 'Role.{boundary:PermissionsBoundary}'

# Which SCP statement actually fires? Simulate and read MatchedStatements
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/backup-role \
  --action-names s3:PutObject --resource-arns 'arn:aws:s3:::reports/*' \
  --query 'EvaluationResults[*].{decision:EvalDecision,matched:MatchedStatements}'
# matched[].SourcePolicyType == 'SCP' -> conform the caller, don't stack allows

KMS, Trust Policies, and Cross-Service Denies

A large share of S3-flavored AccessDenied is actually KMS-flavored: writing to a KMS-encrypted bucket requires kms:GenerateDataKey (and reading needs kms:Decrypt) granted in the key policy, not just S3 permissions on the bucket. Teams grant S3FullAccess, watch PutObject still fail, and conclude IAM is broken — while the key policy quietly denies a principal it was never told to trust. The decoded message plus a kms-scoped simulation names this in seconds.

Trust policies are the second hiding spot: AssumeRole denials (with MFA, external-ID, or IP conditions) fail before any permission evaluation, producing AccessDenied on sts:AssumeRole rather than on the target action. Cross-account patterns add resource-policy counterparts — the bucket policy must allow the foreign principal AND the foreign identity policy must allow the action, with no deny on either side. Miss either half and both teams blame each other.

VPC endpoint and tag conditions complete the usual lineup: policies requiring aws:SourceVpce deny traffic that bypasses the endpoint, and attribute-based (tag) conditions deny when principals or resources lack the expected tags. Each manifests as 'worked from my laptop, fails from the VPC' or 'worked yesterday, fails after retagging' — patterns that scream context, not identity.

# KMS key policy: does it trust YOUR principal? (S3 allows are not enough)
aws kms get-key-policy --key-id abcd-1234 --policy-name default \
  --query Policy --output text | python3 -m json.tool | grep -B2 -A6 'backup-role\|GenerateDataKey'
# Missing -> add the role to the key policy, not more S3 permissions

# Trust policy on the role: do your assume conditions actually hold?
aws iam get-role --role-name backup-role --query 'Role.AssumeRolePolicyDocument'
# Check: MFA present? external ID correct? source IP in range?

# Simulate the FULL call including KMS (S3 + key actions together)
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/backup-role \
  --action-names s3:PutObject kms:GenerateDataKey kms:Decrypt \
  --resource-arns 'arn:aws:s3:::reports/*' 'arn:aws:kms:us-east-1:123456789012:key/abcd-1234'

# Replay the real call with encryption the guardrail demands
aws s3api put-object --bucket reports --key verify.txt --body /tmp/v.txt \
  --server-side-encryption aws:kms --ssekms-key-id abcd-1234

CloudTrail: Every Denial Leaves a Signed Confession

CloudTrail logs every denied API call with the principal, action, resource, error code, and source IP — a complete incident history your team can query instead of reconstruct. Group a day's AccessDenied events by role and action and the pattern usually confesses immediately: one role, one action, 100% denied starting at a deploy timestamp means a scoped policy gap or a fresh deny; scattered denies across actions after a rotation means credential or session confusion.

Use the right tool per trail shape. lookup-events covers the last 90 days for quick incident queries from the CLI. Lake queries (or Athena over the S3 archive) handle the long view — 'when did this denial first appear' across months, which separates fresh breaks from always-broken paths nobody exercised. Alert on the metric, not the anecdote: a CloudWatch alarm on AccessDenied counts per critical role pages on night one, not during the restore drill on day eleven.

Correlate denial start times with change history: SCP edits, boundary attachments, key-policy changes, deploy timestamps. The denial that begins within minutes of a change names its cause. This correlation habit turns 'mysterious AccessDenied' into 'the 14:32 SCP edit' in one query — and makes the fix a revert or a scoped exception instead of a week of archaeology.

# Recent denials for one action (90-day lookup-events window)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject \
  --start-time $(date -u -d '1 day ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --query 'Events[*].CloudTrailEvent' --output text | grep -i accessdenied | head -10

# Pattern view: group denials by role + action (confesses in one glance)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject \
  --start-time $(date -u -d '7 days ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --query 'Events[*].CloudTrailEvent' --output text | \
  python3 -c "import sys,json; from collections import Counter; c=Counter((json.loads(l).get('userIdentity',{}).get('arn'), json.loads(l).get('errorCode')) for l in sys.stdin); print('\n'.join(f'{k} x{v}' for k,v in c.most_common(10)))"

# Correlate: what changed when denials started? (Config + deploy timestamps)
# aws configservice get-resource-config-history ... | grep -i policy
# Then: revert the change or scope the exception — not broader allows.

Least-Privilege Fixes You Can Verify by Replay

The correct fix grants the single missing action on the narrowest resource with the required conditions — nothing more. decoded message says s3:PutObject on arn:aws:s3:::reports/* with KMS context? The fix is that action on that prefix, conditioned on the encryption the guardrail demands. Resist the bundle policies (S3FullAccess, PowerUser) — each wildcard permission is future blast radius sold for present convenience, and the $47K leaked-credential incidents always trace back to a 'temporary' broad grant.

Write the policy as code and simulate before attaching. A JSON statement with explicit Action, Resource, and Condition blocks, checked into version control with a comment naming the incident, beats console click-ops that nobody can review. Simulate the exact denied call (with context entries) and demand allowed; simulate adjacent actions (s3:DeleteBucket, kms:*wildcard) and demand deny — proving both that the hole closed and that no new one opened.

Close the loop by replaying the real call with the real principal, then watching CloudTrail for 24 hours. New allows propagate in seconds but cached sessions and edge cases surprise — the replay plus a denial-count alarm is the only honest definition of fixed. Document the decision (why this action, this resource, this condition) so the next engineer inherits understanding, not just JSON.

# Least-privilege fix as code: one action, narrow resource, required condition
cat > backup-put-fix.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "BackupWritesKMSOnly",
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
    "Resource": "arn:aws:s3:::reports/*",
    "Condition": {"StringEquals": {"s3:x-amz-server-side-encryption": "aws:kms"}}
  }]
}
EOF
aws iam put-role-policy --role-name backup-role --policy-name BackupWritesKMSOnly --policy-document file://backup-put-fix.json

# Prove it: the denied call now allowed, adjacent actions still denied
aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:role/backup-role \
  --action-names s3:PutObject s3:DeleteBucket --resource-arns 'arn:aws:s3:::reports/*' \
  --context-entries 'ContextKeyName=s3:x-amz-server-side-encryption,ContextKeyValues=aws:kms,ContextKeyType=string'
# Expect: PutObject=allowed, DeleteBucket=denied. Ship it.