GCP Impersonation Denied: Token Creator Fix

Your deploy script worked yesterday. Today it dies with permission denied on iam.serviceAccounts.getAccessToken, and nothing in the script changed. Nobody rotated a key because there is no key — the script impersonates a service account, and GCP just refused to mint the token. The whole pipeline halts over one missing IAM binding.

Impersonation denials are GCP's keyless auth doing its job: no Token Creator grant, no token. The error fires before your code touches any data, which makes it feel infrastructural when it's really one IAM line. The usual triggers are painfully ordinary — a new team member, a recreated service account, a binding applied at the wrong resource level — and each produces the identical denial.

These failures cluster around onboarding, CI migrations, and security cleanups, because all three reshape who may act as whom. The audit log records every attempt, yet teams rarely look there first.

This guide shows the exact order: confirm your identity, read the service account's policy, grant Token Creator at the right level, verify with a one-line impersonation, and check org policy when bindings look right. You'll see a real incident where a cleanup script revoked a CI binding, plus the governance pattern that keeps impersonation both usable and safe.

How Impersonation Works and Where It Refuses

Service account impersonation lets one identity act as another without handling its keys: your user account (or a CI service account) asks GCP to mint a short-lived token for a target service account, then uses that token. The minting call is GenerateAccessToken, and it checks one permission first: iam.serviceAccounts.getAccessToken on the target account, normally granted through roles/iam.serviceAccountTokenCreator. Fail that check and you get PERMISSION_DENIED before any bucket, dataset, or API is touched.

This ordering is the whole diagnostic story. Because the denial precedes data access, no application log, no data metric, and no retry will illuminate it — the request never reaches the code that would log something useful. Teams accustomed to debugging data permissions keep auditing buckets and datasets while the actual gate sits one layer earlier, at token issuance. Read the error literally: it names the permission and the account, and both are IAM facts you can query in seconds.

The secure shape of impersonation is worth internalizing now. Short lifetimes (minutes, not hours) bound leaked-token damage. Narrow grants (specific humans or workloads on specific accounts) bound who can borrow what. Audit coverage on every GenerateAccessToken call records who borrowed which badge and when. Denials are the system working — the incident is only when legitimate work can't get its grant.

Granting Token Creator at the Right Scope

The Token Creator role is the standard grant because it bundles exactly the impersonation permissions: getAccessToken for access tokens, signBlob and signJwt for signed payloads, and impersonate for the general capability. Describing the role shows the bundle explicitly, which ends debates about whether some narrower permission would suffice — for token minting, getAccessToken is the load-bearing one and Token Creator is its supported package.

Scope the grant to the service account resource, not the project. A project-level Token Creator grant lets the holder impersonate every service account in the project — convenient, and a breach waiting for a reason. The SA-level binding from the snippet limits borrowing to the one account the workload needs. When someone asks for project-wide impersonation, that request is really asking to skip access reviews for every future account — route it through security, not around them.

Verify from the shell before touching code. The impersonated print-access-token call mints a real token with the real binding chain, succeeding or failing exactly as your application would. A 300-second lifetime keeps the test token nearly harmless. Make this the first and last step of every impersonation change: reproduce with one line, fix, verify with the same line.

# What permissions does Token Creator actually bundle?
gcloud iam roles describe roles/iam.serviceAccountTokenCreator --format="json(name, includedPermissions)"

# Grant impersonation on THIS service account only (least privilege)
gcloud iam service-accounts add-iam-policy-binding <SA_EMAIL> \
  --member=user:<DEV_EMAIL> \
  --role=roles/iam.serviceAccountTokenCreator

# Verify with a real (short-lived) impersonated token
 gcloud auth print-access-token --impersonate-service-account=<SA_EMAIL> --lifetime=300s | cut -c1-20

Member Strings: the Wrong Email Looks Right

Member strings cause a disproportionate share of these denials because they look right while being wrong. user:dev@company.com, group:eng@company.com, and serviceAccount:ci@project.iam.gserviceaccount.com are three different principals, and the binding must name the one actually authenticating. The classic failure pairs a binding for a human with a workload authenticating as a service account, or a personal email with a contractor-domain login. The console shows a healthy-looking binding; the check fails because the caller isn't that member.

Groups add a propagation wrinkle: adding someone to a granted group can take minutes to take effect, so freshly onboarded engineers fail impersonation until propagation completes. Service accounts in groups behave the same way. When onboarding and impersonation coincide, wait, retry, and check timestamps before redesigning grants that are actually fine.

The discipline is comparing exact strings. Dump the SA policy, dump the active auth identity, and diff them character by character — domains, dots, and service-account suffixes included. The snippet below performs both dumps. Most member mysteries resolve the moment both strings sit side by side, because the difference is usually visible to anyone who looks at both at once.

# Who are you right now? The ACTIVE line is the caller
 gcloud auth list

# Who may impersonate this service account? Compare member strings exactly
gcloud iam service-accounts get-iam-policy <SA_EMAIL> --format="json(bindings)"

# For CI callers: which account does the runner authenticate as?
gcloud auth print-identity-token --impersonate-service-account=<SA_EMAIL> 2>&1 | head -c 200; echo

When Org Policy Overrides Your Perfect Binding

When bindings are correct and denials persist, look up the hierarchy: organization and folder policies can restrict credential issuance regardless of IAM grants. The symptom is distinctive — get-iam-policy shows a perfect binding, the one-line impersonation test still fails, and the audit log records denials citing policy rather than missing permission. No grant you add at the project level overrides a higher-level deny.

List the effective policies for the project and walk upward to folder and organization when something looks restrictive. Changes here need security-team partnership: the policy exists because someone decided impersonation should be limited, and your workload needs an exception with a documented scope, not a quiet deletion. Bring the audit-log evidence showing legitimate blocked work — policy owners respond to denied-business-impact faster than to grant requests.

Test policy changes in a sandbox project first. Organization policy edits propagate across everything beneath them, so a fix for one workload can open (or close) impersonation for hundreds. Sandbox validation plus a staged rollout keeps the exception tight. Record the exception next to the binding in infrastructure code so the next auditor sees intent, not accident.

# Effective org policies on this project (walk up to folder/org if restrictive)
gcloud resource-manager org-policies list --project=<PROJECT_ID> --format="table(constraint, listPolicy.allowedValues)"

# Denials citing policy, not missing permission: pull the exact entries
 gcloud logging read 'protoPayload.methodName="GenerateAccessToken" AND protoPayload.status.code=7' \
  --project=<PROJECT_ID> --limit=20 --format="table(protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)"

Chained Impersonation: Testing Every Link

Service-account-to-service-account chains multiply both power and confusion. CI runs as account A, impersonates deployer B, which impersonates data-loader C — each hop needing Token Creator on the next. A denial at hop two looks identical to a denial at hop one unless you test each link separately with the one-line impersonation command, authenticating as each intermediate. Test the chain link by link instead of end to end.

Keep chains short and lifetimes shorter. Every hop adds token-exchange latency and another binding an attacker can exploit. Two hops cover nearly every legitimate pattern (human to CI to workload); three hops deserve a design review asking why the middle account exists. Set explicit short lifetimes per hop so a token intercepted mid-chain expires before it's useful.

Prefer federation where the chain exists only to cross boundaries. Workload Identity Federation lets external CI (GitHub Actions, for example) mint short-lived GCP credentials directly, removing the first hop entirely. Fewer hops mean fewer bindings, fewer denials, and fewer 2 AM pages about the middle link. Chains are a tool for residual cases, not the default architecture. Document the chain shape in the runbook so the next debugger tests links instead of re-reading the whole pipeline.

# Test each hop separately: authenticate as A, mint for B
 gcloud auth print-access-token --impersonate-service-account=<B_SA> --lifetime=300s >/dev/null && echo "hop A->B OK"

# Then as B (via impersonation) mint for C: chains need every link
gcloud auth print-access-token --impersonate-service-account=<B_SA>,<C_SA> --lifetime=300s >/dev/null && echo "chain A->B->C OK"

# Who impersonated whom recently? Audit the chain links
 gcloud logging read 'protoPayload.methodName="GenerateAccessToken"' --project=<PROJECT_ID> --limit=30 \
  --format="table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)"

Governing Impersonation So It Stays Safe

Durable prevention treats impersonation as governed infrastructure. Define each service account and its Token Creator bindings in the same infrastructure module, with members resolved from groups rather than hardcoded emails. A CI check diffs live SA policies against that desired state nightly; drift pages the owning team, not the next deploy's victim. Manual console grants become the exception that expires, not the rule that lingers.

Layer audit coverage proportional to power. Every GenerateAccessToken call on production data accounts logs to a monitored sink with alerts on both anomalies: denials spiking (attacks or breakage) and successes from unexpected callers (compromise or scope creep). Break-glass bindings carry short expiries and require incident-linked justification. Reviews happen quarterly against the live policy dumps, not against someone's memory of the intent.

Onboard people and workloads through the same front door: request the binding, record the reason, set the expiry, verify with the one-line test. When the next denial lands, the responder checks live state against declared state and finds the gap in minutes — because the declared state exists. Impersonation stays both usable and safe exactly to the extent that its grants are written down, watched, and expired.