S3 SignatureDoesNotMatch — Fix SigV4 Signing Fast

Your S3 uploads worked yesterday. Today every PutObject fails with SignatureDoesNotMatch, and the error message helpfully shows two long base64 strings that look identical but aren't. No code shipped, no keys changed (that anyone admits to), and the bucket policy is untouched. The signing math is deterministic — same inputs, same signature — so something in your inputs drifted.

This error loves credential rotations, fresh EC2 instances with unsynced clocks, and filenames with plus signs, tildes, or Unicode characters. It also haunts presigned URLs generated in one region and used against another, and SDK calls where a proxy quietly rewrites the query string after signing. Each cause produces the identical error message, which is why teams burn hours rechecking the key that's actually fine.

The trap is regenerating keys as the first move. New keys fix a rotated-secret cause but change nothing for clock skew, encoding, or region bugs — and now you've invalidated every other service sharing those credentials. Diagnose before you rotate.

By the end of this article you'll run the 30-second identity-plus-clock check, decode what AWS expected via the StringToSign, fix each of the four root causes, and build signing code that survives rotations, special characters, and multi-region deployments.

How SigV4 Turns Four Inputs Into One Signature

SigV4 is a chain of HMACs, and every link names its inputs explicitly. First your SDK builds the canonical request: HTTP method, canonical URI, canonical (sorted, encoded) query string, canonical headers, signed-header list, and the payload hash. That blob is hashed, then embedded in the StringToSign alongside the algorithm name, the request timestamp, and the credential scope (date, region, service, terminator). Finally the signing key — derived by HMAC-chaining your secret with date, region, and service — signs that string. Change any input bit and the output is unrecognizably different; that's cryptographic design, not flakiness.

This determinism is your debugging superpower: identical inputs always reproduce identical signatures. When AWS rejects you, one of your inputs differs from what AWS used — and AWS's inputs are ground truth (its stored secret for your key, its clock, the region the bucket lives in, the bytes it actually received). Your job is re-deriving each of your four inputs independently until you find the liar.

The error message's two strings are the two HMAC outputs — yours and AWS's — and comparing them character-by-character is theater. They differ everywhere because HMAC avalanches; position of first difference carries zero information. Ignore the strings, interrogate the inputs.

# Reproduce the signing inputs independently (no SDK involved)
# 1. What AWS thinks your identity is (proves key validity in 5s)
aws sts get-caller-identity --query Arn --output text

# 2. What your clock says vs reality (SigV4 tolerance: 15 min)
date -u
chronyc tracking 2>/dev/null | grep -E 'System time|Leap status|Stratum'
# Want: System time offset < 1s. Suspect: anything over ~60s.

# 3. What region the bucket actually lives in (signature scope must match)
aws s3api get-bucket-location --bucket my-bucket --query LocationConstraint
# null = us-east-1. Your signer must use exactly this region.

# 4. What bytes S3 received (enable SDK debug to capture canonical request)
# boto3.set_stream_logger('botocore', logging.DEBUG)  -> look for
# 'CanonicalRequest' and 'StringToSign' lines, diff vs a working host

Wrong or Rotated Secrets: Verify Before You Rotate

Secret problems come in three flavors: the key was rotated (old secret deleted, app still holds it), the key was copied wrong (trailing newline, truncated env var, shell escaping mangled a slash or plus), or the app loads a different credential than you think (stale profile, expired SSO token, instance-profile race during rotation). All three produce identical SignatureDoesNotMatch, and only the first is fixed by rotation — the other two survive new keys untouched.

Verify before rotating with the cheapest possible probe: aws sts get-caller-identity using the exact credentials the app loads (same env vars, same profile, same container). Success proves the pair is live and correctly copied, eliminating secrets from suspicion in seconds. Failure with InvalidClientTokenId names the access key as the problem; failure with SignatureDoesNotMatch on STS itself implicates the secret bytes or the clock — check both.

Audit the credential path, not just the value. Dump aws configure list to see which profile, env, or role actually supplies credentials, check for a trailing %0A from a copy-paste newline, and confirm SSO tokens haven't expired (sso sessions silently go stale and some SDKs fall back to older cached keys). When rotation is genuinely needed, generate the new pair first, deploy everywhere, verify, then delete the old — overlap, never gap.

# Prove the EXACT credentials the app uses (same env, same container)
# Dump the resolution chain first — which source actually wins?
aws configure list
# Shows: access_key from [env|shared-credentials-file|iam-role], region source

# Test with the app's literal environment (copy from the pod/task def)
env -i AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
  aws sts get-caller-identity --region us-east-1
# Returns ARN -> keys live and correctly copied. Suspect clock/region/encoding.
# InvalidClientTokenId -> access key wrong/deleted. SignatureDoesNotMatch here -> secret bytes or clock.

# Catch the classic copy-paste injuries
printf '%s' "$AWS_SECRET_ACCESS_KEY" | wc -c   # must be 40 chars, no trailing newline
printf '%s' "$AWS_SECRET_ACCESS_KEY" | grep -q $'\n' && echo 'TRAILING NEWLINE FOUND'

# Rotation done right: create second pair, deploy, verify, THEN delete old
aws iam create-access-key --user-name uploader --query 'AccessKey.[AccessKeyId,SecretAccessKey]'

Clock Skew Past 15 Minutes: The Silent Signature Killer

SigV4 bakes the request timestamp into the StringToSign, and AWS rejects any request whose timestamp differs from its own clock by more than 15 minutes. This is replay-attack protection, and it makes every unsynced host in your fleet a signature-failure machine — while HTTP, DNS, database, and all other traffic work perfectly. The selectivity is what makes clock skew so misleading: 'everything works except S3' reads like a credentials problem, not a time problem.

Fresh instances, new container hosts, and laptops off the corporate VPN are the classic victims. The incident pattern is unmistakable in hindsight: old hosts fine, new hosts failing, identical code and keys. AMI baking commonly masks chrony, Docker hosts without time-sync drift on suspend/resume, and some minimal images ship no NTP client at all — inheriting whatever the hypervisor clock says.

Build time-sync into the platform, not the runbook. Images should assert sub-second offset at boot and refuse traffic until synced; orchestrators should expose clock offset as a health signal; and the signing-failure runbook must list the clock check before key rotation, with rotation gated behind it. A one-line date comparison during the incident would have saved the 18,000 failed uploads in this article's story.

# The 30-second clock verdict (run on the FAILING host, not your laptop)
date -u                                              # what the host believes
curl -s http://169.254.169.254/latest/meta-data/ 2>/dev/null | head -1  # EC2 sanity
chronyc tracking 2>/dev/null | grep -E 'System time|Leap|Stratum'
ntpdate -q pool.ntp.org 2>/dev/null | grep -i offset
# Verdict: offset > 60s -> sync NOW and retry before touching keys

# Emergency resync, then immediate retry of the failing call
sudo chronyc -a makestep        # chrony: step the clock immediately
# or: sudo ntpdate -s pool.ntp.org  (ntpdate hosts)
aws s3api head-bucket --bucket my-bucket && echo 'SIGNING AGAIN'

# Permanent: boot-time gate in user-data / entrypoint (fail loud, never skewed)
# offset=$(chronyc tracking | awk '/System time/ {print $4}');
# awk "BEGIN{exit !($offset > 1.0)}" && { echo 'CLOCK UNSYNCED'; exit 1; }

Special Characters and Double Encoding in Keys and URLs

S3 object keys with spaces, plus signs, tildes, or non-ASCII characters are the encoding minefield of SigV4. The canonical URI must encode each path segment exactly once with a precise safe set — and the classic failures are double-encoding (%2B becoming %252B when two layers each encode), plus-as-space confusion (form-encoding + versus path-encoding %20), and SDK-versus-proxy disagreements where a proxy normalizes the URL after your code signed the original. Each produces a signature over bytes AWS never received.

Presigned URLs concentrate every encoding risk into one string that passes through browsers, email clients, and chat apps — any of which may re-encode or wrap it. A presigned URL generated with one encoding and fetched with another fails exactly like a bad key. The diagnostic is byte comparison: log the canonical request your signer built, capture the actual request line S3 received (SDK debug logging shows both), and diff them. They must match byte-for-byte; the first differing byte names the layer that re-encoded.

Standardize encoding at the SDK boundary and never hand-roll it. Use your language's canonical quoter with the S3 safe set, generate presigned URLs server-side with the SDK (not string templates), and add integration tests with adversarial filenames — spaces, plus, tilde, Unicode, and a literal % — so the next encoding regression fails in CI instead of production.

# Byte-compare what you signed vs what S3 received
python3 - <<'EOF'
from urllib.parse import quote
key = 'photos/vacation 2026/a+b~c%percent.txt'
# S3 canonical URI: encode each segment, safe set is / ~ (plus alphanumerics)
canonical = '/'.join(quote(seg, safe='~') for seg in key.split('/'))
print('sign this:', canonical)
# sign this: photos/vacation%202026/a%2Bb~c%25percent.txt
# If your wire log shows %252B or a literal '+', that layer re-encoded.
EOF

# Generate presigned URLs with the SDK — never string templates
aws s3 presign 's3://my-bucket/photos/vacation 2026/a+b~c%percent.txt' --expires-in 900

# Fetch verbosely and confirm the params arrive unmangled
curl -v "$(aws s3 presign 's3://my-bucket/photos/vacation 2026/a+b~c%percent.txt' --expires-in 900)" -o /dev/null 2>&1 | grep -E '^> GET|HTTP/'
# Adversarial CI filenames: 'a b', 'a+b', 'a~b', 'caf\u00e9', '100%.txt'

Region and Endpoint Mismatches That Masquerade as Bad Keys

The credential scope embeds the region, so a request signed for us-west-2 and sent to a us-east-1 bucket fails with the same SignatureDoesNotMatch as a wrong secret. The usual setups: a client defaulting to us-east-1 against a bucket that lives elsewhere, a presigned URL minted in one region and redeemed in another, or an S3-compatible endpoint (MinIO, LocalStack, VPC endpoint) where the SDK signs for 's3' in one region while the endpoint expects another. Multi-region deployments hit this the moment traffic shifts.

Diagnose with ground truth, not config archaeology: ask AWS where the bucket lives (get-bucket-location; null means us-east-1), then check what region your signer used (SDK debug shows the credential scope line: date/region/s3/aws4_request). Mismatched strings are a complete diagnosis. Also verify the endpoint hostname matches the signing style — path-style versus virtual-hosted-style addressing changes the canonical URI, and mixing styles between signer and sender breaks signatures identically.

Fix by making region explicit everywhere: client constructors with the bucket's region, presigned-URL generation in the bucket's region, and per-bucket clients in multi-region apps rather than one global client. Environment-variable region leaks (AWS_REGION set for one service, inherited by another) are a recurring source — scope region config to the S3 client, not the process.

# Ground truth: where does the bucket live vs what did you sign for?
aws s3api get-bucket-location --bucket my-bucket --query LocationConstraint
# null -> us-east-1. Anything else -> use EXACTLY that string.

echo "env regions: AWS_REGION=$AWS_REGION AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION"
grep -rn 'region' ~/.aws/config 2>/dev/null

# SDK debug exposes the scope AWS will check (must match bucket region)
# boto3.set_stream_logger('botocore', logging.DEBUG)
# Look for: 'CredentialScope: 20260923/us-west-2/s3/aws4_request'
# If scope says us-west-2 but bucket lives in us-east-1 -> mismatch found.

# Fix: explicit per-bucket client, explicit presign region
python3 - <<'EOF'
import boto3
s3 = boto3.client('s3', region_name='us-east-1')  # bucket's real region
url = s3.generate_presigned_url('put_object',
    Params={'Bucket': 'my-bucket', 'Key': 'uploads/a b.txt'}, ExpiresIn=900)
print(url)
EOF

SDK Signing Pitfalls: Proxies, Versions, and Credential Chains

Modern SDKs sign correctly out of the box — failures come from what surrounds them. Egress proxies that rewrite query strings or normalize paths invalidate signatures computed pre-proxy; test by comparing a direct SDK call against an identical proxied one. SDK upgrades occasionally change default signing behavior (addressing style, payload-signing for streaming, checksum headers added to the signed set); pin versions and read changelogs when failures coincide with a bump. Custom middleware that adds headers after signing (or strips signed ones) breaks the signed-headers contract just as thoroughly.

Credential chains add a second failure class that wears the same error costume. SSO sessions expire, instance metadata serves briefly inconsistent keys during rotation, and shared config files get edited by automation that appends duplicate profiles. When failures are intermittent and host-correlated, suspect the chain: dump the resolved source (aws configure list), check SSO token freshness, and correlate failure windows with rotation timestamps.

Harden the signing path as infrastructure: wire-log canonical requests in staging, pin SDK versions with lockfiles, forbid proxies from mutating S3 URLs (CONNECT tunneling instead of rewriting), and retry signing failures once after refreshing credentials — absorbing rotation transients without paging anyone.

# Intermittent failures? Suspect the chain and the path, not the key
aws configure list                      # which source actually supplies creds?
aws sso login --profile uploader 2>&1 | head -2   # refresh stale SSO tokens

# Direct vs proxied: identical call, two paths — divergence indicts the proxy
env -u https_proxy -u HTTPS_PROXY python3 s3probe.py   # direct: expect OK
python3 s3probe.py                                     # proxied: fails = proxy rewrites

# Capture what was actually signed (staging + incident hosts)
python3 - <<'EOF'
import logging, boto3
boto3.set_stream_logger('botocore', logging.DEBUG)
s3 = boto3.client('s3', region_name='us-east-1')
s3.head_bucket(Bucket='my-bucket')
# Diff 'CanonicalRequest:' and 'StringToSign:' blocks between working/failing hosts
EOF

# Pin signing behavior: lockfiles + changelog check on every SDK bump
# pip freeze | grep -i boto ; npm ls @aws-sdk/signature-v4