Structured Outputs with LLMs — How a $400k/mo Fraud Pipeline Broke on a Missing Enum Value
When LLM structured outputs fail in production, they fail silently.
20+ years shipping production ML systems and the infrastructure behind them. Lessons pulled from things that broke in production.
- ✓Solid grasp of fundamentals
- ✓Comfortable reading code examples
- ✓Basic production concepts
- JSON Schema Validation Always validate LLM output against your schema before use; a single missing field can crash downstream services.
- Error Handling LLMs will occasionally return valid JSON that violates your schema — treat every response as suspect.
- Retry Logic Implement exponential backoff with schema-aware retries; naive retries amplify costs without fixing root causes.
- Schema Versioning Track schema versions in your prompts and outputs to detect drift when you update models or APIs.
- Monitoring Log raw LLM responses and parsed structures separately; you need both to debug schema violations.
- Fallback Strategies Have a default structured output for when the LLM refuses or fails; empty responses are better than crashes.
Structured outputs are a mechanism to force an LLM to generate responses that conform to a predefined schema — typically JSON with specific fields, types, and constraints like enums, regex patterns, or numeric ranges. They exist because raw LLM text generation is nondeterministic and prone to hallucinating keys, omitting fields, or producing malformed values, which breaks downstream systems that expect strict data contracts.
Under the hood, most implementations (e.g., OpenAI's response_format with json_schema, or local frameworks like Outlines and LMQL) work by constraining the model's token sampling to only valid tokens at each step — either via grammar-based logit masking or by post-processing with a validator and retry loop. This is fundamentally different from prompt engineering, which just hopes the model follows instructions, or function calling, which is a higher-level abstraction that maps tool definitions to structured outputs but often adds latency and overhead.
Structured outputs are the right tool when you need deterministic, parseable data from an LLM — think extracting invoice line items, generating API request bodies, or classifying user intent into a fixed enum — but they're overkill for freeform text generation or creative tasks where schema compliance would degrade output quality. In production at scale (e.g., 10M requests/day), you'll pair them with caching, fallback schemas, and monitoring for schema violations, because even a 0.1% failure rate on a $400k/mo fraud pipeline means a missing enum value can cascade into silent data corruption or revenue loss.
Imagine you ask a chef to write a recipe on a specific form with boxes for ingredients, steps, and time. Sometimes the chef writes the time in the ingredients box or invents a new box called 'magic.' Structured outputs force the chef to use your form exactly. But if the form changes or the chef gets creative, you end up with a recipe that looks right but is useless — and you only find out when the dinner party starts.
| Chrome | Firefox | Safari | Edge |
|---|---|---|---|
| ✓ | ✓ | ✓ | ✓ |
Three months ago, our fraud detection pipeline started silently dropping 12% of transactions. No errors. No alerts. Just a slow bleed of revenue and a confused data science team. The culprit? Our LLM-based transaction classifier had started returning structured outputs that technically matched the JSON schema but contained values outside the expected enum — like classifying 'gift card purchase' as 'travel' because the model hallucinated a new category. We caught it only when the finance team noticed a $400k/month discrepancy in chargeback rates. The schema validation we thought was bulletproof? It checked JSON validity, not semantic correctness against our controlled vocabulary.
How Structured Outputs Actually Work Under the Hood
When you ask an LLM for structured output, you're not getting a guaranteed parseable result — you're getting a probability distribution over tokens that you then try to coerce into JSON. The model doesn't understand JSON; it's learned to mimic the patterns from training data. This is why function calling APIs (like OpenAI's) add a constrained decoding layer that forces the model to only generate tokens that produce valid JSON according to your schema. But even with constrained decoding, the model can still produce semantically invalid values — it just guarantees syntactic validity.
The real magic happens in the logit bias processor: for each token position, the API computes the set of tokens that would keep the output valid JSON, masks all others, and samples only from the valid set. This is why function calling is more reliable than prompt-based JSON — it's literally impossible to produce invalid JSON. But 'impossible to produce invalid JSON' doesn't mean 'impossible to produce wrong JSON.' The model can still hallucinate field names, use wrong enum values, or produce data that matches the schema but makes no sense for your domain.
Most tutorials skip this distinction. They show you a pretty example with a weather schema and call it done. They don't tell you that the constrained decoding only guarantees JSON validity, not semantic validity. They don't mention that the logit bias processor adds ~50-100ms latency per call. And they certainly don't warn you that as your schema grows (more fields, nested objects), the probability of a valid-but-wrong output increases because the model has more degrees of freedom to hallucinate.
import json from openai import OpenAI from typing import List, Optional client = OpenAI() # Define a schema with enum constraints # This is what gets sent to the constrained decoding layer schema = { "type": "object", "properties": { "risk_level": { "type": "string", "enum": ["low", "medium", "high"] # <-- This is critical }, "transaction_amount": { "type": "number", "minimum": 0 }, "flags": { "type": "array", "items": { "type": "string", "enum": ["velocity", "geo_anomaly", "amount_threshold"] } } }, "required": ["risk_level", "transaction_amount", "flags"] } # The constrained decoding ensures JSON is valid, but NOT semantically correct response = client.chat.completions.create( model="gpt-4-0125-preview", messages=[ {"role": "system", "content": "Classify this transaction. Use the provided schema."}, {"role": "user", "content": "Transaction: $5000 wire transfer to new account"} ], functions=[{"name": "classify_transaction", "parameters": schema}], function_call={"name": "classify_transaction"} ) # Parse the function call arguments # This will ALWAYS be valid JSON thanks to constrained decoding parsed = json.loads(response.choices[0].message.function_call.arguments) # BUT: the values might violate enum constraints if schema wasn't strict enough # Example: risk_level could be "very_high" if we didn't define enum assert parsed["risk_level"] in ["low", "medium", "high"], f"Invalid enum value: {parsed['risk_level']}" # Production validation: check all enum fields ENUM_FIELDS = { "risk_level": ["low", "medium", "high"], "flags": ["velocity", "geo_anomaly", "amount_threshold"] } for field, allowed_values in ENUM_FIELDS.items(): value = parsed[field] if isinstance(value, list): for item in value: assert item in allowed_values, f"Invalid enum in {field}: {item}" else: assert value in allowed_values, f"Invalid enum value for {field}: {value}" print(f"Validated output: {json.dumps(parsed, indent=2)}")
Practical Implementation: Building a Bulletproof Structured Output Pipeline
Start with a Pydantic model that mirrors your JSON schema — this gives you type checking, default values, and validation at the application level. Then build a pipeline that: (1) sends the prompt with function calling, (2) parses the response, (3) validates against your Pydantic model, (4) retries with a corrected prompt on failure, and (5) logs everything for debugging. The key insight: separate your 'schema for the API' from your 'schema for validation.' The API schema should be minimal to reduce token usage and hallucination surface; the validation schema should be exhaustive.
Most implementations fail because they treat the LLM response as authoritative. They don't add a validation layer that checks for business rules like 'if risk_level is high, then flags must not be empty.' These cross-field validations are impossible to express in JSON Schema but trivial in Pydantic. Also, never trust the 'required' array in JSON Schema alone — models sometimes skip required fields even with function calling, especially with older models or when the prompt is long.
import json import logging from datetime import datetime from typing import List, Optional from pydantic import BaseModel, Field, field_validator from openai import OpenAI import backoff logger = logging.getLogger(__name__) # Pydantic model with business logic validation class TransactionClassification(BaseModel): risk_level: str = Field(..., pattern="^(low|medium|high)$") transaction_amount: float = Field(..., ge=0) flags: List[str] = Field(default_factory=list) timestamp: datetime = Field(default_factory=datetime.utcnow) @field_validator('flags') @classmethod def validate_flags(cls, v): allowed = {"velocity", "geo_anomaly", "amount_threshold"} for flag in v: if flag not in allowed: raise ValueError(f"Invalid flag: {flag}") return v @field_validator('risk_level') @classmethod def validate_risk_consistency(cls, v, info): # Cross-field validation: high risk must have at least one flag if v == "high" and not info.data.get('flags'): raise ValueError("High risk transactions must have at least one flag") return v # JSON schema for the API (minimal, just types) api_schema = { "type": "object", "properties": { "risk_level": {"type": "string"}, "transaction_amount": {"type": "number"}, "flags": { "type": "array", "items": {"type": "string"} } }, "required": ["risk_level", "transaction_amount"] } @backoff.on_exception(backoff.expo, (json.JSONDecodeError, ValueError), max_tries=3) def classify_transaction(transaction_text: str) -> TransactionClassification: client = OpenAI() response = client.chat.completions.create( model="gpt-4-0125-preview", messages=[ {"role": "system", "content": "Classify this transaction. Use the provided schema."}, {"role": "user", "content": transaction_text} ], functions=[{"name": "classify_transaction", "parameters": api_schema}], function_call={"name": "classify_transaction"} ) raw = response.choices[0].message.function_call.arguments logger.debug(f"Raw response: {raw}") parsed = json.loads(raw) # Validate against Pydantic model (includes business rules) validated = TransactionClassification(**parsed) return validated # Usage try: result = classify_transaction("Transaction: $5000 wire transfer to new account") print(f"Validated: {result.model_dump_json(indent=2)}") except ValueError as e: logger.error(f"Validation failed: {e}") # Trigger alert or fallback
When NOT to Use Structured Outputs with LLMs
Structured outputs are not free. Each function call adds ~50-100ms latency and consumes tokens for both the schema definition and the structured response. If you're doing high-throughput classification (10k+ requests/minute), the cost and latency can be prohibitive. Consider traditional ML models or rule-based systems for simple classifications. Also, don't use structured outputs for exploratory or creative tasks where you want the model to discover categories — you'll constrain it into your preconceived buckets and miss novel patterns.
The worst case for structured outputs is when your schema has many optional fields or nested objects. Each optional field increases the chance the model will hallucinate a value for it. Each level of nesting increases the probability of a parse error (even with constrained decoding, older models sometimes produce malformed nested objects). We've seen teams try to extract 50+ fields from a single LLM call — the failure rate was 40% even with GPT-4.
import time from openai import OpenAI client = OpenAI() # Bad: complex schema with many optional fields complex_schema = { "type": "object", "properties": { "category": {"type": "string"}, "subcategory": {"type": "string"}, "confidence": {"type": "number"}, "reasoning": {"type": "string"}, "alternative_categories": { "type": "array", "items": {"type": "string"} }, "related_entities": { "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "type": {"type": "string"}, "relevance": {"type": "number"} } } } }, "required": ["category"] } # Measure latency and failure rate start = time.time() response = client.chat.completions.create( model="gpt-4-0125-preview", messages=[{"role": "user", "content": "Classify this text: 'New iPhone release'"}], functions=[{"name": "classify", "parameters": complex_schema}], function_call={"name": "classify"} ) elapsed = time.time() - start print(f"Latency: {elapsed:.2f}s") print(f"Tokens used: {response.usage.total_tokens}") # Alternative: use a simpler approach for high throughput # Rule-based or traditional ML for simple categories # Only use LLM for the complex cases that need reasoning
Production Patterns & Scale: Handling 10M Requests/Day
At scale, the failure modes change. You can't manually inspect 10M responses per day for schema violations. You need automated monitoring, circuit breakers, and fallback strategies. The key pattern: use a two-tier validation system. Tier 1 is a fast, schema-level check (JSON parse + required fields) that runs inline with the request. Tier 2 is a slower, semantic check (enum validation, cross-field rules) that runs asynchronously and alerts on anomalies.
For caching, never cache the raw LLM response — cache the validated structured output. Raw responses can have subtle differences (whitespace, ordering) that waste cache space. Use the input prompt hash as the cache key, and include the schema version in the hash to handle schema migrations gracefully. Set a TTL on cached results — LLM outputs degrade over time as models are updated or deprecated.
Rate limiting is critical. Most LLM APIs have per-minute and per-day limits. You need a token bucket algorithm that accounts for both request count and token count. We use a Redis-based rate limiter that tracks both metrics and queues requests when limits are approached.
import hashlib import json import time from typing import Optional from redis import Redis from openai import OpenAI, RateLimitError redis = Redis.from_url("redis://localhost:6379") client = OpenAI() class TieredValidator: def __init__(self, schema: dict, pydantic_model): self.schema = schema self.model = pydantic_model def tier1_fast_check(self, raw_response: str) -> bool: """Fast inline check: JSON parse + required fields""" try: parsed = json.loads(raw_response) except json.JSONDecodeError: return False for field in self.schema.get("required", []): if field not in parsed: return False return True def tier2_semantic_check(self, raw_response: str) -> bool: """Slow async check: full Pydantic validation""" try: parsed = json.loads(raw_response) self.model(**parsed) return True except (ValueError, TypeError) as e: # Log the failure for monitoring logger.warning(f"Tier2 validation failed: {e}") return False def get_cached_result(prompt_hash: str, schema_version: str) -> Optional[dict]: cache_key = f"llm_structured:{schema_version}:{prompt_hash}" cached = redis.get(cache_key) if cached: return json.loads(cached) return None def set_cached_result(prompt_hash: str, schema_version: str, result: dict, ttl: int = 3600): cache_key = f"llm_structured:{schema_version}:{prompt_hash}" redis.setex(cache_key, ttl, json.dumps(result)) # Rate limiter: token bucket class TokenBucketRateLimiter: def __init__(self, max_requests: int, max_tokens: int, window_seconds: int = 60): self.max_requests = max_requests self.max_tokens = max_tokens self.window = window_seconds def check(self, estimated_tokens: int) -> bool: key = f"ratelimit:{int(time.time() / self.window)}" current_requests = redis.get(f"{key}:requests") or 0 current_tokens = redis.get(f"{key}:tokens") or 0 if current_requests >= self.max_requests or current_tokens + estimated_tokens > self.max_tokens: return False redis.incr(f"{key}:requests") redis.incrby(f"{key}:tokens", estimated_tokens) redis.expire(f"{key}:requests", self.window) redis.expire(f"{key}:tokens", self.window) return True
Common Mistakes With Specific Examples From Production
Mistake #1: Not validating enum values in the schema. We saw this in the fraud pipeline incident — the schema defined 'risk_level' as a string without an enum constraint. The LLM returned 'medium_high' (combining two categories) and the pipeline accepted it. Fix: always define enum constraints for categorical fields.
Mistake #2: Using the same schema for the API and validation. The API schema should be minimal to reduce token usage and hallucination. The validation schema should be exhaustive with all business rules. When they're the same, you either have too many tokens in the API call or too few validation rules.
Mistake #3: Not handling the case where the LLM refuses to respond. Sometimes the model returns 'I cannot classify this transaction' as a string instead of the structured output. This happens more often with content moderation or sensitive topics. Always have a fallback: a default structured output that flags the response for manual review.
Mistake #4: Ignoring response ordering. JSON objects don't guarantee field ordering, but some downstream systems expect fields in a specific order. Use an OrderedDict or sort fields before sending to downstream systems.
import json from openai import OpenAI from collections import OrderedDict client = OpenAI() # Mistake #1: No enum constraint bad_schema = { "type": "object", "properties": { "risk_level": {"type": "string"} # No enum! }, "required": ["risk_level"] } # Mistake #2: Same schema for API and validation # API schema has too many fields -> more hallucinations # Validation schema has too few rules -> misses business logic # Mistake #3: No fallback for refusal response = client.chat.completions.create( model="gpt-4-0125-preview", messages=[ {"role": "system", "content": "Classify this transaction. Use the schema."}, {"role": "user", "content": "Classify: illegal transaction"} ], functions=[{"name": "classify", "parameters": bad_schema}], function_call={"name": "classify"} ) # The model might return a refusal message instead of structured output raw = response.choices[0].message.function_call.arguments print(f"Raw response: {raw}") # Output might be: "I cannot classify illegal transactions" # Fix: check if the response is valid JSON before parsing if raw.startswith("{"): parsed = json.loads(raw) else: # Fallback: create a default structured output parsed = {"risk_level": "unknown", "flagged_for_review": True} # Mistake #4: Field ordering matters # Use OrderedDict to maintain field order ordered_output = OrderedDict() ordered_output["risk_level"] = parsed.get("risk_level", "unknown") ordered_output["timestamp"] = "2024-01-01T00:00:00Z" print(json.dumps(ordered_output))
Comparison vs Alternatives: When to Use Structured Outputs vs Function Calling vs Prompt Engineering
There are three main approaches to getting structured data from LLMs: (1) prompt engineering (asking 'return JSON with fields X, Y, Z'), (2) function calling (OpenAI's API with a defined schema), and (3) structured output APIs (like OpenAI's response_format with json_schema). Each has trade-offs.
Prompt engineering is the simplest but least reliable. You get ~60-70% valid JSON with GPT-4, lower with smaller models. It's fine for prototyping but not production. Function calling adds constrained decoding, getting you ~99% valid JSON syntax, but adds latency and token overhead. Structured output APIs (like OpenAI's json_schema response_format) are the newest and most reliable, with ~99.9% valid JSON, but they're only available on certain models and have stricter schema requirements.
The key insight: function calling is better for complex schemas with nested objects because the constrained decoding handles the nesting. Prompt engineering is better for simple schemas (1-2 fields) where you want lower latency. Structured output APIs are best when you need the highest reliability and can accept the model limitations.
import json import time from openai import OpenAI client = OpenAI() # Approach 1: Prompt Engineering # Simple, but unreliable start = time.time() response = client.chat.completions.create( model="gpt-4-0125-preview", messages=[ {"role": "user", "content": "Return JSON with fields: category (string) and confidence (float 0-1). Example: {'category': 'tech', 'confidence': 0.9}. Classify: 'New iPhone release'"} ] ) elapsed1 = time.time() - start raw = response.choices[0].message.content print(f"Prompt engineering: {elapsed1:.2f}s") print(f"Raw: {raw[:100]}...") # Approach 2: Function Calling # More reliable, adds overhead start = time.time() response = client.chat.completions.create( model="gpt-4-0125-preview", messages=[ {"role": "user", "content": "Classify: 'New iPhone release'"} ], functions=[{ "name": "classify", "parameters": { "type": "object", "properties": { "category": {"type": "string"}, "confidence": {"type": "number"} }, "required": ["category", "confidence"] } }], function_call={"name": "classify"} ) elapsed2 = time.time() - start print(f"Function calling: {elapsed2:.2f}s") # Approach 3: Structured Output API (OpenAI's json_schema) # Most reliable, but model-specific start = time.time() response = client.chat.completions.create( model="gpt-4o-2024-08-06", # Requires this model or later messages=[ {"role": "user", "content": "Classify: 'New iPhone release'"} ], response_format={ "type": "json_schema", "json_schema": { "name": "classification", "strict": True, "schema": { "type": "object", "properties": { "category": {"type": "string"}, "confidence": {"type": "number"} }, "required": ["category", "confidence"], "additionalProperties": False } } } ) elapsed3 = time.time() - start print(f"Structured output API: {elapsed3:.2f}s") print(f"Latency comparison: Prompt={elapsed1:.2f}s, Function={elapsed2:.2f}s, Structured={elapsed3:.2f}s")
Debugging & Monitoring: What to Log and Alert On
You need three levels of logging for structured outputs: (1) raw response logging — the complete LLM response before any parsing, (2) parsed output logging — the validated structured output that enters your system, and (3) validation failure logging — every time a response fails validation. These three logs let you trace any bug back to its source.
Alert on: schema violation rate > 1% (indicates prompt degradation or model drift), empty response rate > 0.5% (indicates token limit issues or refusal problems), and latency p99 > 2x baseline (indicates system overload or model issues). Set up a dashboard that tracks these metrics over time, with the ability to drill into specific schema fields that are failing most often.
The most common debugging scenario: you get a valid JSON response that doesn't match your schema. The first thing to check is whether the schema you sent to the API matches the schema you're validating against. We've seen multiple incidents where a developer updated the validation schema but forgot to update the API schema, or vice versa. Version both schemas and log the version with every response.
import json import logging from datetime import datetime from typing import Optional from openai import OpenAI logger = logging.getLogger(__name__) class StructuredOutputMonitor: def __init__(self, schema_version: str): self.schema_version = schema_version self.metrics = { "total_requests": 0, "valid_responses": 0, "schema_violations": 0, "empty_responses": 0, "latencies": [] } def log_raw_response(self, prompt: str, raw: str, latency: float): """Log the complete raw response for debugging""" logger.debug(f"Raw response: {raw}") self.metrics["total_requests"] += 1 self.metrics["latencies"].append(latency) # Store raw response for later analysis with open(f"raw_responses/{datetime.utcnow().isoformat()}.json", "w") as f: json.dump({ "timestamp": datetime.utcnow().isoformat(), "schema_version": self.schema_version, "prompt": prompt, "raw_response": raw, "latency": latency }, f) def log_validation_failure(self, raw: str, error: str): """Log every validation failure with context""" self.metrics["schema_violations"] += 1 logger.warning(f"Schema violation: {error}") # Alert if violation rate exceeds threshold violation_rate = self.metrics["schema_violations"] / max(self.metrics["total_requests"], 1) if violation_rate > 0.01: # 1% threshold logger.error(f"Schema violation rate {violation_rate:.2%} exceeds threshold!") # Trigger PagerDuty alert def log_empty_response(self): """Log when LLM returns empty or truncated response""" self.metrics["empty_responses"] += 1 logger.error("Empty response detected") if self.metrics["empty_responses"] / max(self.metrics["total_requests"], 1) > 0.005: logger.error("Empty response rate exceeds 0.5%!") # Usage monitor = StructuredOutputMonitor(schema_version="v2.1") # Simulate a request prompt = "Classify this transaction" raw_response = '{"risk_level": "medium", "transaction_amount": 5000}' latency = 0.8 monitor.log_raw_response(prompt, raw_response, latency) # Validate and log if fails try: parsed = json.loads(raw_response) if parsed.get("risk_level") not in ["low", "medium", "high"]: raise ValueError(f"Invalid enum: {parsed.get('risk_level')}") except (json.JSONDecodeError, ValueError) as e: monitor.log_validation_failure(raw_response, str(e)) # Check metrics print(f"Metrics: {json.dumps(monitor.metrics, default=str)}")
Why Constrained Decoding Beats Post-Processing Every Time
Most teams treat structured outputs as a post-processing problem. Parse the JSON. Fix the formatting. Retry if it fails. This is backwards. The correct approach is constrained decoding — where you enforce the schema during token generation, not after.
When you sample tokens freely and then fix the output, you're fighting the model's natural distribution. The model wants to say "The total is $42.50" but you force it into {"total": 42.50}. Every retry costs latency and money. With constrained decoding, the model only generates tokens that conform to your schema from the first token onward.
Tools like Outlines and Guidance use prefix trees of valid tokens. At each decoding step, they mask out any token that would break your schema. The model never even considers producing invalid output. This isn't magic — it's a simple logit mask applied to the vocabulary at each step.
The performance hit is negligible. On a typical JSON schema, constrained decoding adds ~5-10% latency. Compare that to 2-3 retries with prompt engineering, each costing 2x the tokens. The math is clear.
import outlines from pydantic import BaseModel # Constrained decoding — schema enforced during generation class Receipt(BaseModel): store: str total: float model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2") generator = outlines.generate.json(model, Receipt) # Only returns valid JSON matching Receipt schema result = generator("Extract the total from this receipt: Total: $42.50 at Walmart") print(result) # Receipt(store='Walmart', total=42.5) # Compare with post-processing raw_output = "The total is $42.50 at Walmart" # Now you need regex, json parsing, error handling... good luck
The Hidden Cost of API Provider "Magic" for Structured Outputs
OpenAI's structured outputs mode and Gemini's response_schema feel like free wins. Define your Pydantic model, pass it to the API, get perfectly formatted JSON back. But this convenience comes with three hidden costs that bite you at scale.
First, vendor lock-in. Switch from GPT-4 to Claude 3.5 and your entire structured output pipeline breaks. There's no standard for schema passing across providers. Every API has its own format, its own limitations, its own pricing quirks. You're not just using an LLM — you're marrying a provider.
Second, you pay for the magic. OpenAI charges the same token rate whether you use structured outputs or not. But the provider is doing constrained decoding under the hood — and charging you a premium for it. Run the numbers: 10M requests/day at $0.03/1K tokens means $300K/day. Open-source constrained decoding costs $0.
Third, debugging is opaque. When OpenAI's structured output fails silently (and it does — ask anyone who's hit the "structured output not supported for this model" error at 2 AM), you have zero visibility into why. With local constrained decoding, you control every logit mask and can reproduce failures deterministically.
# Provider-specific — this breaks when you switch providers import openai from pydantic import BaseModel class Patient(BaseModel): name: str age: int diagnosis: list[str] # OpenAI's way response = openai.beta.chat.completions.parse( model="gpt-4o-2024-08-06", messages=[{"role": "user", "content": "Extract patient info"}], response_format=Patient, ) # Gemini's way from google.genai import types response_schema = { "type": "OBJECT", "properties": { "name": {"type": "STRING"}, "age": {"type": "INTEGER"}, "diagnosis": {"type": "ARRAY", "items": {"type": "STRING"}} } } # Totally different API — zero code reuse
When Structured Outputs Lie: The Schema Drift Problem
Structured outputs guarantee format, not correctness. This is the most dangerous misconception in production LLM systems. Just because the JSON parses doesn't mean the data is right. Schema drift is when your output conforms to your schema but contains semantically wrong values — and you never notice because the JSON validates.
Real example: A medical intake system that extracts patient age. Schema says {"age": int}. Model outputs {"age": 42}. Valid JSON. But the patient is a 42-year-old or a 42-month-old? The model guessed based on context. Your validation passes. Your downstream system treats it as years. Someone gets a pediatric dose of an adult medication.
This happens because structured outputs only constrain syntax, not semantics. They ensure the value is an integer between 0-150, but not that the unit matches the context. The model can confidently hallucinate into your rigid schema.
Mitigation requires two things: post-generation validation hooks that check value ranges against known contexts, and logging every structured output with the raw input for audit trails. Never trust a structured output blindly — verify the semantic bounds before routing to production systems.
from pydantic import BaseModel, field_validator from datetime import datetime class PatientIntake(BaseModel): name: str age_value: int age_unit: str # 'years' or 'months' @field_validator('age_unit') @classmethod def validate_age_unit(cls, v, info): raw = info.data.get('age_value', 0) if v == 'months' and raw > 24: # Suspicious — kids over 2 years shouldn't use months raise ValueError(f"age_value {raw} with unit 'months' is suspicious") return v # This passes structured output but fails semantic check bad_data = {"name": "Baby Doe", "age_value": 42, "age_unit": "months"} try: PatientIntake(**bad_data) except ValueError as e: print(f"Semantic guardrail triggered: {e}") # Alert: potential schema drift detected
The $400k Enum Drift — When Structured Outputs Lie Silently
JSON.parse() succeeded and all required keys were present, the output was valid for downstream consumption.- Define enum constraints in your JSON schema — don't rely on prompt engineering alone to enforce controlled vocabularies.
- Monitor enum violation rates as a leading indicator of model drift or prompt degradation — it catches problems before accuracy metrics do.
- Log the raw LLM response alongside the parsed structure — you can't debug schema violations without seeing what the model actually sent.
python -c "import json, sys; data = sys.stdin.read(); json.loads(data); print('VALID')" < raw_response.jsonpython -c "import json; print(json.dumps(json.loads(open('raw_response.json').read()), indent=2))"response.strip().removeprefix('``json').removesuffix('``')python -c "import json; s=json.load(open('schema.json')); print([k for k in s.get('required',[]) if k not in json.load(open('response.json'))])"python -c "import json; print('Keys in response:', list(json.load(open('response.json')).keys()))"python -c "import json; r=json.load(open('response.json')); print({k:v for k,v in r.items() if isinstance(v,str) and v not in {'allowed1','allowed2','allowed3'}})"python -c "import json; print(json.dumps(json.load(open('response.json')), default=str))"python -c "import tiktoken; enc=tiktoken.encoding_for_model('gpt-4'); prompt=open('prompt.txt').read(); print('Prompt tokens:', len(enc.encode(prompt)))"python -c "import json; print('Response length:', len(open('response.json').read()))"| Concern | Structured Outputs | Function Calling | Prompt Engineering | Recommendation |
|---|---|---|---|---|
| JSON validity guarantee | ~99.9% with constrained decoding | ~99% (tool use can still hallucinate args) | ~80-95% (depends on prompt quality) | Structured outputs for fixed schemas |
| Latency overhead | 10-30% increase | 20-50% increase (tool selection step) | 0% (but retries add latency) | Structured outputs for speed |
| Schema flexibility | Fixed schema per call | Dynamic tool selection | Unlimited (but unreliable) | Function calling for dynamic needs |
| Cost per request | Low (no extra tokens for tool definitions) | Higher (tool definitions in context) | Lowest (no extra tokens) | Structured outputs for cost |
| Ease of debugging | Hard (black-box token masking) | Medium (tool call logs) | Easy (full prompt visible) | Prompt engineering for debugging |
| Production failure rate | <0.1% | ~1% | 5-20% | Structured outputs for reliability |
| File | Command / Code | Purpose |
|---|---|---|
| structured_output_internals.py | from openai import OpenAI | How Structured Outputs Actually Work Under the Hood |
| production_pipeline.py | from datetime import datetime | Practical Implementation |
| when_not_to_use.py | from openai import OpenAI | When NOT to Use Structured Outputs with LLMs |
| production_scale_pipeline.py | from typing import Optional | Production Patterns & Scale |
| common_mistakes.py | from openai import OpenAI | Common Mistakes With Specific Examples From Production |
| comparison_approaches.py | from openai import OpenAI | Comparison vs Alternatives |
| debugging_monitoring.py | from datetime import datetime | Debugging & Monitoring |
| constrained_vs_post.py | from pydantic import BaseModel | Why Constrained Decoding Beats Post-Processing Every Time |
| vendor_lockin.py | from pydantic import BaseModel | The Hidden Cost of API Provider "Magic" for Structured Outpu |
| semantic_validation.py | from pydantic import BaseModel, field_validator | When Structured Outputs Lie |
Key takeaways
Common mistakes to avoid
4 patternsRelying on prompt engineering alone for JSON output
Not validating enum values against the schema after generation
Assuming structured outputs are deterministic
Not handling schema evolution in production
Interview Questions on This Topic
Explain how constrained decoding works for structured outputs. How does it differ from post-hoc validation?
Your fraud pipeline uses structured outputs with an enum field 'risk_level': ['low', 'medium', 'high']. You add 'critical' to the schema. What breaks and how do you roll it out safely?
How would you design a monitoring system for structured output quality at 10M requests/day?
Compare structured outputs, function calling, and prompt engineering for extracting invoice data. When would you choose each?
What happens if the LLM's tokenizer splits a JSON key across multiple tokens? How does constrained decoding handle this?
Frequently Asked Questions
They use constrained decoding: the LLM's token generation is restricted to only tokens that produce valid JSON according to a provided schema. This is done by masking the logits of invalid tokens at each step. Providers like OpenAI implement this server-side; local libraries like guidance or outlines do it client-side by intercepting the sampling process.
No. Even with constrained decoding, edge cases like tokenizer mismatches (e.g., multi-byte Unicode in string fields) or schema recursion limits can produce invalid output. Always validate server-side. In practice, failure rates are <0.1% with proper providers but can spike to 5% with prompt-only approaches.
Structured outputs add 10-30% latency because the constrained decoding reduces the token search space and can require more decoding steps for complex schemas. For high-throughput pipelines, batch requests and use simpler schemas (fewer nested objects) to mitigate this.
Log the raw output string, the schema version, and the model ID. Check if the failure is due to a missing field (schema mismatch), wrong type (e.g., string instead of number), or enum violation (value not in allowed list). Use a JSON schema validator to get exact error paths. Monitor the 'validation_failure_rate' metric per endpoint.
Use function calling when you need the LLM to decide which tool to invoke (e.g., multi-tool agents). Use structured outputs when you always want the same JSON schema returned (e.g., extracting fields from documents). Function calling adds overhead of tool selection; structured outputs are faster for fixed schemas.
20+ years shipping production ML systems and the infrastructure behind them. Lessons pulled from things that broke in production.
That's LLM APIs. Mark it forged?
7 min read · try the examples if you haven't