Structured Output

Structured Output

Prompted JSON — "reply as JSON" in the system prompt — fails on ~2-8% of calls in the wild: stray prose before the object, trailing commas, unescaped quotes, code fences. That failure rate is fine for a demo and fatal for a loop that runs a thousand times.

The hierarchy — use the strongest that fits

1. Tool use with schema (best) — declare a tool with a JSON Schema for its input. Force the model to call that tool. The API validates the arguments against the schema before you see them. Malformed JSON never leaves the model. Use this whenever the downstream is a real parser.

2. JSON mode / response_format (good) — where supported. Guarantees a valid JSON object at the top level; does not guarantee schema conformance. Cheap upgrade over prompted-JSON.

3. Prompted JSON with strict rules (fallback) — for models/tiers without tool use. Say "output ONLY the JSON object, no code fences, no prose", give an example, and validate on receive. Assume ~5% failure and handle it.

The validate-and-retry pattern

call → parse → if fail: retry once with the parse error appended → parse → if fail: hard fail

• One retry, not a loop. If the model can't produce it in two tries, the schema is too complex or the prompt is wrong. Log and stop.
• Feed the parse error back verbatim on retry — the model will fix specific issues ("expected string at line 3") that it can't guess from a generic "please try again".
• Never silently coerce. If a required field is missing, fail loudly. Auto-defaults hide prompt bugs.

Schema design — keep it flat

• Flat objects beat nested. Every level of nesting is another chance to hallucinate.
• Enums over free strings. "severity": "high|medium|low" not "severity": "...".
• Optional fields default to null explicitly in the schema. Don't ask the model to "omit if unknown".
• No additionalProperties: true without a reason. If the model can add fields, it will, and they'll be inconsistent.

When free JSON is acceptable

• Single scalar field, low-stakes ({"answer": "yes"}).
• Human-in-the-loop reviewing every output.
• Prototype throwaway.

Otherwise use tool_use.

Red flags

• Regex to extract JSON from a code fence. You're one prompt tweak away from breakage. Use tool_use.