A model-to-code interface is the point where an agent’s orchestration code receives a model response and expects to use it in a specified format. A response that seems understandable to a person can still break the agent: code cannot reliably search free-form prose for the intended answer when it expects a particular structure. section 5.0 section 3.3.4
A math-answer handoff
Consider an agent that asks a language model to reason through a student’s math problem and conclude with a final answer. If the agent’s code expects that answer in a specified form, a differently formatted reply creates a handoff problem even when a person could find the answer in the response. Probabilistic sampling makes exact repetition of the requested form uncertain. section 5.0
Two responses to this boundary problem are:
- Constrain generation with a supported grammar or format specification, such as a JSON grammar or schema, when code needs parseable output. section 3.3.4
- Separate structured extraction from less constrained reasoning: pass unstructured content to an extraction errand that produces the structured result. section 3.3.4
- Use repeated sampling as a reliability pattern when stochastic output causes failures, while recognizing that another sample alone does not define a parseable interface. section 5.0
Listing 3.6 structured_extraction_errand.py: Errand for JSON extraction
class ExtractToJSONErrand(Errand):
def __init__(self, client: Client):
self.system_prompt = "You are a helpful assistant that extracts
structured data from a free-form textual response in JSON format. Output
the extracted data as a JSON object. Do not include any other text in your
response."
self.extraction_call = LMCall(
client,
[
{
"role": "system",
"content": self.system_prompt,
}
],
)
async def __call__(self, input: str) -> str:
extraction_response: str = (
await self.extraction_call(
input,
config={
"response_format": {
"type": "json_object"
}
},
)
)
return json.loads(extraction_response)
The JSON extraction errand illustrates both sides of the handoff: a prompt requests only a JSON object, while the model-call configuration sets `response_format` to `json_object`. The endpoint’s format control supplements the prompt rather than relying on the instruction alone. section 3.3.4
