An evaluation prompt tells a model how to respond to a benchmark question so its answer can be checked consistently. For GAIA questions, it asks the model to decide whether it can solve the problem, then return either a final answer or a reason it cannot answer. The answer must be a number, a short string, or a comma-separated list, with formatting rules that support comparison against ground truth. section 2.4.2
The prompt works alongside a structured response and an evaluation pipeline:
- Set `is_solvable` first. Put a solvable response in `final_answer`; otherwise, explain the limitation in `unsolvable_reason`. section 2.4.2
- Send the evaluation prompt as the system message and the benchmark question as the user message. Request the response schema, then compare `final_answer` with the ground truth using a case-insensitive exact match after trimming whitespace. section 2.4.2
Listing 2.11 Pydantic model for GAIA responses
from pydantic import BaseModel
class GaiaOutput(BaseModel):
is_solvable: bool
unsolvable_reason: str = ""
final_answer: str = ""
The `GaiaOutput` schema makes `is_solvable` required while defaulting both answer strings to empty. That accommodates solvable and unsolvable responses without changing the response shape. section 2.4.2
A prompt for an evaluator has a different job: it tells a judge how to grade an agent’s answer. A rubric-based research evaluator receives the query, final answer, evidence excerpts and source metadata. Its prompt requires judgments based only on that evidence, PASS or FAIL for each rubric item, and a single JSON object. An unsupported claim fails even if it sounds plausible; a citation also fails when its excerpt does not support the claim. section 10.3.2
