MT-Bench is a controlled benchmark for evaluating language models through two-turn conversations. Its multi-turn design tests whether a model maintains context, responds to feedback, and builds on earlier answers. section 4.3
The benchmark contains 80 questions across eight categories:
- Writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. section 4.3
- Expert human judgments provide a comparison point for assessing LLM-judge agreement. section 4.3
Listing 4.6 A sample of MT-Bench data in the writing category
{
"question_id": 81,
"category": "writing",
"turns": [
"Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions."
"Rewrite your previous response. Start every sentence with the letter A."
]
}
The sample shows a writing prompt followed by a constraint for rewriting the previous answer. section 4.3
