liveBook · Manning

MT-Bench

As explained in LLM Evaluation and Alignment, The Foundational Ideas

MT-Bench is a benchmark dataset of multi-turn prompts and model responses used to evaluate instruction-following language models.

MT-Bench is a controlled benchmark for evaluating language models through two-turn conversations. Its multi-turn design tests whether a model maintains context, responds to feedback, and builds on earlier answers. section 4.3

The benchmark contains 80 questions across eight categories:

  • Writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. section 4.3
  • Expert human judgments provide a comparison point for assessing LLM-judge agreement. section 4.3
Listing 4.6 A sample of MT-Bench data in the writing category
{
  "question_id": 81,
  "category": "writing",
  "turns": [
    "Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions."
    "Rewrite your previous response. Start every sentence with the letter A."
  ]
}

The sample shows a writing prompt followed by a constraint for rewriting the previous answer. section 4.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Specifies 80 questions, eight categories, two turns per question, and expert annotations; does not detail scoring procedures.
Examples
Uses a Hawaii travel-blog prompt followed by a rewrite constraint requiring every sentence to start with A.
What you can do after
Evaluate multi-turn model responses and compare LLM-judge assessments with expert human judgments.
How it connects
Connects LLM-as-a-judge evaluation with Chatbot Arena as a complementary crowdsourced approach.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.