Chatbot Arena is a crowdsourced battle platform and leaderboard for comparing language models through user-selected pairwise preferences. Users interact with two responses from different models and choose the answer they prefer, bringing evaluation closer to real-world usage. section 4.0 section 4.3
A typical Arena evaluation proceeds as follows:
- Present a user with two generated responses to the same request. section 4.3
- Let the user interact with the responses and select the preferred answer. section 4.3
- Aggregate battle outcomes into comparative model rankings. section 4.0
Controlled benchmark versus real-world preference
MT-Bench uses 80 expert-designed questions across eight categories and two conversational turns, while Chatbot Arena gathers preferences from users in less controlled interactions. The contrast separates systematic testing from validation against real-world preferences. section 4.3 section 4.3
Listing 4.6 A sample of MT-Bench data in the writing category
{
"question_id": 81,
"category": "writing",
"turns": [
"Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions."
"Rewrite your previous response. Start every sentence with the letter A."
]
}
Use the MT-Bench sample to inspect the controlled multi-turn evaluation format that complements Arena's crowdsourced battles. section 4.3
