An Elo score estimates the relative strength of competitors from pairwise contests. In chatbot arenas, people compare two answers, and those judgments update each model’s rating, much like ratings for chess, Go, or esports players. section 10.3.13
Chatbot arena
A user poses a challenging question, judges two chatbot responses, and contributes evidence to their Elo scores. Vicuna-13b reached fifth place below GPT-3.5, illustrating how rankings can compare models with different sizes and speeds. section 10.3.13
Treat leaderboard Elo scores as limited evidence rather than objective intelligence measurements. Commercial gaming reduced the LMSYS leaderboard’s usefulness, while GPT-4 judging consistently favored OpenAI and other commercial bots over human ratings. section 10.3.13
>>>from datasets import load_dataset
>>>arena = load_dataset("lmsys/chatbot_arena_conversations")
Download the LMSYS chatbot-arena conversations dataset for examining the pairwise judgments underlying arena ratings. section 10.3.13
