Reference-free evaluation judges generated text from the input and output themselves, without comparing the result with a preexisting reference text. It is useful when reference summaries are scarce, datasets evolve continuously, or each case is highly specific. section 12.3 section 12.3.1
G-Eval applies this approach by combining chain-of-thought prompting with form filling. An LLM judges generated content against the input prompt and output, examining qualities such as coherence, consistency, and relevance rather than relying on text similarity. section 12.3.1
Common evaluation targets include:
- Groundedness: whether generated answers match and are supported by the supplied context. section 12.3
- Coherence, consistency, relevance, fluency, and GPT similarity as dimensions for assessing generated language. section 12.3 section 12.3.1
Medical report generation
A system generating customized medical reports can be assessed for correctness, consistency, and medical relevance even when patient-specific reference reports are unavailable. Similar applications include legal documents, creative advertising or game storytelling, and content moderation for changing online material. section 12.3.1 section 12.3.1
Table 12.5 LLM evaluation metrics
|
Metric
|
Focus
|
Method
|
When to use?
|
|---|---|---|---|
|
Groundedness
|
It evaluates how well the answers the model produces match the information in the source data (context that the user provides). This metric ensures that the context backs up the answers generated by AI.
|
It evaluates how well the statements in an AI-generated answer match the source context, ensuring that the context supports these statements. It is rated from 1 (bad) to 5 (good).
|
It is used when we want to check that the AI responses match and are confirmed by the given context. It is also used when being factually correct and contextually precise is important, such as when finding information, answering questions, and summarizing content.
|
|
Coherence
|
It evaluates the model’s ability to generate coherent, natural output similar to human language.
|
It evaluates how well the generation is structured and connected. This is rated from 1 (bad) to 5 (good).
|
Use it when evaluating how easy and user-friendly your model’s generated responses are in real-world situations.
|
|
Fluency
|
It measures the grammar proficiency and readability of a model’s generated response.
|
The fluency measure evaluates how well the generated text follows grammatical rules, syntactic structures, and suitable word choices. It is scored from 1 (bad) to 5 (good).
|
This tool assesses the linguistic accuracy of the generated text, ensuring that it follows appropriate grammar rules, syntax structures, and word choices.
|
|
GPT similarity
|
It compares how similar a source data (ground truth) sentence is to the output from an AI model.
|
This assessment involves creating sentence-level embeddings for both the ground truth and the model’s prediction, which are high-dimensional vector representations that encode the semantic meaning and context of the sentences.
|
Use it to get an unbiased measure of a model’s performance, especially in text generation tasks where we have the correct responses available. This lets us check how closely the generated text matches the intended content, which helps us evaluate the model’s quality and accuracy.
|
Use the metric table to compare groundedness and other evaluation dimensions by focus, method, and appropriate use case. section 12.3
