FActScore measures factual accuracy at the claim level: it breaks a generated response into atomic, verifiable facts and scores each against grounding data. This preserves nuance when one answer contains both correct and incorrect claims. section 9.1.2
A product answer with mixed accuracy
A ShopBot says headphones have a 6-hour battery, a 1-year warranty, and a $79.99 sale price. If the battery actually lasts 8 hours and the other two claims are correct, the score is 2 of 3, or 0.67. section 9.1.2
A practical scoring pass follows this pattern:
- Extract single, verifiable claims from the response. section 9.1.2
- Check each claim against grounding data, then divide the number verified by the number extracted. The sample implementation makes LLM calls for extraction and verification; its verify_fact function is not defined in the snippet. section 9.1.2 section 9.1.2
- Use text-overlap metrics for a faster signal when reference responses are available. section 9.1.2
from openai import OpenAI
import json
client = OpenAI()
def extract_atomic_facts(response: str) -> list[str]:
prompt = f'''Extract atomic facts from this response.
Each fact should be a single, verifiable claim.
Return as JSON array of strings.
Response: {response}'''
result = client.chat.completions.create(
model='gpt-5.4',
messages=[{'role': 'user', 'content': prompt}]
)
return json.loads(result.choices[0].message.content)
def calculate_factscore(response: str, grounding: str) -> float:
facts = extract_atomic_facts(response)
verified = sum(1 for f in facts if verify_fact(f, grounding))
return verified / len(facts) if facts else 1.0
The Python listing shows JSON-based fact extraction and the score calculation; fact verification is delegated to a separate function. section 9.1.2
