Safety evaluation systematically probes AI systems for behaviors that could create unacceptable harm or risk. It complements accuracy, helpfulness, and toxicity benchmarks by targeting failures that ordinary testing may miss. section 10.22
Modern safety evaluation combines several forms of adversarial and comparative testing:
- Red teams test jailbreaks, prompt injections, adversarial formatting, deceptive behavior, biased outputs, and unintended tool use. section 10.22
- Agentic systems require tests for hallucinated function calls, memory corruption, multistep failures, and interactions that drift over time. section 10.22
- Evaluation can measure hallucination rates, citation precision, memory consistency, policy violations, and factual errors across configurations. section 10.22
- Human experts remain important, while automated tools such as Microsoft’s open-source PyRIT generate large numbers of adversarial inputs. section 10.22
Safety results are easier to interpret when evaluation details are disclosed. Llama 2 reporting combined human and model-based raters with information about demographic imbalances, while comparison results can still shift with prompting and formatting choices. section 10.19
