liveBook · Manning

User study

As explained in How Large Language Models Work

A user study is a structured investigation of how people interact with or are affected by a system.

A user study is a structured investigation of how people interact with or are affected by a system. For LLM applications, it examines unintended risks across different groups and settings rather than assuming that general bias benchmarks reveal every harm. section 9.1.1

A practical user-study program can help teams:

  • Investigate harms tied to a specific proposed use case, including risks that standard evaluations miss. section 9.1.1
  • Feed late-identified risks into monitoring and logging after deployment. section 9.1.1
  • Treat testing as incomplete because broad model capabilities create unknown problems that cannot all be enumerated in advance. section 9.1.1

Concrete risk areas

A study might look for harmful racial stereotypes or support for debunked medical practices. Postdeployment reporting, such as the FDA’s MedWatch model, can surface unusual adverse events that earlier testing missed. section 9.1.1

How How Large Language Models Work covers it

How deep it goes
Explains the purpose and limits of user studies, but does not specify participant protocols, metrics, or analysis methods.
Examples
Uses racial stereotypes, debunked medical practices, and FDA MedWatch-style adverse-event reporting as risk examples.
What it assumes
Assumes a defined proposed use case and awareness that model-level bias tests cannot represent every deployment harm.
What you can do after
Diagnose unintended deployment risks and inform monitoring or remediation, while recognizing that testing remains incomplete.
How it connects
Connects user studies with bias benchmarks, RLHF, monitoring, logging, and postdeployment monitoring.
Cover of How Large Language Models Work
liveBookmanning.com
Read How Large Language Models Work on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.