An objective function gives improvement a measurable target. For an LLM, training loss measures how far a predicted next token is from the expected token; for a deployed agent, product metrics measure whether its business or user outcome improves. section 2.1.1 section 8.3.6
Separate the target from the layer where it is measured:
- Training objective: reduce loss by correcting model weights through backpropagation. section 2.1.1
- Operational measures: track behavior such as latency, tool success, token usage, errors, and cost. section 8.3.6
- Quality measures: assess grounding, hallucinations, rubric performance, and user feedback. section 8.3.6
- Product measures: assess task completion, human escalation, conversion, resolution, or time-to-resolution. section 8.3.6
From prediction error to product outcome
During training, the model predicts a next-token distribution, a sampling step selects a token, and loss measures the difference from the expected token. After deployment, success is judged separately by whether the agent completes tasks or improves the intended KPI. section 2.1.1 section 8.3.6
Figure 2.1 The training cycle and inference/generation process of a large language model. On the left is the model’s training process, in which documents are ingested to train the LLM in a first pass. On the right, a user enters a prompt, which is first tokenized and fed into the model. The model then outputs probabilities it uses to sample and produce the next token.
Follow the training-and-inference diagram to see where prediction probabilities, sampling, and loss fit in the model lifecycle. section 2.1.1
