Continuous improvement means evaluating what a generative-AI system produces, changing what falls short, and repeating. Outputs can vary with model settings, so a single successful test is not enough to justify broad deployment. Start with a small enterprise use case, learn from its outcomes, and scale gradually. section 1.7
The feedback cycle can operate at several levels:
- At the application level, check outputs against the use case and refine the system while managing risks such as incorrect details, bias, and misuse. section 1.7
- In retrieval-augmented generation, use unsatisfactory responses and user interactions to revisit chunk sizes or chunking strategies. Keep components flexible so the retrieval setup can change as query patterns become clearer. section 7.7.3
- Before and after fine-tuning, compare task performance with the default model and a baseline using qualitative and quantitative measures. Revise training data to address weak areas, then evaluate again. section 9.2
- In reinforcement learning from human feedback (RLHF), human judgments train a reward model; further evaluations update that feedback, while proximal policy optimization iteratively improves the primary model’s policy. section 9.7
Improving a retrieval response
Suppose a user reports that an answer is unsatisfactory. The system can try a different chunk size or chunking strategy, then use the resulting response to assess whether retrieval improved. This is a change to the retrieval mechanism, not evidence that the underlying model has retrained itself. section 7.7.3
Choose the level of change that matches the problem. Fine-tuning is time-consuming: an initial fine-tuned model may be worse than the default, and finding a suitable deployment model commonly takes multiple training iterations. Treat cloud API latency and failure, cost, and responsible-AI policies as part of the improvement decision—not as issues to postpone until rollout. section 9.2 section 1.7
