A training dataset is the collection of examples a model learns from. Generative AI uses patterns in its training data to produce new content; the quality and relevance of that content depend on the quality and relevance of the data. For an LLM, training adjusts model parameters against examples of text and expected outcomes. That does not make the model a reliable judge of fact versus fiction. section 1.3 section 2.2 section 1.4
For a specific enterprise task, prepare a new training dataset only when fine-tuning is justified. Try prompt engineering and retrieval-augmented generation (RAG) first; they can still complement a fine-tuned model. Define the task, identify where the base model falls short, and establish evaluation criteria before curating examples. Fine-tuning can take hours or days, and its first result may perform worse than the base model. section 9.2
When preparing examples for fine-tuning:
- Keep training, validation, and test data distinct in purpose. Training examples update model parameters; validation examples monitor performance and help tune settings; test data measures final accuracy. section 9.3.1
- Curate high-quality examples that show varied ways of handling the target task, including known shortcomings and edge cases. The required amount and format depend on the model; a small API minimum does not mean the dataset will meaningfully change an LLM. section 9.3.1
- Check the file and example format before training. For chat examples, inspect message roles and content; analyze example counts and token use when estimating training cost. section 9.3.1
Emoji-only responses
Suppose a chatbot must answer with a task-specific emoji format rather than ordinary prose. Each curated conversation pairs a user message with the desired assistant response. Store the examples in the format required by the chosen fine-tuning API, then compare the trained model with the base model and the task baseline. The illustrated JSONL conversations are examples of a particular chat-style format, not a universal dataset schema. section 9.2 section 9.3.1
Listing 9.1 JSONL example
{"messages": [{"role": "system", "content": "You're a chatbot that only responds with emojis!"}, {"role": "user", "content": "I heard the koala habitat was destroyed by the fire."}, {"role": "assistant", "content": "(sadkoala)" }]}
{"messages": [{"role": "system", "content": "You're a chatbot that only responds with emojis!"}, {"role": "user", "content": "I've been working nonstop and need a break."}, {"role": "assistant", "content": "(tired)" }]}
{"messages": [{"role": "system", "content": "You're a chatbot that only responds with emojis!"}, {"role": "user", "content": "I just finished reading an amazing book!"}, {"role": "assistant", "content": "(like)" }]}
The JSONL listing shows individual chat-training examples with system, user, and desired assistant messages. section 9.3.1
Data quality is also a governance concern. Diverse, representative datasets and bias checks can help mitigate unfair outcomes; accurate, current data can reduce exposure to hallucinations. Neither measure eliminates hallucinations, so evaluate outputs and retain other safeguards when deploying an application. section 13.3.3 section 13.1.2
