A large language model (LLM) is a neural language model trained on a large text corpus to predict and generate natural-language sequences. Modern LLMs generally use a transformer architecture: attention weighs relationships among parts of a sequence, helping the model use context when processing or generating text. section 1.4 section 1.0
The architecture depends on the task. Encoder–decoder models use input context for tasks such as translation; decoder-only models are common for text completion and conversational AI. Changes to attention and positional encoding can extend the length and complexity of context a model handles. Model size alone is not a selection rule: specialized domains may expose weaknesses, while computation and memory costs can complicate deployment. section 1.4 section 1.5
Working with an LLM involves choices beyond its architecture:
- Adapt a pretrained model to a task through fine-tuning; parameter-efficient fine-tuning aims to reduce the resources needed. section 1.4 section 8.0
- Manage training and evaluation costs with techniques such as quantization, pruning, distillation, and sharding; track experiments and tune hyperparameters with tools such as Ray and Weights & Biases. section 8.0
- Check pretraining data and generated outputs for bias or toxicity, and plan safeguards for production use. section 10.5
Generate a response after fine-tuning
Format a question with the model’s prompt template, tokenize it, move the inputs to the selected device, call `model.generate`, and decode the output. A labeled fine-tuning dataset—for example, one specifying a company policy—can change how the model responds. The inference call alone does not establish whether it follows that policy reliably. section 5.2.3 section 5.2.3
Listing 5.21 Running inference on the trained model
prompt = model_tokenizer( #1
[
alpaca_template.format(
"What is the iconic symbol of freedom at the US east coast?", #2
"", #3
"", #4
)
] * 1, return_tensors="pt").to("cuda") #5
generation_parameters = {
"max_new_tokens": 256, #6
"use_cache": True #7
}
outputs = model.generate(**prompt, **generation_parameters) #8
decoded_outputs = model_tokenizer.batch_decode(outputs,
skip_special_tokens=True) #9
The listing shows the complete inference path from a formatted question through tokenization, generation, and decoding; `max_new_tokens` bounds the generated response. section 5.2.3
