A large language model (LLM) is a generative AI model trained on extensive text data to produce contextually relevant text. It can answer questions, summarize, translate, classify, and generate code without task-specific training for every request. Its fluent output reflects learned patterns, not human understanding or a guarantee of factual accuracy. section 2.2 section 2.8.1
A request begins as a prompt. Tokenization divides its text into units, and embeddings represent those units numerically. The model predicts an output token, adds it to the sequence, and repeats until it produces a completion. Both the prompt and response consume tokens, affecting limits, latency, and cost. section 2.8 section 2.8.2
Common uses include:
- Extracting contact details from a message without training a task-specific named entity recognition model. section 1.3.1
- Building question-answering chatbots, summaries, translations, and other language workflows. section 2.2
- Generating or explaining code, while reviewing the result against software and security practices. section 1.3.4
A small API workflow
An OpenAI chat-completions example sends “Hello, World!” to gpt-3.5-turbo, then sends a second prompt asking the model to translate its response into French. The example limits each response with max_tokens. Its API key is embedded in the illustrative listing; do not treat that shortcut as a production credential practice. section 1.3.4
Configuration helps match output to the task. Max response limits generated length; temperature and top_p influence variation. A concise extraction task may call for predictable output, while creative writing may benefit from more variation. Prompt improvements still cannot fully compensate for a poorly matched model, and long prompts leave less room for a response. section 2.8.5 section 6.7
Table 2.4 LLM configuration settings
|
Configuration
|
Description
|
|---|---|
|
Max tokens
|
This sets a limit on the number of tokens per model response. Depending on the model, a maximum limit is shared between the prompt (including system message, examples, message history, and user query) and the model’s response.
|
|
Temperature
|
This controls randomness. Lowering the temperature means the model produces more repetitive and deterministic responses. Increasing the temperature results in more unexpected or creative responses. Try adjusting the temperature or
top_p but not both. Typically, as sequences get longer, the model naturally becomes more confident in its predictions, and one can use a much higher temperature for long prompts without going off-topic. Conversely, using a high-temperature setting on short prompts can lead to unstable outputs.
|
|
Top probability (
top_p)
|
This is a probability threshold that, like temperature, controls randomness but uses a different method. Lowering
top_p narrows the model’s token selection to the most likely tokens and ignores the long tail of less likely tokens. Increasing
top_p will allow the model to choose from high- and low-likelihood tokens. Try adjusting either temperature or
top_p, but not both.
|
|
Stop sequences
|
This makes the model end its response at a desired point. The model response will end before the specified sequence so that it won’t contain the stop-sequence text.
|
|
Frequency penalty
|
This reduces the chance of repeating a token proportionally based on how often it has appeared in the text. This decreases the likelihood of repeating the same text in response.
|
|
Presence penalty
|
This reduces the chance of repeating any token that has appeared in the text so far. This increases the likelihood of introducing new topics in a response.
|
Compare max tokens, temperature, and top_p when setting response length and variation. section 2.8.5
For enterprise questions about private or current information, retrieval-augmented generation (RAG) pairs the LLM with a retriever. The retriever selects relevant documents; the model uses them as context to form an answer. This is an alternative to custom-training the model on enterprise data, which is more involved. A larger context window alone does not remove accuracy or cost concerns. section 7.3 section 8.0 section 8.1.1
Evaluate answers before relying on them: LLMs can produce coherent but invented claims, particularly when the prompt or context is misleading. Choose a model and deployment approach with the intended workload, data privacy, licensing, and enterprise controls in mind. section 1.4 section 1.1 section 3.4.6
