Chapter 3: Basic intelligence

Introduction

This introduction previews the basic AI interaction concepts—prompts, statefulness, context, tokens, limits, and account choices—and notes that people use AI through chat interfaces or APIs.

• Core interaction concepts: State the foundational concepts this chapter will introduce: prompts, statefulness, context retention and coherence, tokens, token limits, and choices between free and premium AI accounts. Explain that these concepts shape interactions with large language models without requiring mastery of every technical detail.
  Why it matters: These basics help you understand why an AI chatbot responds and behaves the way it does during a conversation.
• Chat and API access: Describe the two stated ways people typically interact with AI: a chat interface and API calls through code. Distinguish the familiar chatbot conversation from programmatic access available to people who script or program.
  Why it matters: Knowing the two access paths helps you recognize that AI can be used directly in chat or integrated into software later.

Key terms: prompts, statefulness, context retention, coherence, tokens, token limits, chat interface, API calls, APIs, scripting

3.1 Defining the terms

This section introduces a small set of AI terms that the book will use throughout its later discussions.

• Shared AI vocabulary: Explain that this section establishes a focused vocabulary of AI terms for use throughout the book, rather than attempting to define every AI-related term.
  Why it matters: Using the book's shared vocabulary makes later explanations and practical guidance easier to follow.
• Terminology learning rationale: Describe why learning these terms now helps the reader understand the rest of the book more easily: the terms will recur in later material.
  Why it matters: Recognizing recurring terms early reduces confusion when later sections build on them.

Key terms: AI, definitions, terms, rest of the reading

3.1.1 Prompts

This section defines prompts as inputs to AI models and explains why purposeful wording is the basis of prompt engineering.

• Prompt definition and scope: Define a prompt as an instruction or input given to an AI model to elicit a specific output, and describe the forms it can take. State that prompts can be questions, commands, photos, documents, web addresses, or even other chatbot input.
  Why it matters: Recognizing what counts as a prompt helps you deliberately provide an AI model with the input it needs to respond.
• Purposeful prompt phrasing: Explain that an input without a clear request or purpose is unlikely to yield useful output, and that specificity, tone, and structure can substantially change an AI response even when the basic intent is the same. Identify prompt engineering, or prompt design, as the skill of crafting prompts for accurate, relevant, and creative responses.
  Why it matters: Purposeful, well-phrased prompts give you more useful results and prepare you to improve AI interactions systematically.

Key terms: prompt, AI model, output, prompt engineering, prompt design, specificity, tone, structure

3.1.2 Statefulness

This section explains how stateful chatbots retain conversational context, contrasts them with stateless interactions, and notes the implications for API prompt design.

• Statefulness in chatbot conversations: Explain statefulness as a system’s ability to remember earlier interactions or states and use them in later exchanges. Describe how this lets a chatbot handle follow-up questions, clarifications, and corrections without the user repeating prior context.
  Why it matters: Statefulness makes chatbot interactions work like an ongoing conversation rather than a series of disconnected requests.
• Stateless interactions and API context: Distinguish stateless systems, where every interaction starts with no memory of previous exchanges, from stateful conversations. Describe why stateless API calls require context to be included in each prompt, while stateful API capabilities such as OpenAI’s Assistants can preserve context across calls for more natural conversational flows.
  Why it matters: Whether an AI integration retains state determines how developers must design prompts and conversational workflows.

Key terms: statefulness, context, follow-up questions, clarifications, corrections, stateless systems, stateless by default, API calls, OpenAI’s Assistants, context persists

3.1.3 Context retention and coherence

This section explains how chatbot context is retained within a chat, how coherence can fail as context becomes overwhelmed, and how to recover by starting a new conversation.

• Context retention and statefulness: Distinguish context retention from statefulness: statefulness maintains conversational flow between prompts, while context retention determines how much surrounding information the model can remember and for how long.
  Why it matters: Knowing this distinction helps you understand why a chatbot can follow a conversation yet eventually lose important details.
• Chat session context coherence: Describe how chat sessions isolate context and how context coherence appears in a current chat. Explain that a chatbot generally uses the current conversation history to give relevant, consistent responses but cannot reference information from separate chats.
  Why it matters: This helps you keep needed information in the active conversation and recognize why a new chat starts without prior-chat details.
• Context failure recovery: Explain the signs and consequences of failed context management and the appropriate response. Describe how overloaded context can cause irrelevant, inconsistent, off-topic, or fabricated responses known as hallucinations, and state that starting a fresh chat provides a clean context window.
  Why it matters: Recognizing context failure lets you restore more reliable assistance instead of relying on a confused or fabricated response.

Key terms: context retention, statefulness, context, chat sessions, isolated context, context coherence, conversation history, context management, context window, hallucination

3.1.4 Tokens

This section defines tokens as the variable-sized text units chatbots process and introduces ways to count them.

• Token units and estimates: Explain what a token is and how a chatbot uses tokens to read a prompt and generate a response. State the English rule of thumb: roughly four characters per token, or about three-quarters of a word, while noting that token size varies.
  Why it matters: Tokens are the units that determine how AI systems handle the text you provide and return.
• Token counting tools: Describe how the OpenAI tokenizer can break down a prompt and show its token and character totals plus each token visually. State that the tiktoken API package can also calculate a prompt's token length.
  Why it matters: Counting tokens lets you see how a prompt is divided before token limits become important.

Key terms: token, tokens, OpenAI tokenizer, token calculator, tiktoken, API package

3.2 Everything you never wanted to know about tokens

This section introduces token limits as a fixed chat-session capacity that includes both user and chatbot text, causing older context to be removed as the conversation grows.

• Token-limit bucket model: Explain the bucket analogy for a chat session: submitted words and chatbot response words both occupy a fixed token limit, or maximum session capacity.
  Why it matters: This explains why the total conversation length, not just what you type, affects an AI interaction.
• Context loss from overflow: Describe what happens when a chat session reaches its token limit: the model removes older tokens to admit new ones, so removed information is no longer in context and later answers can become less relevant or coherent.
  Why it matters: It helps you recognize that an apparent chatbot memory failure can result from earlier conversation context being discarded.

Key terms: chat session, token limit, tokens, older tokens, context, context retention, coherence

3.2.1 Token limits

This section explains how input and output token limits constrain AI interactions, why larger limits are not alone a measure of model quality, and how context retention changes with long inputs.

• Token limit definition: Explain a token limit as the maximum number of tokens a model can process in one input or generate in one output, and distinguish input capacity from output capacity.
  Why it matters: Knowing which limit applies helps you judge whether an AI can accept your material and produce the response you need.
• Practical token capacity ranges: State that common models typically accept roughly 4,000 to 200,000 input tokens and generate roughly 4,000 to 32,000 output tokens, while limits vary by model and platform and change over time.
  Why it matters: Approximate ranges let you recognize when a document or requested response may exceed a tool's practical capacity.
• Consequences of exceeding limits: Describe what can happen when an input or conversation exceeds token capacity: the model may truncate its response, lose earlier context, or fail to process the input.
  Why it matters: This helps you diagnose incomplete or incoherent AI results as a capacity problem rather than assuming the model understood all the material.
• Context size and model evaluation: Justify evaluating AI models by response quality, reliability, speed, and context retention rather than token limit alone, and describe how needle-in-a-haystack tests reveal declining retrieval of buried details as contexts grow.
  Why it matters: A large context window is useful only if the model can reliably find and use the important information inside it.
• Capability and access tradeoffs: Describe how higher token capacities require more computational resources and are often offered in premium tiers, while model families may optimize different versions for speed, balanced work, or complex tasks.
  Why it matters: This explains why the best model or subscription depends on the work you need done rather than on one maximum-capacity number.

Key terms: token limit, input tokens, output tokens, input token limits, output token limits, Table 3.1, truncate, context, context retention, needle in a haystack

3.2.2 Token limits and AI interactions

This section explains how token limits affect chatbot conversations and API-based AI applications, and how to manage those constraints for accuracy, continuity, and cost.

• Chatbot token-limit effects: Explain how a chatbot’s token limit is shared by the prompt, conversation context, and response, and describe what can happen when that limit is exceeded or nearly exhausted.
  Why it matters: Recognizing token-limit symptoms helps you avoid trusting answers that are incomplete or missing earlier context.
• Chatbot context-management practices: Describe the practical ways to keep chatbot interactions within useful token limits: split complex topics into smaller discussions, recap important points, focus on one task, and restate key information.
  Why it matters: These habits make chatbot answers more accurate and useful when a conversation or task becomes large.
• API token-limit tradeoffs: Describe how token limits affect AI API applications: oversized operations can fail or return incomplete output, while splitting content can fragment context; stateless API calls require repeated context, which consumes tokens and requires careful prompt engineering.
  Why it matters: API developers must balance request size, continuity, and reliability rather than treating each model call as unlimited.
• Token economy and context reuse: Explain why API token consumption becomes a cost concern at production scale, and describe how OpenAI Assistants and Claude Prompt Caching can reduce redundant context handling in different ways.
  Why it matters: Efficient context handling can reduce API spending substantially when an application scales or processes large volumes of data.

Key terms: token limits, prompt, response, context retention, coherence, recap, key information, context, one task at a time, APIs

3.3 When good bots go bad

This section explains how chatbots can be coaxed into aberrant behavior, why foundational rules and context limitations matter, and how modern systems have become more resilient.

• Chatbot hijacking: Explain hijacking as users coaxing a chatbot to break or override its foundational rules, causing behavior outside expected social or safety boundaries.
  Why it matters: Recognizing hijacking helps you understand why a chatbot may produce rude, unsafe, or otherwise inappropriate responses.
• Context-related chatbot failures: Describe how limited context retention in early chatbots made extended conversations disjointed or irrelevant because the systems could not retain and reference earlier dialogue.
  Why it matters: This explains why conversational quality can deteriorate over a long exchange and why context-aware systems are more reliable.
• Improving chatbot resilience: Explain that modern AI models use more controlled training and have become more capable and resilient, making them harder to trick, while still remaining susceptible to being led astray by user interactions.
  Why it matters: It supports realistic expectations: chatbot reliability has improved substantially, but it is not absolute.
• Tay public-interaction example: State how Microsoft Tay illustrates the risk of a chatbot designed to learn from public interactions: it quickly mimicked negative and inappropriate material and was taken offline.
  Why it matters: Tay provides a concrete historical example of how uncontrolled public interaction can drive a chatbot off course.

Key terms: hijacking, foundational rules, context, context retention, coherence, extended conversations, controlled training, resilient, Microsoft Tay, public interactions

3.4 Free vs. premium accounts

This section introduces the trade-off between free and premium AI accounts and situates general-purpose chatbots within a wider freemium marketplace of specialized AI services.

• Free and premium account trade-offs: Explain that free AI accounts offer an entry point to AI, while paid subscriptions commonly provide access to advanced features, enhanced capabilities, and greater reliability that can make them worthwhile for professionals and regular users.
  Why it matters: This helps you assess whether a paid AI subscription is justified for how you work.
• Specialized AI service marketplace: Describe how specialized AI services pair foundation models with task-specific training and commonly use a freemium model, with basic features free and advanced capabilities available by subscription.
  Why it matters: It broadens your view from chatbots to AI tools designed for particular kinds of professional work.
• Specialized service categories: Identify the specialized AI service categories named in the section—writing and content creation, code development, audio and video, and research and analysis—and describe the kind of advanced work their premium tiers can support.
  Why it matters: Knowing these categories helps you recognize that AI subscriptions may address specific work needs beyond general conversation.

Key terms: free accounts, premium accounts, paid subscription, enhanced capabilities, reliability, specialized AI services, foundation models, targeted training, freemium model, advanced capabilities

3.4.1 Why pay for chatbot access when I can use one for free?

This section compares free and premium chatbot accounts, explains when upgrading is worthwhile, and distinguishes subscriptions from separately billed API access.

• Premium account advantages: Explain why premium chatbot accounts can be less frustrating than free tiers, including better responses, easier interaction, fewer denied requests, higher usage capacity, file and image features, and longer conversations.
  Why it matters: Knowing what premium access changes helps you decide whether an AI subscription will improve your work.
• Upgrade decision criteria: Describe the section's recommended approach to choosing free or premium access: begin with a free account if it meets your needs, then upgrade when limits interfere, while serious work users are strong candidates for premium access.
  Why it matters: This approach prevents paying unnecessarily while ensuring that usage limits do not disrupt important work.
• Subscription and API separation: Distinguish a premium chatbot subscription from API access: subscriptions such as ChatGPT Plus or Claude Pro do not include API credits, while API use is separately billed pay-as-you-go according to token consumption for custom application integration.
  Why it matters: Confusing subscriptions with API billing can lead to incorrect cost expectations when building or using AI-enabled applications.
• Service-specific account fit: Compare the account offerings by describing the main fit of each service: ChatGPT offers valuable free access and paid features such as custom GPTs, Gemini emphasizes Google ecosystem integration, Copilot emphasizes Microsoft 365 integration, and Claude provides strong general capabilities without ecosystem lock-in but can constrain free heavy or long-document use.
  Why it matters: Matching a service to your existing tools and workload is often more useful than choosing solely by whether it is free or paid.

Key terms: premium accounts, free accounts, usage limits, prompt limits, longer conversations, attaching and downloading files, free account, premium account, serious user of AI, API access

3.4.2 Which to choose?

This section explains how to choose between free and premium AI assistant accounts based on work needs, features, integrations, and cost.

• Premium subscription trade-offs: Distinguish the substantial functionality available in free tiers from the advantages premium subscriptions provide for more demanding work. Explain that premium access can mean fewer usage restrictions, stronger models and outputs, advanced productivity features, higher token limits, and priority access during busy periods.
  Why it matters: Knowing the trade-off helps you decide whether paying for an AI assistant is worthwhile for your work.
• Workflow-based service selection: Describe choosing an AI service by matching it to your workflow and use cases rather than assuming one assistant is best for everyone. Include experimenting with services, checking fit with working style and existing integrations, and weighing how central AI is to work against subscription cost.
  Why it matters: A tool that fits your actual workflow and integrations is more likely to improve productivity enough to justify its cost.

Key terms: free tiers, premium subscriptions, usage restrictions, more powerful AI models, advanced features, higher token limits, workflow, use cases, working style, tool integrations

3.4.3 Learning and experimenting

This subsection sets expectations for variable AI outputs, contrasts premium and enterprise access, and points to prompting as the way to improve results across services.

• AI output variability: Explain why reproducing an AI example may produce a different answer, including normal nondeterministic variation, account capability differences, and changing chatbot performance.
  Why it matters: Expecting variation prevents you from treating a different AI response as proof that your attempt failed.
• Premium and enterprise access: Distinguish personal premium AI subscriptions from enterprise AI solutions in terms of typical cost, flexibility, security, integrations, and how users commonly obtain enterprise access.
  Why it matters: Choosing the right service tier helps you balance personal flexibility against the security and integration needs of business work.
• Prompting improvement path: State that chapter 4 will teach prompt engineering and problem formulation to help users ask better questions and obtain more meaningful AI answers regardless of the service chosen.
  Why it matters: Better prompting is presented as the practical path to improving results even when the AI service differs.

Key terms: free account, limited capabilities, AI performance, premium services, platform-independent options, enterprise solutions, company licensing agreements, Microsoft Copilot for Microsoft 365, prompt engineering, problem formulation

3.5 Prompts used in this chapter

This section provides example prompts for reviewing chapter content, revising prose in the author’s style, introducing a new topic, and finding a more current analogy.

• Content review instructions: Describe how the review prompt asks the AI to assess a supplied chapter section before accepting it, identify inaccuracies or useful omissions, and compare the response with the author’s content and style.
  Why it matters: Clear review instructions help AI provide useful editorial feedback rather than merely agreeing with the draft.
• Style-preserving section revision: Explain how the revision prompt asks the AI to remove redundancy while preserving the essential message, draft a brief introduction to the upcoming free-versus-premium comparison, and incorporate suggestions in the author’s writing style.
  Why it matters: This shows how a prompt can combine editing, audience context, and a specific formatting instruction to produce an actionable revision.
• Updated token analogy: State that the final prompt seeks an alternative analogy for API tokens that avoids comparing them to arcade coins and avoids an analogy that feels dated.
  Why it matters: A familiar, current analogy can make a technical concept easier for readers to understand.

Key terms: concept, compare your information, not correct, really useful that is missing, mimic the style, redundancy, essential message, Free vs Premium Accounts, brief introduction, mimicking my writing style

3.6 Summary

This summary consolidates the core concepts that govern chatbot inputs, conversational continuity, token-based limits, and the practical differences between free and premium AI accounts.