Chapter 6: The fine print

Introduction

The chapter introduces reasons to examine AI claims critically and consider the broader effects of using or building AI products.

• Critical awareness of AI claims: Explain why the author urges readers to assess AI announcements critically and consider the broader effects of AI work.
  Why it matters: This perspective helps you question AI claims and think beyond a product’s immediate benefits.

Key terms: exaggeration, speculation, deception, broader effects

6.1 Copyright

This section explains the main copyright disputes around AI training data and generated work, while emphasizing where the legal outcomes remain unsettled.

• Two infringement allegations: Distinguish the allegation that models reproduce copyrighted material from the allegation that providers use copyrighted material for training without authorization. Explain how memorization from overfitting can cause verbatim reproduction.
  Why it matters: The two allegations raise different questions, so reducing copied output would not settle the training-data dispute.
• Fair use and competition: Explain why fair use is central to the unauthorized-training-data dispute and how the possibility of replacing or competing with an original product figures in the examples.
  Why it matters: It explains why an argument that models merely learn patterns does not, by itself, end the copyright dispute.
• Proposed training data opt-out: Describe the author’s suggested possible outcome in which data owners indicate through machine-readable metadata that they do not want their content used for AI training, analogous to robots.txt for search engines.
  Why it matters: The proposed opt-out shows one way the author thinks disputes over training data might be addressed.
• Copyright in generated work: Distinguish work solely generated by AI from a work containing protectable human-authored elements, and describe the uncertainty the author raises about prompt writing and refinement.
  Why it matters: Copyright questions concern ownership of AI-generated output as well as the data used to train AI.

Key terms: copyright infringement, verbatim, training data, memorizes, overfitting, fair use, competing products, replacement, scraping, opt-out requests

6.2 Economics of AI

This section questions whether AI’s impressive revenues, new products, and promised productivity gains will translate into durable profits or broad economic benefits.

• Revenue versus lasting profitability: Explain why high revenue does not by itself make an AI provider a successful business, using the section’s discussion of costs and losses.
  Why it matters: Impressive sales figures can conceal a business that still needs outside funding.
• Competitive moats in AI: Explain why the section doubts that either large model providers or thin AI-wrapper businesses can reliably protect their profits from competitors.
  Why it matters: A useful AI product may still struggle to earn lasting profits if competitors can readily copy it.
• Promised versus observed productivity: Distinguish forecasts of large AI-driven productivity gains from the economic evidence described in the section, and explain why business gains may be harder to realize.
  Why it matters: Forecasts of AI’s economic value should not be mistaken for gains businesses have already achieved.

Key terms: revenue, profitable, training, serving, moat, competitive advantage, AI providers, AI wrappers, foundation models, productivity

6.3 Smoke and mirrors

The section uses hidden human labor and overstated model capabilities to show why impressive AI announcements deserve scrutiny.

• Hidden human assistance: Explain how human involvement can make an AI system appear more autonomous than it is, using the examples of self-driving cars and automated shopping receipts.
  Why it matters: Knowing where people remain involved helps you judge claims that a system works on its own.
• Overstated language-learning claim: Describe why the Bengali-language announcement was misleading and how it illustrates a form of AI hype beyond hidden human labor.
  Why it matters: Checking the premise behind an extraordinary claim can change what the result actually demonstrates.
• Thinking claims versus described process: Explain the process the section describes behind OpenAI o1’s advertised thinking, and distinguish that account from the author’s speculation about its reinforcement-learning training.
  Why it matters: Looking past words such as “thinking” to the described process makes AI announcements easier to evaluate.

Key terms: Mechanical Turk, self-driving cars, just walk out, Bengali, training data, emergent properties, OpenAI o1, chain-of-thought prompting, reinforcement learning, RLHF

6.4 Regulation

Introduces the EU AI Act and explains why its reach matters to people building AI products.

• EU AI Act reach: Explain when the EU AI Act can apply to an AI product, including one developed or run outside the EU.
  Why it matters: An AI product built elsewhere may still be subject to the Act if it or its outputs are used in the EU.

Key terms: AI Act, inside the EU

6.5 Prohibited AI practices

This section identifies AI practices the Act prohibits because they are considered serious violations.

• Prohibited AI practices: Describe why the Act prohibits certain AI practices and name the examples given: manipulation or deception, exploitation of vulnerabilities, and social scoring.
  Why it matters: These examples show what the Act treats as unacceptable uses of AI, not merely uses that need extra precautions.

Key terms: prohibited AI practices, manipulate or deceive, vulnerabilities, social scoring

6.6 High-risk systems

The section identifies which systems the EU AI Act treats as high risk and explains its human-oversight requirement.

• High-risk system categories: Describe the kinds of products and AI applications covered by the Act’s high-risk provisions.
  Why it matters: These examples show that the provisions cover both existing regulated products and additional AI uses.
• Human oversight requirement: Explain what human oversight must enable people to do with a high-risk system.
  Why it matters: Oversight means people can critically assess a system’s output and retain the ability to reject or stop it.

Key terms: high-risk products, third-party conformity assessment, targeted job ads, visa applications, human oversight, capabilities and limitations, over-reliance

6.7 Transparency obligations

The section explains when the AI Act requires disclosure of AI use, when disclosure is unnecessary, and how disclosure can work for AI art.

• AI interaction disclosure requirements: Explain that companies must generally tell users when they are interacting with an AI system, including a system that generates or manipulates deepfake images, audio, or video. State that this can apply even when a system is not classified as high risk, but disclosure is not required when AI use is obvious or the system is used for legal purposes such as crime detection.
  Why it matters: A system does not have to be high risk for its users to need notice that AI is involved.
• Reviewed content and AI art: Explain that disclosure is no longer required for AI-generated content that a person thoroughly reviews and takes editorial responsibility for. Describe how AI use in art can be indicated without hampering the work’s display or enjoyment.
  Why it matters: The section distinguishes an exception for editorially reviewed content from a flexible way to disclose AI use in art.

Key terms: transparency obligations, deep fake, high risk, editorial responsibility, AI art

6.8 Foundation models

This section outlines EU AI Act rules for foundation models, including documentation, systemic-risk obligations, and training on scraped data.

• Foundation model documentation: Explain that the Act calls foundation models “general-purpose AI models” and requires providers to document their development, including data used for training, testing, and validation and known or estimated energy consumption.
  Why it matters: These are concrete obligations the section identifies for foundation model providers.
• Systemic-risk model obligations: Describe how the Act identifies a special category of very large foundation models by a training-computation threshold, and what their creators must do.
  Why it matters: The systemic-risk category adds obligations beyond the general documentation requirement.
• Scraped-data training allowance: Explain the condition under which the Act allows training on scraped data without authorization and how the section says that allowance was established.
  Why it matters: The allowance comes with an opt-out condition and rests on a directive whose scope is disputed.

Key terms: foundation models, general-purpose AI models, AI provider, systemic risk, floating-point operations during training, scraped data, opt-outs, data mining

6.9 Resource consumption

The section examines the electricity, carbon emissions, and other resources involved in using and training AI models, while explaining why estimates need careful interpretation.

• Limits of consumption estimates: Explain why AI electricity consumption is difficult to gauge and why the section draws on an independently measured study.
  Why it matters: The source and method behind a consumption figure affect how much confidence to place in it.
• Per-use electricity comparisons: Describe what the reported use-phase electricity figures count and the main qualifications on comparing text and image generation.
  Why it matters: A per-use average is useful for perspective, but it is not a universal energy cost for every AI request.
• Electricity and carbon emissions: Explain why the environmental concern cannot be judged from electricity use alone and how the section puts emissions in perspective.
  Why it matters: The same electricity use can have different carbon consequences depending on its power source.
• Training and cooling resources: Describe the resource demands beyond electricity used for individual AI responses, using the section’s examples.
  Why it matters: Considering only the electricity for a response misses other substantial demands of providing AI.

Key terms: electricity consumption, open source models, GPUs, per 1,000 uses, text generation, image generation, variability across models, CO2, carbon intensity, kWh

6.10 Brains and consciousness

The section explains why comparisons between AI and brains, and confident claims about AI consciousness, deserve caution.

• Limits of brain knowledge: Explain why current knowledge of neurons and their connections does not provide a complete account of how brains work.
  Why it matters: The limits of brain science make claims of close AI–brain correspondence harder to substantiate.
• Caution with brain analogies: Explain why similarities between AI models and biological brains should not be taken as evidence that they work the same way.
  Why it matters: A brain analogy can be suggestive without establishing how an AI model actually works.
• Uncertainty about AI consciousness: Explain why the section rejects definitive claims about whether AI can be conscious.
  Why it matters: Unresolved scientific and philosophical questions make certainty about AI consciousness premature.

Key terms: neurons, dendrites, neuromodulators, connectome, artificial neural networks, convolutional neural networks, visual cortex, brain analogy, consciousness, computation

6.11 Summary