Chapter 10: Mitigating impacts: platforms, individuals, and society
The introduction recaps recommender systems’ benefits and risks and argues that mitigating platform harms requires collective as well as individual action.
After this chapter you will be able to
- Collective action against harms. Explain why limiting platform harms requires collective action as well as individual precautions.
- Platform power over discussion. Explain how platforms influence online discussion directly through decisions about content and users, and indirectly through incentives for promotion.
- Societal reach and responsibility. Explain why platform decisions affect both users and non-users, and describe the resulting question of balancing economic incentives with society’s well-being.
- Unilateral moderation power. Explain how platforms use and change moderation rules, and describe the consequences a ban or restriction can have for users.
- Opaque moderation decisions. Explain why moderation decisions can be difficult for users to understand or contest, including the role of automation and platforms’ reasons for limiting disclosure.
- Procedural justice and moderation. Explain procedural justice and contrast its emphasis on transparent process and recourse with platforms’ emphasis on fast moderation, while acknowledging limits of judicial processes.
- Optimization Targets and Creator Incentives. Explain how platforms’ choice of optimization target shapes what is promoted and how creators respond, using the YouTube example.
- Consequences of Engagement Optimization. Explain why optimizing for engagement is a value choice and how it can affect content and public discourse.
- Alternative Ranking Signals. Describe alternatives to engagement-based ranking and the work needed to adopt them.
- Financial Barriers and Regulation. Explain why platforms may resist alternatives to engagement ranking and describe the regulatory response mentioned in the section.
- Duty of care standard. Explain duty of care and distinguish reasonable prevention of foreseeable harm from eliminating all risk.
- Choice and environmental design. Explain how platform design can shape choices even when users retain the freedom to choose, using the cafeteria analogy.
- Uncertain content harm. Explain why identifying harmful online content can be harder than identifying unhealthy food, using the pandemic example.
- Platform governance trade-offs. Explain the unresolved trade-off between platform decisions about amplification and the risks of government control.
- Platform choice and transparency. Explain why switching platforms is not necessarily a sufficient response to disagreement with moderation.
- Reducing platform dependency. Explain why using recommendation platforms less reduces dependency, and why doing so may be difficult.
- Platform incentives and meta-awareness. Explain what meta-awareness means and why it requires looking beyond platforms’ claims of neutrality and public benefit.
- External accountability and incentives. Explain why supporting policies and candidates favoring platform transparency and external accountability may encourage change over the long term.
- Product design and personal defenses. Explain how understanding platform product design can help people question their own actions and decide whether to reinforce rewarded behavior or content.
- Friction in platform use. Explain how time limits and changes to the physical environment can interrupt habitual platform use.
- Personalization and chronological feeds. Explain why platforms favor algorithm-based personalization and how a reverse-chronological feed can change the user’s experience.
- Reflective online engagement. Explain why pausing before responding is useful in emotional online discussions and what question to ask before engaging.
- Privacy settings and incentives. Explain how users may reduce personal data use and why the relevant settings can be difficult to find and disable.
- Data sharing trade-offs. Explain the trade-off presented by more invasive data sharing and why selling personal data can have lasting consequences.
- Fabricated content and influence. Explain how fabricated online content and profiles can create a misleading impression, using product reviews or political campaigns as an example.
- Playing both sides. Describe the Russian IRA-linked accounts' activity around Black Lives Matter and how the section connects it to polarization and voting.
- Critical online source habits. Describe how to approach important online claims critically, particularly when a source seems to share your views.
- Network Effects and Dependence. Explain how network effects contribute to platform dominance and make it hard for users and creators to avoid platforms.
- Collective Safety and Oversight. Explain why individual action alone may be insufficient to protect people from platform harms, using a public-health or environmental analogy.
- Platform Regulation and Speech. Distinguish the industry and consumer-protection arguments about platform regulation, and explain the free-speech concern.
- Platform lock-in and portability. Explain why leaving an online platform can be costly and how the section contrasts this with phone carrier portability.
- Shared protocols and clients. Explain how shared protocols enable different services or clients to communicate, using email or the proposed social media model.
- Competition and innovation incentives. Explain how protocols could change providers' incentives and lower barriers for new services.
- Distributed moderation tradeoffs. Describe how moderation could shift toward user-facing clients and what consequence this has for harmful posts.
- Independent verification of platforms. Explain why independent evaluators need access to platform data to verify platform claims, and why imperfect audits are still valuable.
- Vetted access to platform data. Describe the section's account of access under the EU Digital Services Act, including the distinction between public and nonpublic data and the concerns around access.
- Political content selection choices. Explain why selecting and classifying political content for an amplification audit involves consequential choices and potential errors.
- Ranking and visibility measures. Explain why counting which side ranks higher can misrepresent attention, and how visibility cutoffs affect the result.
What it builds on
- The pressure from above: VC funding (chapter 9) · Financial pressures influencing platform companies.
- The risk of information pollution: genocide (chapter 7) · The societal risk of harmful information spreading.
- The triangle and the elephant: Measuring amplification (chapter 5) · The challenge of measuring amplification.
- Decisions and consequences (chapter 3) · Decisions about content and their consequences provide context for platforms’ direct power.
- Lack of transparency (chapter 3) · The lack of transparency in content moderation.
- Nuanced decisions (chapter 3) · The difficulty of making nuanced moderation decisions.
- Appeals (chapter 3) · Appeals against moderation decisions.
- Heavy ranking revisited: Training a RecSys (chapter 2) · How recommender systems are trained to rank content
Section by section
Introduction
The introduction recaps recommender systems’ benefits and risks and argues that mitigating platform harms requires collective as well as individual action.
- Collective action against harms: Explain why limiting platform harms requires collective action as well as individual precautions.
Why it matters: This explains why personal precautions alone cannot address the wider power and effects of platforms. - Benefits and harms at scale: Describe the contrasting effects of recommender systems operating at scale on social media.
Why it matters: Both expanded participation and the spread of harmful content matter when judging platform impacts. - Amplification measurement challenges: Explain why objectively and universally measuring algorithmic amplification is difficult.
Why it matters: It helps explain why assessing recommender systems is not simply a matter of finding one universally accepted measurement.
Key terms: collective action, regulatory action, negative externalities, limiting exposure, recommender systems, gatekeepers, harmful and hateful content, algorithmic amplification, personal preferences
10.1 Platforms: power and duty over users
This section explains how platforms shape online discussion and why their power raises questions about responsibility to users and society.
- Platform power over discussion: Explain how platforms influence online discussion directly through decisions about content and users, and indirectly through incentives for promotion.
Why it matters: Recognizing both forms of power helps you see that platforms shape discussion beyond simply removing content. - Societal reach and responsibility: Explain why platform decisions affect both users and non-users, and describe the resulting question of balancing economic incentives with society’s well-being.
Why it matters: Platform choices can have consequences even for people who never use the service.
Key terms: incentives, promoted content, duty of care, economic incentives, society’s well-being
10.1.1 Power over users, and lack of contestability
This section examines platforms’ unilateral moderation power, its consequences for users, and the contrast between opaque moderation and procedural justice.
- Unilateral moderation power: Explain how platforms use and change moderation rules, and describe the consequences a ban or restriction can have for users.
Why it matters: A moderation decision can affect much more than a single post. - Opaque moderation decisions: Explain why moderation decisions can be difficult for users to understand or contest, including the role of automation and platforms’ reasons for limiting disclosure.
Why it matters: Without a clear explanation or recourse, users may struggle to challenge a decision. - Moderation errors and inequality: Describe how moderation decisions can be mistaken or applied unequally, and how that can affect groups’ visibility.
Why it matters: Mistakes and unequal enforcement can determine whose voices are visible online. - Procedural justice and moderation: Explain procedural justice and contrast its emphasis on transparent process and recourse with platforms’ emphasis on fast moderation, while acknowledging limits of judicial processes.
Why it matters: A decision’s fairness depends not only on its result but also on whether people can understand and challenge it.
Key terms: content moderation, terms of service, ban, transparency, recourse, automated decision-making systems, content moderator, differential treatment, procedural justice, appeal
10.1.2 The power of incentives
This section explains how platforms’ ranking incentives shape creators and public discourse, and considers alternatives and barriers to adopting them.
- Optimization Targets and Creator Incentives: Explain how platforms’ choice of optimization target shapes what is promoted and how creators respond, using the YouTube example.
Why it matters: A ranking choice can influence both what people see and what creators produce. - Consequences of Engagement Optimization: Explain why optimizing for engagement is a value choice and how it can affect content and public discourse.
Why it matters: An incentive that works for entertainment can also influence news, politics, and vulnerable users. - Alternative Ranking Signals: Describe alternatives to engagement-based ranking and the work needed to adopt them.
Why it matters: Alternatives can reflect what people value rather than relying solely on what holds their attention. - User Choice of Feeds: Describe how Bluesky’s selectable algorithmic feeds offer different ways to rank content.
Why it matters: Feed choice illustrates an alternative to having one platform-defined ranking for everyone. - Financial Barriers and Regulation: Explain why platforms may resist alternatives to engagement ranking and describe the regulatory response mentioned in the section.
Why it matters: Understanding financial incentives helps explain why promising ranking alternatives may not be adopted voluntarily.
Key terms: optimization target, algorithmic incentives, algorithmic tastemakers, engagement optimization, public discourse, misinformation, bridging content, explicit value judgments, behavioral signals, algorithmic feeds
10.1.3 Duty of care
This section examines duty of care for online platforms and the difficulties of governing content choices that shape users’ experiences.
- Duty of care standard: Explain duty of care and distinguish reasonable prevention of foreseeable harm from eliminating all risk.
Why it matters: This standard frames the question of what platforms reasonably owe their users. - Choice and environmental design: Explain how platform design can shape choices even when users retain the freedom to choose, using the cafeteria analogy.
Why it matters: Freedom to choose does not mean the presentation of options has no influence. - Uncertain content harm: Explain why identifying harmful online content can be harder than identifying unhealthy food, using the pandemic example.
Why it matters: Even when platforms want to reduce harm, deciding which content is harmful can be difficult. - Platform governance trade-offs: Explain the unresolved trade-off between platform decisions about amplification and the risks of government control.
Why it matters: Protecting users raises questions about who should exercise power over public discussion. - Platform choice and transparency: Explain why switching platforms is not necessarily a sufficient response to disagreement with moderation.
Why it matters: Meaningful user choice depends on having comparable alternatives and understanding their policies in practice. - Public service broadcasting model: Describe the public service broadcaster model and its intended independence from government.
Why it matters: The model illustrates an alternative to both private editorial control and direct government control.
Key terms: duty of care, foreseeable injury or harm, reasonable steps, neutral tools, recommender systems, high-visibility shelves, healthy and unhealthy content, misinformation, COVID-19 pandemic, content amplification
10.2 Taking action as individuals
This section explains the limits and tradeoffs of reducing individual dependence on recommendation platforms.
- Reducing platform dependency: Explain why using recommendation platforms less reduces dependency, and why doing so may be difficult.
Why it matters: Reducing use can lessen dependency, but it may also come at a social or professional cost.
Key terms: dependency, addictive nature of platforms
10.2.1 Seeing through the lie: meta awareness
This section explains how awareness of platform incentives and product design, alongside support for external accountability, can help individuals respond to platforms’ influence.
- Platform incentives and meta-awareness: Explain what meta-awareness means and why it requires looking beyond platforms’ claims of neutrality and public benefit.
Why it matters: Seeing platforms’ incentives helps you question how they present themselves and why their services change. - External accountability and incentives: Explain why supporting policies and candidates favoring platform transparency and external accountability may encourage change over the long term.
Why it matters: External accountability can give platforms financial reasons to change when internal incentives do not. - Product design and personal defenses: Explain how understanding platform product design can help people question their own actions and decide whether to reinforce rewarded behavior or content.
Why it matters: Recognizing how a product shapes habits gives you a chance to make more deliberate choices.
Key terms: meta-awareness, incentives, enshittifying, transparency, external accountability, regulation, bottom line, product design, habit, behavior and content is rewarded
10.2.2 Disrupt the disruption
This section presents ways individuals can add friction to platform use and online engagement to resist passive consumption and impulsive responses.
- Friction in platform use: Explain how time limits and changes to the physical environment can interrupt habitual platform use.
Why it matters: Small obstacles can help you notice and interrupt use that might otherwise continue automatically. - Personalization and chronological feeds: Explain why platforms favor algorithm-based personalization and how a reverse-chronological feed can change the user’s experience.
Why it matters: The way a feed is ordered can shape both platform incentives and how you spend your attention. - Autoplay and passive consumption: Describe how autoplay can capture attention and prolong a session beyond what the user intended.
Why it matters: A default setting can keep you consuming content without a fresh decision to continue. - Reflective online engagement: Explain why pausing before responding is useful in emotional online discussions and what question to ask before engaging.
Why it matters: A pause can help you avoid adding to an unhelpful pile-on when emotions are high.
Key terms: friction, time limits, timeout notification, algorithm-based personalization, engagement, reverse-chronological view, autoplay, default choices, passive consumption, confirmation bias
10.2.3 Reclaim privacy
This section explains how individuals can limit personal data sharing, weigh its irreversible costs, and exercise caution about trust online.
- Privacy settings and incentives: Explain how users may reduce personal data use and why the relevant settings can be difficult to find and disable.
Why it matters: Knowing where the friction comes from can help you take control of the data you share. - Data sharing trade-offs: Explain the trade-off presented by more invasive data sharing and why selling personal data can have lasting consequences.
Why it matters: The convenience of more relevant content comes with a data-sharing decision that may be impossible to reverse. - Trust and online interactions: Explain why trust is important but should not be extended indiscriminately, especially online.
Why it matters: Trust enables cooperation, but online interactions can make misplaced trust harder to avoid.
Key terms: targeted advertising, third-party sharing, app settings, data sharing, targeted ads, third party, trust, collaboration, scam artists
10.2.4 “On the internet, nobody knows you are a dog”
This section examines the benefits and risks of online anonymity, illustrates coordinated manipulation, and recommends a more critical approach to online information.
- Benefits of online anonymity: Explain how anonymity can support freer expression and broaden whose ideas are heard.
Why it matters: Anonymity is not just a risk; it can also help people express ideas and be heard. - Fabricated content and influence: Explain how fabricated online content and profiles can create a misleading impression, using product reviews or political campaigns as an example.
Why it matters: What looks like independent feedback or widespread opinion may be a concealed campaign. - Playing both sides: Describe the Russian IRA-linked accounts' activity around Black Lives Matter and how the section connects it to polarization and voting.
Why it matters: Manipulators can exploit opposing groups simultaneously rather than simply argue for one side. - Critical online source habits: Describe how to approach important online claims critically, particularly when a source seems to share your views.
Why it matters: Checking different perspectives can reduce reliance on a persuasive source whose identity or agenda is unclear.
Key terms: anonymity, gatekeepers, generative AI, disinformation campaign, confirmation bias, fake profiles, Internet Research Agency, Black Lives Matter, Blue Lives Matter, polarization
10.3 E pluribus unum: collective action to ensure customer protection
This section argues for collective oversight of powerful platforms, while weighing industry objections and risks to free speech.
- Network Effects and Dependence: Explain how network effects contribute to platform dominance and make it hard for users and creators to avoid platforms.
Why it matters: This explains why people may remain on platforms even when they have little power over their terms. - Generative AI Data Flywheel: Describe how the data flywheel can reinforce the power of centralized generative AI operators.
Why it matters: This illustrates another way platform power may become concentrated. - Collective Safety and Oversight: Explain why individual action alone may be insufficient to protect people from platform harms, using a public-health or environmental analogy.
Why it matters: The distinction helps explain why empowering users cannot substitute for platform oversight. - Platform Regulation and Speech: Distinguish the industry and consumer-protection arguments about platform regulation, and explain the free-speech concern.
Why it matters: This frames the challenge of seeking accountability without giving government unchecked control over speech.
Key terms: network effect, monopoly, power structure, data flywheel, Generative AI, centralized operators, collective intervention, platforms’ safety and oversight, second-hand smoking, self-governing
10.3.1 Platform portability
This section argues that shared protocols could make social platforms more portable, competitive, and open while changing how moderation works.
- Platform lock-in and portability: Explain why leaving an online platform can be costly and how the section contrasts this with phone carrier portability.
Why it matters: Portability addresses the social and content losses that can keep people on a platform they no longer like. - Shared protocols and clients: Explain how shared protocols enable different services or clients to communicate, using email or the proposed social media model.
Why it matters: Shared rules can let people communicate across services without forcing everyone to use the same interface. - Competition and innovation incentives: Explain how protocols could change providers' incentives and lower barriers for new services.
Why it matters: The ease of leaving a service can influence how providers treat users and whether alternatives emerge. - Distributed moderation tradeoffs: Describe how moderation could shift toward user-facing clients and what consequence this has for harmful posts.
Why it matters: Distributing moderation changes both who makes filtering decisions and where excluded content can remain visible. - Transition challenges and precedent: Describe the obstacles to adopting shared protocols and the telephone-network precedent for overcoming resistance.
Why it matters: The telephone precedent shows that an interconnection effort can face strong resistance and still be completed.
Key terms: platform portability, content, contacts, protocol, client, shared and open protocols, enshittifying, competing services, user-facing client, content moderation
10.3.2 Data sharing
This section argues for independent access to platform data and explains the consequential choices involved in auditing political amplification.
- Independent verification of platforms: Explain why independent evaluators need access to platform data to verify platform claims, and why imperfect audits are still valuable.
Why it matters: Independent scrutiny lets the public verify claims rather than relying solely on platforms. - Vetted access to platform data: Describe the section's account of access under the EU Digital Services Act, including the distinction between public and nonpublic data and the concerns around access.
Why it matters: Useful independent research can require sensitive data without making that data public. - Operational transparency without moderation mandates: Explain how disclosing platform operations could encourage accountability without requiring government-directed removal or downranking of content.
Why it matters: Transparency can reveal safety gaps without prescribing how individual posts must be treated. - Political content selection choices: Explain why selecting and classifying political content for an amplification audit involves consequential choices and potential errors.
Why it matters: An audit's conclusions depend on which posts enter the comparison and how they are classified. - Ranking and visibility measures: Explain why counting which side ranks higher can misrepresent attention, and how visibility cutoffs affect the result.
Why it matters: A measure of favored content should reflect what people actually have a chance to encounter. - Content supply and engagement: Explain why an audit must consider differences in how much content each side produces and how much engagement it receives.
Why it matters: Without considering the available content and engagement, an audit can mistake differences in supply for favoritism.
Key terms: third-party auditing, transparency, consumer protection, Digital Services Act, vetted researchers, systemic risks, users’ engagement, reach, transparency requirements, moderators
10.4 Summary
Concepts explained in this chapter
- external auditAn independent examination of an organization's systems, processes, or decisions by an outside party.
- data sharingThe practice of making data available to other people or organizations for access or use.
- duty of careA responsibility to take reasonable steps to avoid causing harm to people who may be affected by one's actions.
- content moderationThe review, filtering, labeling, or removal of user-generated or platform-hosted content according to defined rules.
- algorithm auditAn independent evaluation of an algorithm’s behavior, outputs, or effects according to defined questions or criteria.
- data accessThe ability or permission to obtain and use data held by an organization or service.
- protocolA technical specification defining shared rules for how actions or communications are performed so independently provided services can interoperate.
- online platformAn internet-based service that provides digital functionality and connects users, content, or transactions.
- regulationThe use of legally enforceable rules and oversight to govern conduct in a market or sector.
- automated decision systemA computational system that makes decisions automatically using data, rules, or learned models.
- procedural fairnessA theory that emphasizes the fairness and legitimacy of decision-making processes, including transparency, clarity, and opportunities for recourse.
- engagement optimizationAn algorithm that selects or orders content primarily to maximize user interaction.
- optimization targetAn objective that a recommender system optimizes when ranking content.
- powerThe capacity to influence outcomes, decisions, or public interpretations by controlling resources, processes, or information.
- autoplayAutomatic playback of a media item without requiring the user to initiate playback for that item.
- privacyThe condition of being protected from unauthorized access to or disclosure of personal information and activities.
- polarizationA social or political condition in which opinions, groups, or identities become increasingly divided into opposing extremes.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
