liveBook · Manning

Algorithm audit

As explained in Hidden Influences

An independent evaluation of an algorithm’s behavior, outputs, or effects according to defined questions or criteria.

An algorithm audit lets external evaluators independently test platform claims about algorithmic behavior or effects, rather than relying on the platform’s account. section 10.3.2

Assessment may require access to nonpublic data, such as engagement or reach. Vetted researchers can use such access to study systemic risks, while privacy and intellectual-property concerns remain. section 10.3.2

Independent checks have familiar analogies:

  • Vehicle crash tests provide consumers with an independent guide to safety. section 10.3.2
  • Financial audits let third parties verify company claims using sensitive proprietary information, though conflicts or fraud can undermine the process. section 10.3.2
Figure 10.2 By counting only how many times content from group A is ranked above content from group B, we can see that overall B is ranked higher more times than A. But, A is ranked higher in the most valuable real estate, the top of the list, where users pay the most attention. Users might not even scroll down the full list, making B’s advantage only real on paper.

A ranking assessment can mislead if it counts wins without considering position: appearing near the top may matter more than winning overall. section 10.3.2

Figure 10.3 While a threshold like 5 or 10 might seem a nice round number, and maybe even a natural choice, it’s really just an arbitrary one. By varying the threshold, the result changes.

Results can change when an evaluator varies a threshold, so a seemingly natural cutoff may shape the conclusion. section 10.3.2

How Hidden Influences covers it

How deep it goes
Compares third-party validation and data access, but leaves privacy and intellectual-property concerns unresolved.
Examples
Uses vehicle crash tests and financial audits as analogies, alongside EU platform-data access under the Digital Services Act.
What you can do after
Evaluate platform claims by seeking independent assessment and recognizing that evidence access and safeguards remain contested.
How it connects
Connects platform accountability to data access, systemic-risk research, and third-party auditing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Hidden Influences. The section links point to the parts of the book it draws on.