liveBook · Manning

Weak supervision

As explained in LLM Customization and Fine-Tuning

A data-labeling approach that produces training labels from noisy, indirect, or programmatic sources rather than manual annotation.

When an organization has a large backlog of unlabeled records, manual annotation may be too costly to cover it. Weak supervision uses programmatic rules and heuristics, called labeling functions, to generate training labels. It is most useful for high-volume categories with clear rules: labels are cheaper and faster to produce, but generally less reliable than human labels. section 3.4.2 section 3.4

A rule-based labeling workflow has three steps:

  • Write labeling functions for specific signals, such as category keywords, IP address patterns, or hardware failure phrases. section 3.4.2
  • Apply every function to the unlabeled corpus and collect its votes; functions can abstain when their signals are absent. section 3.4.2
  • Assign a label when a clear majority of non-abstaining functions agree. Route records without a clear majority to a human. section 3.4.2

IT ticket triage

For exported ServiceNow and Jira tickets, one function might vote for network when it finds “VPN,” “DNS,” “firewall,” or “latency.” Another might vote for access on “reset my password,” “locked out,” or “MFA.” Individual rules can misfire: a broken-laptop ticket might mention VPN only in passing. Combining votes helps limit reliance on any one noisy rule. section 3.4.2

Monitor the share of tickets receiving no majority label as a calibration diagnostic. For ticket triage, roughly 10%–25% is a suggested range: 40% may indicate rules are too narrow or poorly calibrated, while 2% may signal overconfidence in low-precision rules. These are practical indicators, not guarantees of label quality. section 3.4.2

Table 3.2 Different data acquisition strategies and what is best used for
Strategy
Relative cost
Label quality
Best used for
Human annotation
High
Highest
Seed data, evaluation sets, high-stakes edge cases
Weak supervision
Low
Moderate
High-volume categories with clear labeling rules
Synthetic generation
Medium
High (with gate)
Rare categories, gap-filling, controlled variation
Production log mining
Low
Variable
Distribution alignment, temporal freshness

Compare weak supervision with human annotation, synthetic generation, and production log mining by cost, label quality, and best use. section 3.4

Use human annotation for seed data, evaluation sets, and high-stakes edge cases; use weak supervision to scale clear rules across a larger corpus. Personal data must be scrubbed before training examples reach annotators, annotation platforms, or training infrastructure. section 3.4 section 3.9

How LLM Customization and Fine-Tuning covers it

How deep it goes
Details labeling functions, majority voting, human escalation, and abstention-rate diagnostics.
Examples
Uses exported ServiceNow and Jira tickets with network, access, and hardware labeling rules.
What you can do after
Practice/perform rule-based labeling by writing functions, combining their votes, and routing unresolved records for human review.
How it connects
Connects weak supervision with human annotation, synthetic data generation, and fine-tuning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.