When an organization has a large backlog of unlabeled records, manual annotation may be too costly to cover it. Weak supervision uses programmatic rules and heuristics, called labeling functions, to generate training labels. It is most useful for high-volume categories with clear rules: labels are cheaper and faster to produce, but generally less reliable than human labels. section 3.4.2 section 3.4
A rule-based labeling workflow has three steps:
- Write labeling functions for specific signals, such as category keywords, IP address patterns, or hardware failure phrases. section 3.4.2
- Apply every function to the unlabeled corpus and collect its votes; functions can abstain when their signals are absent. section 3.4.2
- Assign a label when a clear majority of non-abstaining functions agree. Route records without a clear majority to a human. section 3.4.2
IT ticket triage
For exported ServiceNow and Jira tickets, one function might vote for network when it finds “VPN,” “DNS,” “firewall,” or “latency.” Another might vote for access on “reset my password,” “locked out,” or “MFA.” Individual rules can misfire: a broken-laptop ticket might mention VPN only in passing. Combining votes helps limit reliance on any one noisy rule. section 3.4.2
Monitor the share of tickets receiving no majority label as a calibration diagnostic. For ticket triage, roughly 10%–25% is a suggested range: 40% may indicate rules are too narrow or poorly calibrated, while 2% may signal overconfidence in low-precision rules. These are practical indicators, not guarantees of label quality. section 3.4.2
Table 3.2 Different data acquisition strategies and what is best used for
|
Strategy
|
Relative cost
|
Label quality
|
Best used for
|
|
Human
annotation
|
High
|
Highest
|
Seed data,
evaluation sets, high-stakes edge cases
|
|
Weak
supervision
|
Low
|
Moderate
|
High-volume
categories with clear labeling rules
|
|
Synthetic
generation
|
Medium
|
High (with
gate)
|
Rare
categories, gap-filling, controlled variation
|
|
Production log
mining
|
Low
|
Variable
|
Distribution
alignment, temporal freshness
|
Compare weak supervision with human annotation, synthetic generation, and production log mining by cost, label quality, and best use. section 3.4
Use human annotation for seed data, evaluation sets, and high-stakes edge cases; use weak supervision to scale clear rules across a larger corpus. Personal data must be scrubbed before training examples reach annotators, annotation platforms, or training infrastructure. section 3.4 section 3.9
