Signal fidelity measures how closely a producer’s signal matches a reference, or oracle, signal. In portfolio management, it links a change in the signal to changes in the agent’s state, policy, and performance. section 12.7
Evaluate fidelity with two complementary measures:
- Mean error against the oracle measures how far predicted values differ. section 12.7
- Directional agreement measures how often the producer gets the signal’s direction right. section 12.7
Comparing signal producers
In one evaluation, a naive rule agreed with the oracle 44% of the time, an off-the-shelf model 40%, and a fine-tuned model 91%. Their agents’ Sharpe ratios were 2.52, 2.62, and 8.47, respectively. Comparing fidelity with downstream performance helps show whether a stronger result tracks a better signal. section 12.8
Figure 12.7 Signal accuracy against the oracle, compared with trading Sharpe.

Compare directional accuracy with trading Sharpe across signal producers. section 12.8
