liveBook · Manning

Signal fidelity

As explained in Applied Reinforcement Learning

Signal fidelity is the degree to which a producer's signal matches the direction of a reference signal.

Signal fidelity measures how closely a producer’s signal matches a reference, or oracle, signal. In portfolio management, it links a change in the signal to changes in the agent’s state, policy, and performance. section 12.7

Evaluate fidelity with two complementary measures:

  • Mean error against the oracle measures how far predicted values differ. section 12.7
  • Directional agreement measures how often the producer gets the signal’s direction right. section 12.7

Comparing signal producers

In one evaluation, a naive rule agreed with the oracle 44% of the time, an off-the-shelf model 40%, and a fine-tuned model 91%. Their agents’ Sharpe ratios were 2.52, 2.62, and 8.47, respectively. Comparing fidelity with downstream performance helps show whether a stronger result tracks a better signal. section 12.8

Figure 12.7 Signal accuracy against the oracle, compared with trading Sharpe.

Compare directional accuracy with trading Sharpe across signal producers. section 12.8

How Applied Reinforcement Learning covers it

How deep it goes
Defines fidelity through mean error and directional agreement, then compares producers; it does not detail how to construct the oracle.
Examples
Compares a naive rule, an off-the-shelf model, and a fine-tuned model using directional agreement and trading Sharpe.
What you can do after
Evaluate whether differences in portfolio results track differences in signal fidelity.
How it connects
Connects signal quality to oracle signals, state representations, policies, and portfolio performance.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.