Implicit feedback

Q: How deeply does Reinforcement Learning from Human Feedback cover implicit feedback?

Defines behavioral signals and their optimization risk, but stops short of specifying a settled measurement or training procedure.

As explained in Reinforcement Learning from Human Feedback

Implicit feedback is information about user preferences inferred from users' actions rather than explicit evaluations.

Implicit feedback infers response quality from user actions rather than explicit evaluations. In chatbot optimization, it can supplement preference data when downstream behavior differs from contractor judgments. section 14.1.1

Behavioral signals can indicate how users evaluated a previous response:

Re-rolling an output may indicate dissatisfaction with the original response. section 14.1.1
Closing the tab may indicate that the response failed to retain the user. section 14.1.1
Writing an angry message may provide a direct behavioral signal of poor response quality. section 14.1.1

Using these signals to make the reward function more specific creates instability risks: reinforcement learning may exploit a smooth behavioral objective. A proposed direction combines pairwise preference data with additional steering loss functions. section 14.1.1

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Building Reliable AI SystemsImplicit feedback is evaluative information inferred from user behavior rather than directly supplied by users.

Compare how 2 books explain implicit feedback →

Related concepts

monitoringused by · Monitoring is the ongoing observation and measurement of a system’s behavior, state, or operational signals to assess its condition and detect changes or problems.
feedbackkind of · Feedback is information about an action, system, or outcome that can inform evaluation, decisions, or subsequent behavior.
click modelproduced by · A click model is a predictive model of the likelihood that a user clicks a particular result, advertisement, or other presented item.
customer satisfactionmeasures · Customer satisfaction is a measure of how favorably customers or users evaluate a product, service, interaction, or outcome in relation to their needs and expectations.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.