Online learning

As explained in Kafka for Architects

Incremental learning is a machine-learning technique that updates a model progressively as new data arrives rather than retraining only from scratch.

Online learning, also called incremental learning, updates a model as new data arrives rather than relying only on a static dataset and scheduled retraining. It fits decisions that must respond quickly to changing patterns, such as fraud detection, personalization, and operational monitoring. section 12.4.1 section 12.4

In a Kafka-based design, a consumer reads events from a topic in small batches and feeds them to the model. The model updates its parameters after each batch. This differs from using Kafka only to collect data for later batch training—or to supply events to an already deployed model for predictions. section 12.4.1

Flink ML offers another route: its unified API supports training on historical data and learning from streams. Its pipeline can include preprocessing, feature engineering, training, and low-latency inference; it provides algorithms for common tasks and interfaces for custom ones. section 8.4.2

The design changes operational responsibilities:

• Learning from arriving events can reduce the need to retain large historical datasets, but only when retaining them is not otherwise required. section 12.4.1
• Continuous updates require safeguards against concept drift, bias, and noise. Discarding consumed data can also make decisions harder to audit or reproduce. section 12.4.1

Figure 12.7 Retraining a machine learning model incrementally

The diagram shows the Kafka-connected model receiving events for incremental retraining, rather than waiting for a periodic batch-training cycle. section 12.4.1

How Kafka for Architects covers it

How deep it goes

Defines parameter updates and outlines Kafka consumer batches and Flink ML capabilities, without detailing algorithm implementations.

Examples

Illustrates a Kafka consumer feeding event batches to a model, with fraud detection, personalization, and operational monitoring as applications.

What you can do after

Apply it via Flink ML to combine historical training with learning from streaming data.

How it connects

Connects batch training, Kafka topics, stream processing, feature engineering, inference, and concept drift.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Related concepts

• traininguses · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.
• out-of-core learningequivalent to · Out-of-core learning is an incremental training technique that processes data from external storage in portions without loading the entire dataset into memory.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• inferencecontrasts with · Inference is the process of applying a trained model to input data to produce outputs or predictions using its learned parameters.
• clientused by · A client is a person, organization, device, or software component that receives or uses a product, service, resource, or functionality provided by another entity.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.