An application instance is one running deployment of a streaming application. It consumes input events, processes them through tasks, produces output events, and hosts local state. Multiple instances can share a group.id, forming a consumer group whose members collectively process input partitions. section 8.1.2 section 8.3.4
An instance has several architectural consequences:
- Scales horizontally by adding or removing instances, but parallelism cannot exceed the number of partitions in the input topics. section 4.3.1 section 8.1.2
- Runs one or more tasks, each combining assigned input partitions, a processing topology, and local state stores; tasks may run on separate threads or instances. section 8.3.4
- Recovers through consumer rebalancing: when an instance fails, its complete task units, including partition assignments and state stores, are reassigned to healthy instances. section 8.1.2 section 8.3.4
- Requires operational monitoring through consumer lag, which indicates the processing backlog and informs decisions about adding instances. section 4.3.1
Orders processing
An Orders streaming application reads raw orders from an input topic, processes them, and writes results to an output topic. Deploying several instances under one group.id distributes input partitions across them; each instance runs assigned tasks and maintains the local state needed by its topology. section 8.1.2 section 8.3.4
Figure 8.16 When a client requests the state for a given key, the instance first checks its locally cached metadata. If the state resides on another instance, the metadata identifies the correct instance to forward the query to.
Use the instance-routing diagram to understand how key metadata locates state on another application instance and how application code forwards a remote query. section 8.3.8
