A streaming table is a continuously updated key-value representation of current state derived from records in a stream. It complements a data stream: the stream captures changes, while the table makes the latest state queryable without consulting an external database. section 12.4.1 section 12.7
Unlike a relational table, it has only a key and a value. Creating a table from a topic reads records from the beginning, updates each key’s intermediate state, and stores that state. In Kafka Streams, builder.table("topic") creates a KTable; Flink provides Tables. section 12.4.1
Operationally, a streaming table involves several linked concerns:
- Partitioning means each application instance knows only part of the table. section 12.4.1
- Local storage, commonly RocksDB, holds intermediate state for fast key-value access. section 12.4.1
- A compacted changelog topic records state changes so the table can recover after a crash. section 12.4.1
- Aggregations such as count() update table values as new records arrive; other application components can query the results or use them in joins. section 12.4.1 section 12.7
Current product or account state
A topic containing product records can become a table whose value for each product key represents its latest state. Similarly, an account transaction aggregation can maintain the current total for each account. New records update the table, while the changelog preserves those updates for recovery. section 12.4.1 section 12.7
Figure 12.6 We can convert streams into tables, and the transformation works in the opposite direction as well.
Use the stream-to-table diagram to trace how continuous records become stored current state, and how the transformation can also run in reverse. section 12.4.1
