liveBook · Manning

Materialized

As explained in Kafka Streams in Action, Second Edition

Materialized is a Kafka Streams DSL class for specifying how a stateful operation's result is materialized, including its state-store name and Serdes.

Materialized configures the state store behind a Kafka Streams stateful operation. Use it with aggregations such as reduce and count to supply key and value Serdes, give the store a stable name, select a store implementation, and tune logging or caching behavior. section 7.2.1 section 7.4.7

State stores hold key-value pairs as byte arrays, so Serdes serialize records into the store and deserialize retrieved values. Stateful aggregation results are KTables; convert one with toStream when forwarding results to downstream stream operators or an output topic. section 7.2.1

Use Materialized to make stateful processing explicit and maintainable:

  • Name each store uniquely with Materialized.as("store-name"). Stable names prevent topology edits from changing the on-disk directory and changelog-topic names; duplicate names cause a TopologyException. section 7.4.5
  • Provide Serdes with Materialized.with or withKeySerde and withValueSerde so store keys and values can be serialized. section 7.2.1 section 7.4.6
  • Pass a StoreSupplier, such as Stores.inMemoryKeyValueStore, to Materialized.as to use an in-memory store. It can improve access speed but has bounded memory and restores from its changelog after restart. section 7.4.6
  • Configure a store-specific changelog cleanup policy with withLoggingEnabled, or disable caching for one stateful operation with withCachingDisabled. Disabling logging can leave state unable to restore. section 7.4.7

A named in-memory score total

Group poker-game scores by key, sum each player’s values, and materialize the result in a named in-memory store. The Serdes remain necessary even though the store is not persistent: pokerScoreStream.groupByKey().reduce(Double::sum, Materialized.<String, Double>as(Stores.inMemoryKeyValueStore("memory-poker-score-store")).withKeySerde(Serdes.String()).withValueSerde(Serdes.Double())).toStream().to("total-scores", Produced.with(Serdes.String(), Serdes.Double())); section 7.4.6

A supplied store name also enables Interactive Queries. Configure application.server with a unique host and port per application instance, then query the named local store that owns the relevant partition; state is distributed across instances as tasks are distributed. section 13.2 section 13.3

Listing 13.1 Naming state stores to enable them for querying
Materialized.<String, LoanAppRollup>as(
 Stores.inMemoryKeyValueStore(loanAppStoreName))     #1
                    .withKeySerde(stringSerde)
                    .withValueSerde(loanAppRollupSerde))


Materialized.<String, LoanAppRollup>as(
                     loanAppStoreName)                       #2
                    .withKeySerde(stringSerde)
                    .withValueSerde(loanAppRollupSerde))

Materialized.with(stringSerde, loanAppRollupSerde)    #3

Compare a named in-memory store, a named default persistent store, and an unnamed Serde-only configuration. Only the named alternatives are eligible for Interactive Queries. section 13.2

How Kafka Streams in Action, Second Edition covers it

How deep it goes
Explains Serdes, naming, store selection, caching, logging, and recovery tradeoffs without detailing state-store internals.
Examples
Demonstrates count naming, poker-score reduction with an in-memory store, and loan-application store configurations.
What it assumes
Assumes Kafka Streams stateful operations and introduces the Serde requirement for byte-array state stores.
What you can do after
Apply Kafka Streams Materialized to name, serialize, configure, and query stateful operation results.
How it connects
Connects state stores with aggregations, KTables, Serdes, changelog topics, repartitioning, and Interactive Queries.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.