In-memory database

As explained in Kafka Streams in Action, Second Edition

An in-memory state store keeps its contents in process memory and loses them when the application shuts down.

An in-memory state store keeps key-value records in application memory, giving fast lookups but losing its local contents when the Kafka Streams application shuts down. Kafka Streams backs the store with a changelog topic, so startup can rebuild the state and resume processing. section 7.4 section 7.4.1

Choose an in-memory store when the performance and storage trade-offs fit the topology:

• Gain faster access because lookups do not retrieve values from disk. section 7.4.6
• Limit capacity to available memory; use Stores.lruMap to evict entries at a configured maximum. section 7.4.6
• Expect restoration from the beginning of the changelog after restart or task movement, while persistent stores may resume faster on a clean restart. section 7.4.1 section 7.4.6
• Name the store through Materialized so its state store and changelog topic remain identifiable and compatible across topology updates. section 7.4.5

Materializing an in-memory poker-score store

KStream<String, Double> pokerScoreStream = builder.stream("poker-game",\n Consumed.with(Serdes.String(), Serdes.Double()));\n\npokerScoreStream\n .groupByKey()\n .reduce(Double::sum,\n Materialized.<String, Double>as(\n Stores.inMemoryKeyValueStore("memory-poker-score-store"))\n .withKeySerde(Serdes.String())\n .withValueSerde(Serdes.Double()))\n .toStream()\n .to("total-scores",\n Produced.with(Serdes.String(), Serdes.Double()));\n\nThe StoreSupplier switches the reduce operation from the default persistent store to an in-memory store; the Serdes define its key and value formats. section 7.4.6

Figure 7.17 As the key-value records get written to the store, they also get written to the changelog topic for data durability.

Follow the record path from the key-value store to its changelog topic to see how in-memory state remains restorable. section 7.4

How Kafka Streams in Action, Second Edition covers it

How deep it goes

Explains memory, changelog restoration, capacity limits, naming, and Java configuration, but defers detailed store variants.

Examples

Demonstrates an online poker-game score reduction using Stores.inMemoryKeyValueStore and Materialized.

What it assumes

Assumes Kafka Streams stateful operations, keys, changelog topics, and Serdes while explaining in-memory selection from the configuration point.

What you can do after

Build it from scratch by configuring and naming an in-memory Kafka Streams store with Java APIs.

How it connects

Connects state stores with Kafka Streams, changelog topics, persistent stores, Materialized, Serdes, and interactive queries.

liveBookmanning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Amazon Web Services in Action, Third Edition: An in-depth guide to AWSAn in-memory data store keeps data primarily in volatile memory to provide low-latency reads and writes.
• 

  liveBookmanning.com
  ASP.NET Core in Action, 3eAn in-memory database provider is a database-provider implementation that stores test data in process memory instead of persistent database storage.
• 

  liveBookmanning.com
  Build and Orchestrator in Go (From Scratch)An in-memory store is a storage implementation that keeps its data in volatile memory rather than persistent storage.
• 

  liveBookmanning.com
  Rust Servers, Services, and AppsAn in-memory data store is a data-storage component that keeps its working data in main memory rather than relying primarily on persistent storage.

Compare how 5 books explain in-memory database →

Related concepts

• databasealternative to · A database is an organized collection of persistent data managed by software for storage, retrieval, querying, and modification.
• Redisincludes · Redis is an open-source, in-memory data-structure store that provides network-accessible data operations and supports persistence.
• AddDistributedMemoryCacheconfigured by · An ASP.NET Core service-registration method that configures an IDistributedCache implementation backed by memory.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.