ksqlDB

As explained in Kafka for Architects

ksqlDB is a Confluent streaming database system that defines and executes continuous event-processing queries using a SQL-like language over Kafka data.

ksqlDB provides a SQL-like way to define and deploy continuous event-processing queries over Kafka data. It addresses the administrative and coding overhead of creating a full application by running declarative KSQL queries continuously on a server cluster. section 8.4.1

Traditional SQL usually queries a static database snapshot. ksqlDB instead evaluates queries against an ongoing flow of events, making filtering, trend detection, and aggregation available as changes arrive rather than only after batch processing. section 8.4.1

Typical ksqlDB definitions include:

• Declaring an Orders stream with fields, a Kafka topic, JSON value format, and eight partitions. section 8.4.1
• Creating Expensive_Orders by selecting orders whose amount_paid exceeds 100 and emitting changes continuously. section 8.4.1
• Building a table that counts orders by country, optionally using a one-hour tumbling window for time-bounded aggregation. section 8.4.1

Figure 8.18 In the ksqlDB server, the KSQL query is transformed into multiple instances of Kafka Streams applications. These applications consume events from the Orders topic, process the data, and produce the results to the Orders_Count_By_Country topic.

The architecture view shows how a ksqlDB server turns a query into multiple Kafka Streams application instances that consume, process, and publish Kafka records. section 8.4.1

How Kafka for Architects covers it

How deep it goes

Explains continuous queries, stream definitions, filtering, aggregation, windows, and Kafka Streams execution, but not low-level implementation.

Examples

Uses order filtering, an Orders stream definition, country counts, and one-hour tumbling-window aggregation.

What it assumes

Assumes Kafka-based event data and familiarity with SQL-like selection, filtering, grouping, and aggregation concepts.

What you can do after

Apply via ksqlDB by defining, deploying, and continuously evaluating SQL-based filtering and aggregation queries.

How it connects

Connects SQL-like queries with Kafka, Kafka Streams, stream processing, aggregation, and windowing.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA ksqlDB server is the server-side service that executes ksqlDB statements and manages continuously running stream-processing applications.

Compare how 2 books explain ksqlDB →

Related concepts

• Kafka Streamsuses · Kafka Streams is a client library for building applications that continuously process data from and write data to Apache Kafka topics.
• SQLuses · SQL is a standardized declarative language for defining, querying, and manipulating data and structures in relational database systems.
• Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• KSQL queryuses · A KSQL query is a SQL-based query that continuously processes streams or tables backed by Kafka topics.
• schema registryuses · A schema registry is a service that stores, versions, and provides schemas for structured data, often validating compatibility between schema versions.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.