SQL is a declarative language for expressing data retrieval, transformation, and aggregation. A few lines can replace much imperative or functional code, while leaving efficient execution to the database or processing engine. section 12.3
In relational databases, a query usually returns a result when requested. In stream processing, a query consumes events and typically produces an ongoing, potentially infinite stream of changes. Tables can retain state and expose a complete result at a point in time. section 12.3
Use SQL in stream processing to:
- Connect a table definition to a Kafka topic rather than create an empty database table. section 12.3
- Filter events, such as selecting products whose price exceeds 100. section 12.3
- Aggregate an event stream, such as counting price changes per product. section 12.3
- Balance expressive queries against coupling: shared relational databases can make services depend heavily on one another. section 17.4.1
Filtering and aggregating product-price events
Define a Flink SQL table over the products.prices.changelog Kafka topic, then run `SELECT * FROM productprices WHERE price > 100;` to retain expensive products or `SELECT product_id, COUNT(*) AS num_price_changes FROM productprices GROUP BY product_id;` to count changes per product. section 12.3
CREATE TABLE productprices ( product_id INT, product_name STRING, price FLOAT, oldPrice FLOAT ) WITH ( 'connector' = 'kafka', 'topic' = 'products.prices.changelog', 'properties.bootstrap.servers' = 'localhost:9092', 'format' = 'json', 'key.format' = 'json', 'key.fields' = 'product_id', 'value.fields-include' = 'EXCEPT_KEY' );
Inspect the complete CREATE TABLE definition, including Kafka connector, topic, JSON format, key fields, and schema. section 12.3
