Timestamp

As explained in Kafka Streams in Action, Second Edition

A timestamp is a time value associated with a record that indicates when the represented event occurred or is ordered relative to other events.

A timestamp is the time attached to an event record. Along with a key and value, it identifies when an event occurred and lets Kafka retain, process, and compare records over time. If a producer does not supply one, KafkaProducer uses the producer machine’s current system time. section 1.2 section 4.2.6

Choose timestamp semantics for the question the application must answer:

• Event time records when the event occurred; it can come from ProducerRecord metadata or a time field embedded in the value. section 9.4 section 11.2
• Ingestion time records when the broker appends a record, using the topic’s LogAppendTime setting. section 4.2.6 section 9.4
• Processing time records when a record begins flowing through a processing pipeline. section 9.4
• Normalize timestamps to UTC when clients and brokers might use different time zones. section 9.4

Topic configuration determines whether Kafka preserves the producer timestamp (CreateTime) or replaces it at append time (LogAppendTime). Timestamps also support retention: the broker compares a segment’s oldest timestamp with the current time and removes the segment after its configured retention period. section 4.2.6

In Kafka Streams, timestamps drive windows, stream joins, changelog updates, and Processor API punctuation. A TimestampExtractor selects the timestamp semantics used by a topology. Stream time is the highest timestamp the application has observed while processing. section 9.4 section 9.7

Use an embedded event time in ksqlDB

For fitness activity, use the event_time field rather than the producer timestamp when it better represents when a user entered activity on a mobile device. Declare timestamp = 'event_time'; add timestamp_format for a string field, or use a BIGINT event_time for epoch milliseconds. section 11.2

Bound a stream join by time

A 30-minute JoinWindows setting joins a coffee-stream record at 12:00 p.m. with a store-stream record timestamped from 11:30 a.m. through 12:30 p.m. Use before or after to narrow either side of that interval and express event order. section 7.3.4

Figure 9.33 There are three categories of timestamps in Kafka Streams: event time, ingestion time, and processing time.

Compare event time, ingestion time, and processing time before choosing a TimestampExtractor. section 9.4

How Kafka Streams in Action, Second Edition covers it

How deep it goes

Defines producer and broker timestamp behavior, then works through timestamp semantics, extraction, ksqlDB selection, and join windows.

Examples

Uses purchases, sensors, fitness activity, loan processing, and coffee and store records to illustrate timestamp choices.

What it assumes

Introduces timestamps as one of three event components before applying Kafka producer, broker, and Streams APIs.

What you can do after

Apply it via Kafka and ksqlDB configuration to preserve, extract, inspect, or replace the time used for records and processing.

How it connects

Connects event records to keys, values, retention, Kafka Streams, ksqlDB, windows, joins, state, and stream time.

liveBookmanning.com

Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Pandas Workout: 200 exercises to make you a stronger data analystA timestamp object represents a specific date and time as a structured value supporting temporal calculations and comparisons.
• 

  liveBookmanning.com
  Logs and TelemetryA timestamp is a time value associated with an event to indicate when it occurred or was received.
• 

  liveBookmanning.com
  Kafka for ArchitectsA message timestamp is the time value associated with a Kafka message for determining its retention age.
• 

  liveBookmanning.com
  Rust in Action: Systems programming concepts and techniquesA timestamp is a representation of a point in time, commonly expressed as an elapsed value or formatted date and time.
• 

  liveBookmanning.com
  Apache Kafka in ActionA timestamp is a recorded indication of when a Kafka message or record was produced or otherwise associated with a time.
• 

  liveBookmanning.com
  ScyllaDB in ActionA timestamp is a value representing the time assigned to a database write or cell value for conflict resolution.
• 

  liveBookmanning.com
  The Well-Grounded Rubyist, Fourth EditionDateTime is a Ruby class provided by the date library for representing calendar dates together with times and related calendar information.
• 

  liveBookmanning.com
  Grokking Relational Database DesignA TIMESTAMP data type records a date and time representing an exact event moment, typically with automatic time-zone conversion by the database system.
• 

  liveBookmanning.com
  Acing the CCNA Exam: Volume 2 Advanced Networking and SecurityA timestamp is a date-and-time value attached to an event or record to indicate when it occurred.

Compare how 10 books explain timestamp →

Related concepts

• datekind of · A date is a calendar value identifying a particular day.
• time zoneconfigured by · A time zone is a geographic or rule-defined region whose local civil time is determined by offsets and calendar-time rules relative to a reference time standard.
• CPU timecontrasts with · CPU time is the amount of processor execution time consumed by a program, process, thread, or computation.
• database functionkind of · A database function is a named routine stored or executed in a database that accepts inputs and computes a result, often a scalar or table-valued result.
• service timestamps logconfigured by · service timestamps log is a Cisco IOS global-configuration command that configures the time information attached to log messages.

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.