liveBook · Manning

Avro

As explained in Kafka Streams in Action, Second Edition

Apache Avro is a data serialization system that uses schemas and supports language-independent serialization, code generation, and schema evolution.

Avro defines record schemas in JSON and turns those schemas into a serialization contract for event data. A record declares its namespace, type, name, fields, field types, and optional defaults. In a Kafka workflow, the schema is registered under a subject so producers and consumers can agree on the expected shape of a message. section 3.2.5 section 3.9

Listing 3.2 Avro schema for Avengers
{"namespace": "bbejeck.chapter_3",   #1
 "type": "record",                         #2
 "name": "Avenger",                         #3
 "fields": [                                 #4
     {"name": "name", "type": "string"},
     {"name": "real_name", "type": "string"},          #5
     {"name": "movies", "type":
                      {"type": "array", "items": "string"},
      "default": []         #6
    }
  ]
}

The Avenger record shows a JSON Avro schema with required string fields and a movies array whose default is an empty list. section 3.2.5

Generate Java domain objects from an Avro schema rather than hand-writing matching classes. Generated code removes repetitive boilerplate, keeps field types aligned with the schema, and lets a team regenerate objects after a schema change committed to source control. section 3.2.6

Use Avro with Schema Registry as an application pipeline:

  • Register the schema, then build generated Java classes from it with Gradle tasks. section 3.2.9
  • Configure a Kafka producer with KafkaAvroSerializer and the Schema Registry URL; the producer supplies bytes after serialization. section 3.2.9
  • Configure a consumer with KafkaAvroDeserializer and the same registry URL. Enable the specific-record reader to receive generated Avro objects, or leave it disabled to receive GenericRecord values accessed by field name. section 3.2.9
  • Evolve a schema only through changes allowed by the subject's compatibility setting, then register the new version before using it. section 3.2.9 section 3.9

Producing an Avenger event

Create an AvengerAvro value through its generated builder, set the name, real name, and movie list, then send it with the Avro-configured producer. The generated builder permits the schema's expected fields and types; the consumer can deserialize the event back into a specific Avro record and call typed getters. section 3.2.9

Avro is one of three schema formats supported alongside Protobuf and JSON Schema. Avro and Protobuf provide serialization-platform tooling for generated objects, while JSON Schema requires a separate code-generation tool in this workflow. Registry serializers and deserializers cache schemas locally and contact the registry only on a cache miss. section 3.2.6 section 3.9

The same format can also be used when defining ksqlDB data sources, alongside JSON and Protobuf, while building SQL-based event-streaming applications. section 11.0

How Kafka Streams in Action, Second Edition covers it

How deep it goes
Explains JSON record schemas, generated Java objects, serializer configuration, and specific versus generic consumer records; defers compatibility details.
Examples
Uses an Avengers schema and generated AvengerAvro objects, including a Black Widow event sent to and read from Kafka.
What it assumes
Introduces schema registration and serializer roles while assuming a running Kafka broker, Schema Registry, Docker Compose, Java, and Gradle workflow.
What you can do after
Apply it via KafkaAvroSerializer, KafkaAvroDeserializer, Schema Registry, and generated Java records.
How it connects
Connects Avro with Schema Registry, KafkaProducer, KafkaConsumer, Gradle, Protobuf, JSON Schema, and ksqlDB.

Also explained in

Compare how 3 books explain Avro →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.