Kafka Schema Registry

Kafka Schema Registry provides a RESTful interface for storing and retrieving schemas.

Schema Registry can store and retrieve Avro schemas.

Starting in version 6.0.0, Schema Registry can store and retrieve Avro, JSON Schema, and Protobuf schemas.

When you implement Kafka Schema Registry for your data schemas, they can self-evolve for compatibility with downstream consumers.

Kafka Schema Registry acts as a standalone serving layer for metadata, interacting with both the producer and consumer. It stores schemas for keys and values of records.

Kafka Schema Registry enables you to perform the following tasks:

  • Store a versioned history of all schemas.
  • Provide multiple compatibility settings.
  • Support schema evolution according to the configured compatibility settings and the expanded support for the schema format.
  • Provide serializers that interface with Kafka clients and manage schema storage and retrieval for Kafka messages that are sent in one of the supported schema formats.
  • Develop your own custom formats to use with this interface.

You can also perform these tasks:

  • List schemas by subject and also list all versions of a subject (schema).
  • Retrieve a schema by version or ID.
  • Retrieve the latest version of a schema.
  • Verify that a schema is compatible with a certain version.

Architecture

Kafka Schema Registry is designed to be distributed with a single master architecture. ZooKeeper coordinates the master election, based on the configuration. Kafka-coordinated master election is not currently supported.

HPE Ezmeral Data Fabric Streams is designed to be the durable backend for schema registry, providing a write-ahead change log for the state of schema registry and the schemas it contains.

Interoperability

Kafka Schema Registry can interface with the following components:

  • Kafka Client (producer, consumer APIs)
  • KStreams
  • KSQL
  • Kafka Connect
  • Kafka REST

Performance and Scalability Impact

You can improve performance by decreasing the size of the message payload. Without Kafka Schema Registry, the message payload contains the user data and the schema metadata. With the Kafka Schema Registry, the message payload contains the user data and only the schema ID that is unique for each schema.

For scalability, you can launch Kafka Schema Registry on several nodes.

For More Information