Schema registries are not optional: how a single field change breaks production

The outage you didn’t see coming

A producer team renames a field. They test against their own service, update their internal docs, and ship. The consumer team finds out when their dead-letter queue starts filling at 3 a.m. This is not a hypothetical. It is the most common class of silent failure in event-driven architectures, and it happens precisely because schema governance was treated as a nice-to-have rather than a hard contract.

The subtler version is worse. Because of how some JSON and Avro deserializers handle missing fields, a consumer may not throw an exception at all. It assigns null to the variable and keeps running. One well-documented case involved a fraud model making decisions on missing data for nearly a month before anyone noticed. No alerts, no errors, just quietly wrong outputs.

Schema registries exist to prevent exactly this. But a registry alone is not enough, and understanding why requires getting specific about what these systems actually do, and where they stop.

What a schema registry actually does

A schema registry is a centralised store that assigns a unique integer ID to each registered schema version. The Confluent wire format encodes this compactly: every message payload is prefixed with five bytes, one magic byte set to 0x00 followed by a four-byte big-endian schema ID. The payload carries no schema definition, so every consumer performs a registry lookup by ID, cached indefinitely since IDs are immutable.

This design means producers and consumers never need to embed or negotiate schemas at runtime. It also means that if the registry is unavailable and the cache is cold, deserialization fails. The registry is infrastructure, not an afterthought.

Three registries dominate the open-source landscape. Confluent Schema Registry is the original reference implementation, source-available under a license that restricts commercial redistribution. Apicurio Registry (Apache 2.0, Red Hat-backed) supports Avro, JSON Schema, and Protobuf, plus additional formats including OpenAPI, AsyncAPI, and GraphQL; it can store schemas in PostgreSQL or in a Kafka topic via its KafkaSQL storage option. Karapace (Apache 2.0, Aiven) provides a Confluent-compatible API with Kafka-backed storage and a REST proxy, and its documentation describes OIDC authentication with optional role-based authorisation. All three support the serialization formats that matter most in Kafka pipelines: Avro, JSON Schema, and Protobuf.

Compatibility modes: the part most teams misconfigure

This is where the real complexity lives, and where most production incidents originate. Confluent Schema Registry defines four compatibility modes, each with a transitive variant.

• BACKWARD: consumers using the new schema can read data written with the old schema. You can add optional fields or remove fields. This is the default, and the reason it is the default for Kafka is specifically so consumers can be rewound to the beginning of a topic and reprocess historical data.
• FORWARD: consumers using the old schema can read data written with the new schema. You can remove optional fields or add fields. Useful when you need to roll out producers before consumers.
• FULL: both backward and forward simultaneously. Only adding or removing optional fields is permitted. The most conservative choice for long-lived topics.
• TRANSITIVE variants: BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, and FULL_TRANSITIVE check compatibility against every previous schema version, not just the latest. Non-transitive modes check only against the immediately preceding version, which means a chain of individually valid migrations can accumulate into an incompatible total delta.

For Kafka Streams specifically, only BACKWARD compatibility is supported. If you are running Streams topologies, that decision is made for you.

A schema registry without a correctly configured compatibility mode doesn’t prevent bad schemas. It just registers them. It is a ledger, not a bouncer. Compatibility enforcement only activates when a non-NONE mode is explicitly set per subject or globally.

The default BACKWARD mode is not BACKWARD_TRANSITIVE. A team that migrates through three schema versions, each individually backward-compatible with the previous one, may end up with a version-three schema that cannot read version-one data. If consumers can be rewound to topic start, that gap matters. BACKWARD_TRANSITIVE closes it.

The failure modes that don’t page you immediately

The dramatic failure, a deserialization exception that kills a consumer, is actually the easy case. You see it, you roll back, you fix it. The dangerous failures are the ones that keep running.

Consider a type-narrowing change: a producer changes a field from long to int. Most messages fit. A small fraction of values exceed the integer range. In a high-throughput stream, even a 0.1% failure rate means a continuous trickle of failed events into dead-letter queues, hard to distinguish from normal noise, while 99.9% of events process cleanly. The stream looks healthy. The alerts don’t fire. The data is wrong.

Schema drift, where different services accumulate conflicting schema versions over time, compounds this. Analytics queries start returning incorrect results. PII fields that compliance tooling was tracking disappear or move. Reprocessing historical data becomes expensive or impossible because the schema at write time is no longer recoverable.

Schema governance in practice: what the pipeline needs

Running schema governance well requires three things working together: a registry with compatibility enforcement configured, a CI/CD gate that validates schema changes before they reach production, and observability that surfaces schema-related failures distinctly.

The CI/CD gate is non-negotiable. A proposed schema change should be validated against the registry in the pre-production pipeline, before the producer is deployed. If the change violates the configured compatibility mode, the pipeline fails. This catches breaking changes at the point where they are cheapest to fix, before any consumer is affected.

Observability needs to cover rejected registrations, serialization errors, deserialization errors, dead-letter queue volume, and consumer lag in a single incident view. Serialization errors on the producer side and deserialization errors on the consumer side are different signals; conflating them slows diagnosis.

We run NATS JetStream as our own production event-streaming backbone at DIVERSITY. NATS JetStream does not ship a built-in Confluent-compatible schema registry enforcing BACKWARD/FORWARD/FULL compatibility modes; the internal jsm.go schema registry covers API message types for the server itself. For full producer-consumer schema contract enforcement in our managed Kafka-compatible pipelines, we operate dedicated registry infrastructure, and we treat compatibility mode configuration and CI/CD gating as part of the service contract, not optional add-ons. That operational posture, backed by ISO 27001 certification and a contractual 99.9% uptime SLA, is what makes schema governance a reliability guarantee rather than a best-effort recommendation.

Choosing the right compatibility mode for your topic

There is no universal answer, but there are clear heuristics. Use BACKWARD when consumers will reprocess historical data and you control producer rollout. Use FORWARD when producers deploy ahead of consumers and old consumers must keep running during the transition. Use FULL when the topic is long-lived, consumed by many independent teams, and schema changes need to be safe in both directions. Add the TRANSITIVE variant whenever your deployment history spans more than two schema versions and you cannot guarantee consumers are always on the latest.

Set compatibility modes per subject, not just globally. A high-churn internal topic and a public event bus used by downstream partners have different risk profiles and should have different enforcement levels. The registry lets you configure this; use it.

The contract that keeps services independent

Event-driven architectures promise loose coupling: producers and consumers deploy independently, scale independently, evolve independently. That promise only holds if the message contract between them is enforced somewhere. Without a schema registry with active compatibility enforcement and a CI/CD gate in front of it, that contract exists only in documentation, which is to say it does not exist at all.

A schema registry treated as optional will eventually prove it is not. The question is only whether you find out in a code review or at 3 a.m. with a fraud model that has been running on null data for a month.