Schema Registry and Event Contracts

Data Engineering & Modern Data Platforms · Lesson 43 · Real-Time Data Engineering and Kafka

Lesson Overview

commerce.purchase_line left Kafka Connect and Data Integration as plain JSON. A week later commerce added currency on the producer and started writing it. The morning job's from_json schema did not mention currency. Spark's default is to ignore the extra field, so the job stayed green and the warehouse stored every new line as if it were USD. A second change renamed unit_price_cents to amount_cents in one service. That job did not stay green. Half the promotion hour failed, the other half wrote nulls where the parser was permissive, and finance had two definitions of the same hour.

A schema is the structure of one record. A registry is the place that stores those structures, gives each version an identifier, and refuses a version that breaks the compatibility rule you set. An event contract is the rest of the promise: the key, the event-time field, the idempotency key, who may add a field, who upgrades first, and which fields must never appear. The registry will not know that unit_price_cents is money. You write that down beside the schema, or the next compatible change will still be the wrong number.

This lesson uses two public designs and does not invent a third. Confluent Schema Registry stores Avro, JSON Schema, and Protobuf subjects and serves a schema id. Its wire format is a magic byte, that id, and the payload. Apicurio Registry is the other widely used open-source registry and can speak a compatible API. Apache Kafka itself does not embed a registry. Producers and consumers that want one use a serializer from the registry's project, not a class in kafka-clients. Spark 3.5's from_avro takes a schema you supply. It does not call a registry unless the platform you run has documented that extra option. learning.progress stays JSON until you plan a migration, because Structured Streaming with Spark already parses it as JSON.

/learn/kafka covers the record as bytes and a key. This lesson covers what those bytes mean when more than one team writes them. Estate-wide governance, review boards, and mesh ownership are lesson 69. Here you will evolve one order-line schema without breaking the morning job, and you will write the contract that makes the next change reviewable.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate a schema, a registry subject, and an event contract, and say which failures each one catches.
2. Register an Avro schema for purchase lines and set a compatibility level on that subject.
3. Explain backward, forward, and full compatibility in terms of who upgrades first, including why a default changes the answer.
4. Describe the Confluent wire format: magic byte, schema id, payload. Distinguish it from Connect's JSON envelope.
5. Choose a subject name strategy for one event type per topic, and say when a record-name strategy is the alternative.
6. Evolve currency as an optional field with a default, and reject a rename that is not a new field plus a migration.
7. Keep learning.progress on a JSON Schema contract without pretending the Spark job queries the registry.
8. State what the registry does not check: grain, money semantics, prohibited columns, and late data.
9. Plan a breaking change as a new subject or a new topic, with a dual-write window you can name.
10. Point at lesson 69 for organization-wide contract governance without skipping the local rule this topic needs now.

Prerequisites

• Data Formats and Serialization: Avro, JSON, and Parquet already differ. This lesson puts Avro and JSON Schema on a topic.
• Lesson 42: the purchase topic is plain JSON with schemas.enable=false. You are changing that decision on purpose.
• Data Quality Engineering: prohibited fields and the uniqueness of (order_id, line_id) remain checks. A schema does not replace them.
• Kafka Producers and Consumers: serializers sit in the client. The registry is a dependency of the serializer, not of the broker.
• /learn/kafka if the words "key" and "value" are still fuzzy. The registry stores a schema for each, under different subjects.

Detailed Core Concepts

Schema, subject, version, id

A schema is a definition of fields and types. Avro writes it as a JSON document with a record name. JSON Schema writes it as a JSON Schema document. Protobuf writes it as a .proto file. The payload on the topic is not required to carry that document every time. Something has to, or the reader is guessing.

A subject is the registry's name for a stream of versions. The usual default, topic-name strategy, uses commerce.purchase_line-value for the value and commerce.purchase_line-key for the key. A version is one schema document in that subject, numbered by the registry. A schema id is a cluster-wide integer. The id is what the Confluent wire format puts on each record. Versions are per subject. Ids are not. Two subjects never share a version number sequence, and they might still receive different ids for similar documents.

The registry answers three operational questions. What is the schema for this id? Is this new document allowed under the subject's compatibility setting? What is the latest version a producer should use? It does not move bytes through the broker. Producers register or look up a schema, cache the id, and serialize. Consumers read the id and fetch the schema, then cache it. The cache is why a registry outage does not instantly stop a pipeline that is still using schemas it has already seen, and why a brand-new schema cannot be written until the registry is reachable.

Compatibility is a rule about readers and writers

Confluent's levels, which Apicurio also implements for the formats it supports, are the ones you will be asked about. They follow Avro's resolution rules. State them as who can read whom.

Backward compatibility means a consumer using the new schema can read data written with the previous schema. You may add a field that has a default, because old bytes lack the field and the new reader fills the default. You may remove a field, because the new reader ignores it. You upgrade consumers before producers. Old data is still readable. This is the level to set when the warehouse and the sink must keep reading an hour that was written yesterday, and producers will roll later.

Forward compatibility means a consumer using the old schema can read data written with the new schema. You may add a field, because the old reader ignores writer fields it does not know. You may remove a field only if the old reader still has a default for it. You upgrade producers before consumers. Use this when you cannot find every consumer, which is another way of saying you will have old readers for a while. It is a weaker operational story for a lake sink you do control: you would rather upgrade the sink first.

Full compatibility means both directions. You add or remove a field only when a default makes both the old and the new reader safe. This is the level for a topic with several consumers you do not deploy in lockstep, once you have the discipline to put defaults on every new field.

Transitive variants (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE) check the new schema against all previous versions, not only the latest. Without transitive checks, a field can be removed and re-added with a different type across two hops, and each hop looked legal against its neighbor. Purchase lines should use BACKWARD_TRANSITIVE or FULL_TRANSITIVE. The history is the product. A non-transitive check protects only the last edit.

NONE disables the check. It belongs on a scratch subject in a lab. It does not belong on commerce.purchase_line-value.

These rules are structural. They do not notice that you changed the unit of unit_price_cents from cents to dollars while keeping the type long. They do not notice that sale and refund swapped meanings. Compatibility is necessary and not sufficient. The contract carries the meaning.

Defaults are the mechanism, not a courtesy

In Avro, the reader's schema is what the application uses, and the writer's schema is what produced the bytes. Resolution, not wishful parsing, fills the gap.

Add currency as a string with default "USD":

• A new reader and an old writer (no currency in the bytes): the reader applies "USD". That is backward compatible.
• An old reader and a new writer (currency present): the old reader skips the field. That is forward compatible.
• Both, so the change is full compatible, because the default exists for the old bytes and unknown fields are skipped by old readers.

Add currency without a default:

• New readers fail on old bytes, or rather, Avro resolution fails because the reader's field has no default and the writer's record omitted it. Not backward compatible.
• Old readers can still skip the new field. Forward compatible only.

That is why "we only added an optional column" is not a compatibility statement until the default is in the schema. JSON Schema in a registry is checked by the registry's JSON Schema rules, which are not Avro resolution. A JSON Schema change can be accepted by the registry and still be parsed differently by Spark from_json, which uses the schema you passed into the job, not the subject's latest version. If the progress topic stays on JSON, the job's schema and the registry's schema are two documents. The contract says they move together.

Wire format is not Connect's envelope

The Confluent serializer writes, for Avro:

1. One magic byte, 0x00.
2. A four-byte schema id, big-endian.
3. The Avro binary payload, without the schema document embedded.

Protobuf uses the same prefix and then message indexes before the bytes. JSON Schema serializers from the same project use the same prefix and a JSON payload. A consumer that is not using the matching deserializer will see binary and fail. A JsonConverter with schemas.enable=false will not read this. Lesson 42's purchase topic cannot be switched by flipping a worker default. You migrate it.

Connect's JSON envelope (schema plus payload) is a different invention. It does not contain a registry id. Mixing the two on one topic splits the consumers into the ones that happen to guess right.

Apache Kafka's own producers will happily write these bytes because they are bytes. The broker does not validate the magic byte. Validation happens in the client that was configured with the registry URL, or it does not happen.

Subject names decide how many event types share a check

TopicNameStrategy (the usual default) registers topic-key and topic-value. One value schema stream per topic. Purchase lines are one event type. This strategy matches them. A compatibility failure blocks every producer of that topic, which is what you want when there is one shape.

RecordNameStrategy uses the fully qualified Avro or Protobuf name as the subject. The same record can be produced to several topics and share a schema. It also means a topic can carry several record names, and the registry will not treat "whatever is on this topic" as one schema. Use it when the record, not the topic, is the unit of governance, and be ready for consumers that must branch on type.

TopicRecordNameStrategy combines them: the subject includes the topic and the record name. Several event types can share a topic and still evolve separately. The cost is a consumer that demultiplexes. For TechLearningPro, sale and refund are two values of line_type on one record, not two schemas. Keep one subject. If a later "payment attempted" event has a different grain, give it another topic rather than a second record on commerce.purchase_line.

A contract is the schema plus the decisions the schema cannot store

Write this next to the subject. It can live in the repository that reviews schema changes. It is not a registry endpoint.

The registry enforces the compatibility row if you configured it. The prohibited row is a review rule and a lesson 17 check. The money row is a review rule. A producer that passes the registry and ships dollars in a cents field has met every structural test you automated.

Lesson 69 is where contracts become an estate practice: ownership across domains, review, and how a break is scheduled. This lesson's contract is local so the purchase topic is not waiting on that program to stop guessing currency.

What evolution is allowed to look like

Allowed without a new topic, under full compatibility: add a field with a default; remove a field that had a default, knowing you will not read it anymore; add an enum symbol only if your Avro version and your compatibility setting agree it is safe. Adding an enum symbol is a common surprise. In classic Avro, a new symbol is not readable by an old consumer that has the old enum. Treat a new line_type symbol as a breaking change unless you have tested the exact registry rule and every consumer. Prefer a string with a documented set, validated in the quality gate, if the set will grow. The purchase schema below uses an enum anyway, with the rule that new symbols require a new version review and a consumer-first rollout, and that FULL may reject the add. If the registry rejects it, that rejection is the system working.

Not allowed in place: renaming unit_price_cents to amount_cents, changing long to double, changing the default's meaning, or reusing a field name for a new concept. Avro renames need aliases and a migration. The safe pattern in a pipeline is a new field, producers write both, consumers cut over, and the old field stays until no reader uses it. Removing it is a later, reviewed version.

A break you cannot avoid is a new subject, often a new topic (commerce.purchase_line.v2), with a dual-write or a replay window. The old topic stays until the morning job has cut over. Sharing one topic and flipping NONE for an afternoon is how the promotion-hour incident happened.

Architecture and Design Diagrams

mermaidCopy
flowchart LR
    subgraph clients [Clients]
        prod[Commerce producer]
        connect[Connect JDBC source]
        spark[Morning Spark job]
    end

    subgraph reg [Schema Registry]
        subj[subject commerce.purchase_line-value]
        cfg[FULL_TRANSITIVE]
        store[(schema versions and ids)]
    end

    subgraph kafka [Kafka]
        topic[commerce.purchase_line bytes]
    end

    prod -->|register or look up schema| subj
    connect -->|Avro converter uses the same subject| subj
    subj --> cfg --> store
    prod -->|magic byte plus schema id plus payload| topic
    connect --> topic
    spark -->|fetch writer schema by id, apply reader schema| store
    topic --> spark

The broker stores bytes. The registry stores documents and ids. Spark is shown calling the registry only if you added a deserializer that does so. Stock from_json on the old topic does not.

mermaidCopy
sequenceDiagram
    participant Owner as Schema owner
    participant Reg as Registry
    participant Prod as Producer
    participant Topic as Topic
    participant Job as Morning job

    Owner->>Reg: PUT compatibility FULL_TRANSITIVE
    Owner->>Reg: POST currency field with default USD
    Reg-->>Owner: New version, or 409 if the rule fails
    Job->>Reg: Reader schema is the new version
    Note over Job: Deploy the job before producers
    Prod->>Reg: Look up id for the new schema
    Reg-->>Prod: schema id
    Prod->>Topic: 0x00, id, Avro payload
    Job->>Topic: Read bytes
    Job->>Reg: Resolve id if not cached
    Note over Job: Old records without currency resolve to USD

If the producer deploys first and the job's reader schema has no currency field, a forward-compatible add is still readable. The incident in the overview was not an extra field. It was a rename, which no compatibility mode treats as a synonym.

Technical Explanation

Registering a schema

The Schema Registry REST API is the registry's, not Kafka's. A register call looks like this, against a registry base URL you run:

httpCopy
POST /subjects/commerce.purchase_line-value/versions
Content-Type: application/vnd.schemaregistry.v1+json

{
  "schemaType": "AVRO",
  "schema": "{ \"type\": \"record\", \"name\": \"PurchaseLine\", \"namespace\": \"pro.techlearning.commerce\", \"fields\": [ ... ] }"
}

The schema field is a string containing the Avro document, not a nested object. A 409 means the compatibility check failed. Read the body. Do not retry in a loop. GET /subjects/commerce.purchase_line-value/versions/latest returns the current version. GET /schemas/ids/{id} returns the document for a wire id, which is the call a consumer cache is avoiding on the hot path.

Set compatibility on the subject, not only as a global default:

httpCopy
PUT /config/commerce.purchase_line-value
Content-Type: application/vnd.schemaregistry.v1+json

{"compatibility": "FULL_TRANSITIVE"}

A global BACKWARD with one subject left at the default is how a scratch producer lowers the bar for a financial topic. Set the subject explicitly and review a change that widens it.

Avro document for the line

jsonCopy
{
  "type": "record",
  "name": "PurchaseLine",
  "namespace": "pro.techlearning.commerce",
  "doc": "One insert-only purchase line. unit_price_cents is integer minor units.",
  "fields": [
    {"name": "line_seq", "type": "long"},
    {"name": "order_id", "type": "string"},
    {"name": "line_id", "type": "string"},
    {"name": "course_id", "type": "string"},
    {"name": "line_type", "type": {"type": "enum", "name": "LineType", "symbols": ["sale", "refund"]}},
    {"name": "unit_price_cents", "type": "long"},
    {"name": "quantity", "type": "int"},
    {"name": "event_time", "type": {"type": "long", "logicalType": "timestamp-millis"}},
    {"name": "event_id", "type": "string"}
  ]
}

The evolved field, as its own version after consumers are ready:

jsonCopy
{"name": "currency", "type": "string", "default": "USD"}

event_id is the producer idempotency key from the quality lesson's vocabulary. line_seq remains the JDBC cursor and may be absent if a future producer writes the topic without going through the database. If that path is real, line_seq needs a default or a union with null, and the contract must say which system assigns identity. Do not make line_seq optional just in case. Decide the producer.

Logical types are hints. timestamp-millis is still a long on the wire. A consumer that ignores logical types will print epoch milliseconds and look "wrong" while being bitwise correct. The morning job converts to a timestamp in UTC. It does not divide by 1000 in one place and not in another.

Key schema

The key is a record with order_id and line_id, registered under commerce.purchase_line-key, also FULL_TRANSITIVE. A null key is a contract breach. Compaction, if you later compact this topic, keeps one record per key and would collapse two lines that shared a null key into an arbitrary survivor. The topic does not need compaction while it is an insert log. The key still needs to be the business key so any compacted derived topic can be built without a repartition surprise.

JSON Schema for a topic that is already JSON

learning.progress is parsed by Spark with a declared schema. Forcing Avro this week changes the tile, the checkpoint, and every replay file. Register a JSON Schema for the subject learning.progress-value so the fields, required list, and types have a version, and keep producers on JSON with schemas.enable=false until you explicitly adopt the registry serializer. The registry can store that document and apply its JSON Schema compatibility rules. Spark will not pick the document up by itself.

The practical control is a review that updates two artifacts in one change: the registry document, and the schema literal in the streaming job and the batch job. A test reads one fixture through both. If they disagree, the change does not ship. That is clumsier than Avro resolution, and it matches the estate you have. Write the clumsiness in the contract so nobody "finishes the migration" by flipping the producer to Avro over a weekend.

A minimal required-field list for the progress contract, aligned with lesson 32: event_id, course_id, lesson_id, event_time, event_type. event_time is a string or a number, but only one of those, matching what from_json already parses. Changing that representation is a breaking change even if both are "timestamps" in conversation.

Connect and the registry

The JDBC source can switch from JsonConverter to io.confluent.connect.avro.AvroConverter with value.converter.schema.registry.url set to the registry. The converter registers the Connect schema it sees. That derived schema may not match the Avro document you wrote by hand: names, nullability, and timestamp representation often differ. For purchase lines, prefer a producer or a transform path you control over "let the JDBC converter invent the subject." If the source remains the only writer, take the Connect schema it registers on a lab subject, review it, and only then point the production subject at it. Do not let the first poll against production create version 1 unsupervised.

value.converter.schemas.enable does not apply to AvroConverter. Leaving the JSON flag in the config is harmless noise or a sign the converter class was not actually changed. Read one record's first byte. 0x00 means you are on the registry wire format. { means you are still on JSON.

Failure modes the registry will not page you for

A producer gets a 409 or a schema-parse error and retries the same document. That is a bad deploy, and the topic simply stops receiving new versions. Old consumers keep reading old data. The page belongs to the deploying team, not to lag.

A consumer meets an id it cannot fetch because the registry is down and the id is not cached. That consumer stalls. The topic does not. Lag grows on that group only.

A field passes compatibility and violates the money rule. The topic grows. The morning number is wrong. Lesson 17's non-negative check still runs, and it will not catch a unit change. The contract test that decodes a fixture and asserts unit_price_cents == 1999 for a known course is the check that catches it.

Practical Implementation or Design Walkthrough

Problem

Move commerce.purchase_line from unsupervised JSON to an Avro subject with FULL_TRANSITIVE compatibility. Add currency safely. Leave learning.progress on JSON with an explicit schema document.

Steps

1. Stand up Schema Registry against the same Kafka cluster the topics use. The registry stores its own log on a compacted topic (_schemas in the Confluent deployment). Back that topic up with the same replication rules as connect-offsets. Losing it loses the id map, and the bytes on commerce.purchase_line become unreadable Avro.
2. Write the Avro document from this lesson, without currency, into version 1. Set the subject to FULL_TRANSITIVE before any producer is pointed at it.
3. In a lab, produce one record with the registry serializer and consume it with the matching deserializer. Confirm the first byte and that unit_price_cents survives a round trip. Confirm a null key is rejected by the producer wrapper you own, because the registry will not reject it for you.
4. Add currency with default "USD". The POST should succeed. Produce a new record. Read an old record with the new reader schema and assert currency USD. Read the new record with the old reader schema and assert the line still parses and the extra field is absent in that reader's object.
5. Attempt the rename of unit_price_cents as a replacement field with no alias. The registry should reject it under full compatibility. If a test double accepts it, you are not talking to a registry with the level set. Do not "fix" the test by switching to NONE.
6. Attempt to add enum symbol chargeback. Record the registry's response for your exact version. If it is rejected, the rollout plan is a string field or a consumer-first process you have tested, not a Friday add.
7. Point the JDBC source at AvroConverter only after the lab subject matches the document. Keep errors.tolerance=none on that source, from lesson 42.
8. Update the morning job to deserialize Avro with the reader schema you deployed. Deduplicate on (order_id, line_id) as before. Do not sum the raw prefix from the JSON era and the Avro era in one pass without a version flag.
9. For learning.progress, check in a JSON Schema file with the required fields. Open a review checklist item: Spark schema literal and registry document updated together. Do not change the producer serializer in the same change.

Cutover of the existing JSON topic

Do not flip the production topic from JSON to Avro in place. Consumers will parse 0x00 as JSON and fail, or worse, a permissive parser will drop the hour. Use a new topic, commerce.purchase_line.avro, or stop writers, drain consumers, and migrate with a window in which you accept downtime. For a financial extract, a new topic and a dual-run of the sink until counts match on (order_id, line_id) is the cleaner cut. The JSON topic remains readable for the retention you already set. The contract names the cutover date and the deprecation date.

Testing you can automate

A schema fixture test does not need a broker. It needs the Avro library and the two schema documents. Assert resolution in both directions for the currency add, and assert that a negative unit_price_cents is still representable so you remember the quality gate, not the schema, forbids it. A second test hits a disposable registry and expects 409 on the rename. Pin org.apache.avro to the version your serializer uses. Avro 1.11 is a reasonable pin for new work; match whatever the registry serializer was tested with, because logical-type handling has moved between minor versions.

Enterprise Real-World Scenarios

The promotion hour and the rename

Commerce runs a two-hour promotion. The producer deploy that renames the money field starts at the top of the second hour. Half the lines are old JSON, half are a shape the job does not share, and the job's permissive mode writes null amounts. The compatible alternative, reviewed the week before, adds nothing in that hour: currency was already on the schema with default USD, consumers were deployed Tuesday, producers Wednesday. The rename never entered the subject. Chargebacks, which the enum could not accept safely, go to a separate commerce.adjustment topic with its own subject rather than a new symbol sneaked into LineType. Volume is the promotion spike, still well inside one JDBC task. The security constraint is unchanged: the schema has nowhere to put a card number, and review rejects a field named pan or email even though Avro would consider the add compatible if it had a default. Compatibility is the wrong tool for that rejection. The contract's prohibited list is the right one.

Progress events stay on a documented JSON schema

The tile's team wants the registry because the purchase topic has one. Adopting Avro on learning.progress in the same sprint would invalidate the checkpoint conversation and the raw JSON already in the lake. The estate registers JSON Schema, versions it when event_type gains a value, and updates the Spark schema in the same pull request. A new event_type is not an Avro enum, so the registry's JSON Schema compatibility may allow it while the job's parser drops unknown types into a quarantine column. The contract says unknown event_type values are quarantined, not discarded, which is lesson 17's severity applied to a stream. The registry did not do that quarantine. The job did, because the contract told it to. Freshness stays the tile's watermark problem, not a schema problem.

Best Practices

• Set compatibility on the subject. Prefer FULL_TRANSITIVE for topics with more than one consumer and a real history.
• Put defaults on new Avro fields in the same change that adds them. Review the default's business meaning. "USD" is a statement that old lines were US dollars, not a harmless filler. If some old lines were not, the default is a lie and the migration is a backfill.
• One event grain per topic and TopicNameStrategy, unless you have a written reason to demultiplex.
• Version-control the schema document. The registry is the runtime copy. Git is the review copy. They should match the latest version you intend.
• Treat enum growth as a breaking-change review until a test on your registry version says otherwise.
• Migrate JSON to Avro onto a new topic. Do not mix magic-byte records and bare JSON in one partition.
• Keep the registry's schema topic replicated and monitored. It is now part of the read path for anyone who does not have the id cached.
• Pair every structural change with the contract fields it does not cover: units, prohibited columns, event time, key, upgrade order.
• For Spark jobs on JSON, change the registry document and the from_json schema together and test a fixture through both.
• Let lesson 69 define the cross-team process. Do not wait for it to set FULL_TRANSITIVE on the purchase subject.

Common Mistakes and Anti-Patterns

Compatibility NONE on a shared subject. Every rename and type change lands. The registry becomes a folder. Set a level before the first production producer.

A default that is semantically wrong. currency defaulting to "USD" on a catalog that already sold in other currencies rewrites history as the new reader schema is applied. Old bytes did not store the currency. The default invents it. Backfill or choose a default the business accepts as unknown, and make the quality gate quarantine unknown if a report must not assume dollars.

In-place rename. Avro does not treat a new name as the old field. Readers see a missing field and a new field. Use a new field and a migration, or a new topic.

Assuming Spark reads the registry. The job keeps using the schema string from the jar. The registry says currency exists. The column is absent in the warehouse. Wire the job to the schema on purpose, or keep the dual-review checklist.

Mixing wire formats on one topic. Half the partition is JSON, half is Avro. One consumer's offset cannot move without failing, so it stalls, and a more tolerant consumer skips. Cut over on a new topic.

Registry as a quality gate. The schema allows any long for money and any string for course_id. Negative prices and unknown courses pass. Keep lesson 17's checks. The registry is the structural half.

Auto-register from every connector in production. The JDBC converter's first poll creates version 1 from a guessed schema, and you spend the quarter compatible with a mistake. Disable auto-registration on production clients (auto.register.schemas=false on the Confluent serializer) and register from the review pipeline.

Global compatibility only. A lab subject and the purchase subject share a default of BACKWARD, someone sets the global level to NONE to unblock a prototype, and the financial subject moves with it. Set the subject.

Storing prohibited data because the field is "temporary." A compatible add of email with default "" is still a privacy incident once a producer fills it. The contract's prohibited list is enforced in review and in the producer, not by Avro.

Troubleshooting and Design Review

Symptom: Producers fail with 409 from the registry. The topic's end offset is flat.

Likely cause: the new schema is incompatible with the level on the subject. The broker is fine. The client is refusing to write a shape the subject forbids.

Investigation: the response body, the subject's compatibility setting, and a diff of the last version against the candidate. Look for a removed field without a reader default, a type change, or a new enum symbol.

Corrective action: fix the schema or, if the change is intentionally breaking, stop and run the new-topic plan. Do not drop the level to NONE on the production subject to clear the alert.

Symptom: Consumers throw an Avro resolution exception on records written last week. New records are fine.

Likely cause: the reader schema added a field without a default, and the subject's level was not transitive or was bypassed by a direct register. Old bytes cannot fill the field.

Investigation: version history of the subject, and whether the deployed reader matches the document you think you registered.

Corrective action: restore a reader schema that has defaults for every field old writers omit. Replay is unnecessary if the bytes are still valid for the previous reader. Fix the producer only after the reader can tolerate both.

Symptom: The morning job still has no currency column after a successful register.

Likely cause: Spark is on from_json or from_avro with a schema literal that was not updated. The registry is not in that path.

Investigation: the job's schema string and one decoded record from a registry-aware consumer.

Corrective action: ship the job change. Backfill the column with the default only if the contract says the default is true of old rows.

Symptom: After a registry restore, consumers cannot decode, or they decode with the wrong fields.

Likely cause: the schema topic was restored to a cluster whose ids do not match the ids already written on the data topic, or the registry lost _schemas and started ids again from 1.

Investigation: GET /schemas/ids/{id} for an id taken from a recent record's header, compared with the document you expect.

Corrective action: restore the registry log and the data topic as a pair. Do not point old Avro bytes at a fresh empty registry. This is a backup requirement, like the Connect offset topic.

Symptom: Two teams added different version 4 documents and both tests passed locally.

Likely cause: each test ran a private registry. The production subject has one history.

Investigation: production version list, not the laptop.

Corrective action: a pipeline that registers the candidate against a copy of production history, or against production with a permission that can register and cannot set compatibility to NONE.

A design review asks: What is the subject and the compatibility level? Who upgrades first? What is the default, and is it true of historical bytes? Where is the key schema? Is auto-register off in production? How does the Spark job obtain the schema? What is prohibited even if Avro would allow it? What is the topic cutover if the next change is a break? Which team owns the schema topic's backups?

Interview Questions and Answers

Q1. What does a schema registry store, and what does the Kafka broker store?

The registry stores schema documents, subject versions, compatibility settings, and integer ids. The broker stores record bytes. In the Confluent wire format those bytes begin with a magic byte and a schema id so a reader can ask the registry for the writer schema. Apache Kafka does not require that format. A topic can be plain JSON with no registry at all, which is how learning.progress already works. I add a registry when more than one service must evolve the bytes without guessing.

Q2. What is the difference between backward, forward, and full compatibility?

Backward means a new reader can read old data, so I upgrade consumers first. Adding a field with a default and removing a field are the typical backward-safe edits. Forward means an old reader can read new data, so I can upgrade producers first. Adding a field is forward-safe because old readers ignore it, and removing a field requires the old reader to have a default. Full means both, which in practice means I add or remove fields only with defaults. I use the transitive forms so the check covers the whole history, not only the last version.

Q3. Why does a missing Avro default break one direction and not the other?

If the reader schema has a field the writer omitted, Avro uses the reader's default. No default, no legal way to fill the field, so a new reader cannot read old bytes. That fails backward compatibility. An old reader that does not know about a new writer field skips it, which does not need a default on the new field. I still put the default on every add so the change is full compatible and so the default is reviewed as business meaning, not as boilerplate.

Q4. How does the Confluent Avro wire format differ from Connect's JSON envelope?

The Avro serializer writes 0x00, then a four-byte schema id, then Avro binary. Connect's JsonConverter with schemas.enable=true writes a JSON object containing a schema and a payload, and there is no registry id. schemas.enable=false writes bare JSON. I do not point an Avro consumer at a JSON topic or the reverse. A migration is a new topic or a stopped cutover, plus a dual-run of counts on the business key.

Q5. When would you choose something other than topic-name strategy?

The default subject topic-value is right when the topic has one event type, which is the purchase line. Record-name strategy fits when the schema is governed as a type reused across topics. Topic-record-name strategy fits when several event types share a topic and must evolve separately. Sharing a topic means every consumer demultiplexes. I would rather give a different grain its own topic than adopt a strategy to avoid a conversation about ownership.

Q6. A producer wants to rename unit_price_cents to amount_cents. What do you allow?

Not an in-place replacement under full compatibility. The new name is a new field, the old name is missing, and readers that still ask for unit_price_cents break or fill a default that is not the renamed value. I add amount_cents only if it is a genuinely new concept, or I keep the old name. If the name must die, I use a new topic, dual-write or replay, and cut the morning job over when the keys match. I do not set compatibility to NONE to permit the rename.

Q7. Why is a compatible schema not an event contract?

The schema fixes structure and, with a level set, the ways structure may change. It does not state the grain, the event-time field, the idempotency key, the currency unit, the prohibited columns, or who upgrades first. A long named unit_price_cents can carry dollars and stay compatible. I keep a short contract beside the subject and I keep lesson 17's checks for uniqueness, sign, and prohibited fields. Lesson 69 is where that local contract becomes an estate process. The purchase topic does not wait for the estate process to write down cents.

Q8. How do you evolve learning.progress while Spark still uses from_json?

I register a JSON Schema subject so the document is versioned, and I change the Spark schema literal in the same reviewed change. I test one fixture through the document and through from_json. I do not claim the registry is on the Spark read path. Stock from_avro also takes a schema the job supplies. A platform-specific registry integration has to be written down from that platform's docs before anyone depends on it. Until then, JSON Schema plus a paired review is the control, and an Avro migration is a planned topic cutover.

Q9. What does auto.register.schemas=false protect you from?

From a producer or a JDBC Avro converter creating version 1, or version 12, from whatever shape it happened to build on first contact with production. Registration belongs in the change that reviewed the document. The client looks up the schema and fails if it was not registered. That failure is louder and earlier than a week of data compatible with a guessed nullability.

Q10. How should enum evolution be treated on line_type?

As a suspected break. Old Avro readers fail when they meet a symbol that is not in their enum, and registry implementations differ on whether adding a symbol is compatible. I test the add against the registry version we run. If it is rejected, I keep the new business event on its own topic or I move the field to a string validated by the quality gate. I do not discover the answer during the promotion deploy.

Q11. What do you monitor once schemas are on the critical path?

Registry availability, because new ids cannot be resolved or registered without it. The schema topic's replication and disk. A sudden rise in producer errors on schema rejection, which is a bad deploy, not consumer lag. I do not page on "number of versions" by itself. I do page if production clients regain the ability to set compatibility to NONE or to auto-register, which is a permission problem.

Hands-On Exercises

Exercise: Evolve a purchase-line schema without a rename

Problem statement. Register PurchaseLine, lock compatibility, add currency safely, and show one breaking change being refused.

Tasks.

1. Write the Avro document and a one-page contract: key, event time, money unit, prohibited fields, compatibility, upgrade order.
2. Start a local registry or use the HTTP API against a lab instance. Create the subject at FULL_TRANSITIVE before version 1.
3. Register version 1. Produce and consume one record with a registry serializer. Record the schema id and the first byte.
4. Register version 2 with currency defaulting to "USD". Resolve an old payload with the new schema and a new payload with the old schema. Write down both results.
5. Submit a schema that replaces unit_price_cents with amount_cents. Capture the 409 or the equivalent rejection. If you get a 200, your compatibility level is not what you think it is. Fix the level and repeat.
6. Add a JSON Schema file for learning.progress with the required fields from lesson 32. Do not change the Spark job's wire format. Describe the paired review you would require for a new field.
7. State the cutover plan from the existing JSON topic to Avro, including how you will compare (order_id, line_id) counts.

Expected deliverables. The two Avro documents, the contract table, the registry responses, the resolution notes, and the cutover paragraph.

Evaluation rubric.

You pass when the 409 is real, the default is justified in a sentence, and the progress topic's plan does not claim a serializer the Spark job does not use.

Lesson Summary

A registry gives each value schema a subject, a version, and an id, and it can reject a new version that breaks the compatibility level. Backward, forward, and full describe which side upgrades first. Defaults are what make an added Avro field readable in both directions, and a default is a business claim about old bytes. The Confluent wire format is a magic byte and an id, not Connect's JSON envelope, and Apache Kafka will not enforce it for you. FULL_TRANSITIVE on commerce.purchase_line-value, registration from review with auto-register off, and a contract for money, keys, and prohibited fields stop the rename incident. They do not replace the quality gate. learning.progress keeps JSON until a planned cutover, with the Spark schema and the JSON Schema document moved together. Organization-wide contract governance continues in lesson 69.

Kafka Streams Fundamentals is next. The purchase events will be aggregated by course, and the schema you just froze is the one the topology is allowed to depend on.

Further Reading

• Confluent Schema Registry overview — subjects, versions, ids, and the REST API used above. This is not an Apache Kafka component.
• Schema evolution and compatibility — backward, forward, full, and transitive checks.
• Serdes and wire format — magic byte, schema id, and serializer flags such as auto.register.schemas.
• Apache Avro specification — resolution, defaults, enums, and logical types.
• JSON Schema — the language of the progress document. Registry enforcement of JSON Schema is still the registry's implementation, not Spark's from_json.
• Apicurio Registry — an alternative registry that can store the same kinds of artifacts. Confirm the API on the version you run before copying a Confluent URL.
• Data Contracts and Schema Governance — lesson 69, when the local purchase contract has to scale across domains.
• /learn/kafka — bytes, keys, and retention, which the registry does not replace.