Skip to content

chore: kafka-source and kafka-sink can use a schema registry (#3183) - #3203

Merged
davsclaus merged 2 commits into
apache:mainfrom
davsclaus:ci-issue-3183
Nov 18, 2026
Merged

davsclaus merged 2 commits into
apache:mainfrom
davsclaus:ci-issue-3183

Conversation

@davsclaus

@davsclaus davsclaus commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Part of #3183 and #3144. kafka-source and kafka-sink can now read and write records through a schema registry. The vendor logic is in camel-kafka (CAMEL-27130, Camel 4.24), and the Kamelets only pass the options through.

New options on kafka-source and kafka-sink

Option Values Maps to
schemaRegistryVendor confluent, apicurio, aws-glue, azure (optional) schemaRegistryVendor
schemaRegistryUrl URL, or for azure the namespace name (optional) schemaRegistryURL
schemaFormat avro (default), json schemaFormat
schemaRegistryRegion AWS region (optional, aws-glue) additionalProperties.region
schemaRegistryName Glue registry name (optional, aws-glue) additionalProperties.registry.name
schemaGroup Azure schema group (sink only, required for azure) additionalProperties.schema.group
  • confluent also covers registries with the Confluent API, such as Redpanda and Karapace.
  • apicurio is Apicurio Registry 3.x.
  • aws-glue and azure take their credentials from the environment: the AWS default credentials chain, or the Azure DefaultAzureCredential.
  • Unset optional options are not passed to camel-kafka.
  • Without schemaRegistryVendor, nothing changes.

What the body becomes

  • kafka-source: the value read with its schema is turned into JSON text, so the next step gets text, as it does without a registry. An Avro record gives its JSON form, such as {"id": 7, "item": "camel"}, and a JSON Schema value goes through the json data format. Without this, the body was a GenericRecord or a Map, which json-serialize-action and the text sinks cannot use. That is why both Kamelets now depend on camel:jackson3.
  • kafka-sink, with json: JSON text is turned into an object first, so the record is written with an object schema instead of "string".
  • kafka-sink, with avro: a text body is written as an Avro string.

Tests

New Citrus test, kafka-schema-registry-route-test: timer-source sends JSON to kafka-sink, which writes it with a JSON Schema through the schema registry of Redpanda (Confluent serdes, --repos=confluent). kafka-source reads it back as JSON text, and the test checks that the registry has a JSON schema for the topic. All 4 KafkaIT tests pass locally.

Manual runs with Camel JBang 4.24.0-SNAPSHOT:

Case Result
No registry Unchanged, plain text
Confluent JSON, JSON text in {"id":1,"item":"camel"} out
Confluent Avro, text in Text out; the schema "string" is registered
Confluent Avro record written by another application {"id": 7, "item": "camel"} as String
Apicurio 3.0.8 Avro, text in Text out
azure with schemaGroup, no Azure account The serde is downloaded and used; the send fails at the Azure login (EnvironmentCredential authentication unavailable)
azure without schemaGroup The producer does not start: Schema group configuration property is required
aws-glue json with region, no AWS account The Glue serializer makes the schema; the send fails at the AWS login (Unable to load credentials)
aws-glue avro, text in Unsupported Type of Record received: Glue needs an Avro record, which the docs say

Azure and AWS Glue have no local emulator, so these runs check that the options reach the serde, not a full round trip.

Limits, stated in the docs:

  • Schema check: the sink registers a schema made from the message. With json that schema accepts any JSON object, and the sink does not check the message against a schema that is already registered.
  • Registry login: a confluent or apicurio registry that requires one is not supported yet, since camel-kafka has no options for it.
  • apicurio with json: it needs the schema registered first, and even then the producer fails on Camel JBang with an IncompatibleClassChangeError in com.networknt.schema. Camel pins json-schema-validator 3.0.9, while the Apicurio serde was built against an older version (CAMEL-27175). So the docs say to use avro with apicurio.

Not in this PR (follow-ups for #3183)

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

davsclaus and others added 2 commits November 18, 2026 21:21
…3183)

New schemaRegistryVendor (confluent, apicurio), schemaRegistryUrl and
schemaFormat (avro, json) options pass through to the camel-kafka options of
CAMEL-27130, which set the serde classes and registry properties of the vendor.

kafka-source turns the value read with its schema into JSON text (an Avro
record by its JSON form, a JSON Schema value with the json data format), so the
next step gets text as without a registry. kafka-sink with json turns JSON text
into an object, so the record is written with an object schema rather than a
string. Without schemaRegistryVendor nothing changes.

A Citrus test writes and reads a JSON Schema record through the schema
registry of Redpanda, with the Confluent serdes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ema registries (apache#3183)

schemaRegistryVendor also takes aws-glue and azure. New options:
schemaRegistryRegion and schemaRegistryName (AWS Glue, passed as the region
and registry.name serde properties) and, on kafka-sink, schemaGroup (Azure,
schema.group). Unset, they are not passed. schemaRegistryUrl also takes the
Event Hubs namespace without a scheme, as Azure expects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit 0f142d5 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3183 branch November 18, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants