Repository navigation
chore: kafka-source and kafka-sink can use a schema registry (#3183) - #3203
Merged
Merged
Conversation
This was referenced Nov 18, 2026
oscerd
approved these changes
Nov 18, 2026
…3183) New schemaRegistryVendor (confluent, apicurio), schemaRegistryUrl and schemaFormat (avro, json) options pass through to the camel-kafka options of CAMEL-27130, which set the serde classes and registry properties of the vendor. kafka-source turns the value read with its schema into JSON text (an Avro record by its JSON form, a JSON Schema value with the json data format), so the next step gets text as without a registry. kafka-sink with json turns JSON text into an object, so the record is written with an object schema rather than a string. Without schemaRegistryVendor nothing changes. A Citrus test writes and reads a JSON Schema record through the schema registry of Redpanda, with the Confluent serdes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ema registries (apache#3183) schemaRegistryVendor also takes aws-glue and azure. New options: schemaRegistryRegion and schemaRegistryName (AWS Glue, passed as the region and registry.name serde properties) and, on kafka-sink, schemaGroup (Azure, schema.group). Unset, they are not passed. schemaRegistryUrl also takes the Event Hubs namespace without a scheme, as Azure expects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
davsclaus
force-pushed
the
ci-issue-3183
branch
from
November 18, 2026 20:22
23d884c to
b3e8207
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3183 and #3144.
kafka-sourceandkafka-sinkcan now read and write records through a schema registry. The vendor logic is in camel-kafka (CAMEL-27130, Camel 4.24), and the Kamelets only pass the options through.New options on
kafka-sourceandkafka-sinkschemaRegistryVendorconfluent,apicurio,aws-glue,azure(optional)schemaRegistryVendorschemaRegistryUrlazurethe namespace name (optional)schemaRegistryURLschemaFormatavro(default),jsonschemaFormatschemaRegistryRegionaws-glue)additionalProperties.regionschemaRegistryNameaws-glue)additionalProperties.registry.nameschemaGroupazure)additionalProperties.schema.groupconfluentalso covers registries with the Confluent API, such as Redpanda and Karapace.apicuriois Apicurio Registry 3.x.aws-glueandazuretake their credentials from the environment: the AWS default credentials chain, or the AzureDefaultAzureCredential.schemaRegistryVendor, nothing changes.What the body becomes
{"id": 7, "item": "camel"}, and a JSON Schema value goes through the json data format. Without this, the body was aGenericRecordor aMap, whichjson-serialize-actionand the text sinks cannot use. That is why both Kamelets now depend oncamel:jackson3.json: JSON text is turned into an object first, so the record is written with an object schema instead of"string".avro: a text body is written as an Avro string.Tests
New Citrus test,
kafka-schema-registry-route-test:timer-sourcesends JSON tokafka-sink, which writes it with a JSON Schema through the schema registry of Redpanda (Confluent serdes,--repos=confluent).kafka-sourcereads it back as JSON text, and the test checks that the registry has a JSON schema for the topic. All 4 KafkaIT tests pass locally.Manual runs with Camel JBang 4.24.0-SNAPSHOT:
{"id":1,"item":"camel"}out"string"is registered{"id": 7, "item": "camel"}asStringazurewithschemaGroup, no Azure accountEnvironmentCredential authentication unavailable)azurewithoutschemaGroupSchema group configuration property is requiredaws-gluejsonwith region, no AWS accountUnable to load credentials)aws-glueavro, text inUnsupported Type of Record received: Glue needs an Avro record, which the docs sayAzure and AWS Glue have no local emulator, so these runs check that the options reach the serde, not a full round trip.
Limits, stated in the docs:
jsonthat schema accepts any JSON object, and the sink does not check the message against a schema that is already registered.confluentorapicurioregistry that requires one is not supported yet, since camel-kafka has no options for it.apicuriowithjson: it needs the schema registered first, and even then the producer fails on Camel JBang with anIncompatibleClassChangeErrorincom.networknt.schema. Camel pins json-schema-validator 3.0.9, while the Apicurio serde was built against an older version (CAMEL-27175). So the docs say to useavrowithapicurio.Not in this PR (follow-ups for #3183)
protobuf;kafka-batch-source;confluentandapicurio;Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code