Skip to content

chore: the Kinesis and DynamoDB Streams sources can limit a poll, the CloudWatch sink can take the metric as options, the CloudTrail source can look back for late events (#3170) - #3233

Merged
davsclaus merged 2 commits into
apache:mainfrom
davsclaus:ci-issue-3170-aws-streams-data
Nov 18, 2026

Conversation

@davsclaus

@davsclaus davsclaus commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Part of #3170: the Kinesis and DynamoDB Streams sources can read fewer records in one poll, and the Kinesis source can read one shard; the CloudWatch sink can take the metric name, value and unit as options instead of headers; the CloudTrail source can tune how it looks back for events that CloudTrail made visible late.

The options

Kamelet Option Template mapping What it does
aws-kinesis-source maxResultsPerRequest maxResultsPerRequest: "{{?maxResultsPerRequest}}" The most records read from each shard in one poll (1000 when unset).
aws-kinesis-source shardId shardId: "{{?shardId}}" Reads one shard only, such as shardId-000000000001. pattern: ^shardId-[0-9]+$.
aws-kinesis-source shardClosed shardClosed: "{{?shardClosed}}" ignore (component default), silent or fail, once a shard closed by resharding has been read to its end.
aws-kinesis-source applicationName applicationName: "{{?applicationName}}" The KCL application, which names the DynamoDB table of the KCL (the stream name when unset). Only with useKclConsumers. pattern: ^[A-Za-z0-9_.-]{3,255}$, the rules of a DynamoDB table name.
aws-cloudwatch-sink metricName name: "RAW({{?metricName}})" The metric name for the messages without a metric-name header. pattern: ^[ -~]{1,255}$.
aws-cloudwatch-sink metricValue value: "{{?metricValue}}" The value for the messages without metric-value (1 when neither is set).
aws-cloudwatch-sink metricUnit unit: "{{?metricUnit}}" The unit for the messages without metric-unit (Count when neither is set); an enum of the 27 CloudWatch units.
aws-cloudtrail-source lookBackWindow lookBackWindow: "{{?lookBackWindow}}" How far back, in ms, it looks up the events again (600000 when unset); 0 turns it off.
aws-cloudtrail-source lookBackInterval lookBackInterval: "{{?lookBackInterval}}" How often, in ms, it does so (60000 when unset).
aws-ddb-streams-source maxResultsPerRequest maxResultsPerRequest: "{{?maxResultsPerRequest}}" The most changes read in one poll, shared between the shards (100 when unset).
  • Defaults: all the options are optional and have no default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs are byte-identical to main's.
  • Option names: the CloudWatch options are metricName, metricValue and metricUnit, as name, value and unit say too little next to cwNamespace. None of the names is one the kamelet: component takes (chore: Kamelet options must not use a name the kamelet: component takes, such as location #3201).
  • A header wins over the option in the CloudWatch sink, as the component does: the template sets CamelAwsCwMetricName and the others only from the headers, and the component falls back on the endpoint options.
  • RAW() on metricName: without it, Orders Processed+1% arrived in CloudWatch as Orders Processed 1%. With it, Orders Processed+1% (eu) is kept as written.
  • lookBackWindow and lookBackInterval need Camel 4.24: they are not in the released camel-aws-cloudtrail 4.22.0 and 4.23.0 jars. The descriptions say "Needs Camel 4.24 or later".
  • sequenceNumber and messageTimestamp of aws-kinesis-source came with chore: aws-kinesis-source can choose where it starts reading (#3170) #3172; this PR tests them again, below.
  • Not added:
    • aws-ddb-sink consistentRead: the component uses it only for GetItem, Query, Scan and ExecuteStatement. The JSON transformation of the sink supports PutItem, UpdateItem and DeleteItem only: a message with CamelAwsDdbOperation: GetItem fails with Unsupported operation 'GetItem'. Read operations would also need an output that a sink in a Pipe does not have, so neither the read operations nor consistentRead are added.
  • Descriptions:
    • aws-kinesis-source "Good to know": it reads all the shards, up to 1000 records from each in one poll; it goes on with the new shards after a reshard, also with shardClosed: fail, where every poll then fails with an error in the log; the KCL table is named after applicationName, or after the stream.
    • shardClosed: fail is documented as tested: once a shard is closed, every later poll fails and logs an error with a stack trace, but the records of the other shards are still read.
    • aws-cloudwatch-sink: the summary, Input and "Good to know" say the options are the fallback for the headers, and that an unknown unit fails the message.
    • aws-cloudtrail-source "Good to know" was wrong: it said the Kamelet also reads the events from before it starts, and reads them again after a restart. Since Camel 4.22 (doStart sets the cursor to the start time; checked in the 4.22.0 and 4.23.0 jars and in a run) it reads only the events after it starts, and the events that happen while it is stopped are not read. It now says so, and explains the late-event lookup.
    • aws-ddb-streams-source "Good to know": up to 100 changes in one poll.
  • Partial pages: new sections "One shard, and smaller polls" and applicationName in the KCL example (Kinesis), "The Same Metric for Every Message" (CloudWatch), "Events That Become Visible Late" (CloudTrail), "Fewer Changes in One Poll" (DynamoDB Streams).

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT against floci/floci 2.5.0, which has Kinesis (with SplitShard), CloudWatch, CloudTrail, DynamoDB and DynamoDB Streams. Floci's CloudTrail returns no events, so the look-back cases ran against a small fake LookupEvents server that logs each request and can add an event with an older event time. Each case ran with camel run --local-kamelet-dir, and camel get endpoint --json gave the endpoint URI.

Options unset

Case Result
All five Kamelets (aws-ddb-sink too), the same parameters, main and this branch Byte-identical endpoint URIs

aws-kinesis-source, stream w6s with records r1…r5 (r1–r3, then a 4 s gap, then r4, r5), and stream w6split, whose shard 0 was split into shards 1 and 2 before six records were written

Case Result
Unset r1…r5
AT_SEQUENCE_NUMBER, the sequence number of r2 r2…r5
AFTER_SEQUENCE_NUMBER, the same r3…r5
AT_TIMESTAMP, a time in the gap r4, r5
delay: 3000 All five in the first poll
maxResultsPerRequest: 2, delay: 3000 r1,r2 at :29.1, r3,r4 at :32.1, r5 at :35.2: two per poll; the URI has maxResultsPerRequest=2
w6split, unset The six records of shards 1 and 2, then WARN The shard with id=shardId-000000000000 ... reached CLOSE status, once
shardId: shardId-000000000001 Only the four records of shard 1
shardId: shardId-000000000000 (the closed shard) Nothing, and the CLOSE warning
shardId: shardId-000000000009 Nothing; WARN Configured shardId ... not found in shard list on every poll
shardClosed: ignore, four records written 7 s after the start The six records, the CLOSE warning once, then the four new records
shardClosed: silent The six records, no warning
shardClosed: fail, four records written 7 s after the start The six records in the first poll; then every poll fails with ReachedClosedStatusException (29 times in 15 s), but the four new records are still read from the new shards
shardId of the closed shard + fail Every poll fails with ReachedClosedStatusException
useKclConsumers + applicationName: w6-orders-app, and a second integration without it (AWS_ENDPOINT_URL pointed the KCL's own DynamoDB client at Floci) DynamoDB tables w6-orders-app and w6s were created, holding the KCL's worker and leader entries. Floci's KCL support stops there: no shard leases, so no record was read in 150 s

aws-cloudwatch-sink, data points checked with aws cloudwatch list-metrics and get-metric-statistics

Case Result
Headers metric-name: Hdr1, metric-value: 5, metric-unit: Seconds, main and this branch Hdr1, 5, Seconds, as before
No headers; metricName: OrdersProcessed, metricValue: 42.5, metricUnit: Milliseconds OrdersProcessed, 42.5, Milliseconds; URI name=OrdersProcessed&unit=Milliseconds&value=42.5
Headers FromHeader/7/Bytes and options FromOption/99/Seconds FromHeader, 7, Bytes: the header wins
Only metricName: Ticks Ticks, 1, Count
metricName: Rate, metricValue: 2048, metricUnit: Bytes/Second Rate, 2048, Bytes/Second
metricName: "Orders Processed+1% (eu)" Kept as written (without RAW(): Orders Processed 1%)
No header and no option, main and this branch Fails: The parameter MetricData.member.1.MetricName is required
Header metric-unit: Parsecs Fails: Unknown unit Parsecs: use one of Seconds, ...

aws-cloudtrail-source, against the fake: an event OLD from one hour before the start; 20 s after the start the event NEW1, then 2 s later LATE, with an event time 5 s ago, so older than NEW1

Case Result
Unset, main and this branch NEW1 at once; LATE 60 s after the start, from a lookup with EndTime; OLD never
lookBackWindow: 0 NEW1 only; no lookup with EndTime
lookBackInterval: 5000 NEW1, then LATE 2 s after it was added
lookBackWindow: 3000, lookBackInterval: 5000 NEW1 only: LATE is older than the window
lookBackInterval: 0 Does not start: lookBackInterval must be above 0 unless lookBackWindow is 0, but was 0

aws-ddb-streams-source, table w6t with a stream; five items written in one batch-write-item 15 s after the start

Case Result
Unset, main and this branch Five INSERTs at once
delay: 3000 Five at once
maxResultsPerRequest: 2, delay: 3000 2, 2 and 1, 3 s apart; the URI has maxResultsPerRequest=2
maxResultsPerRequest: 1, delay: 3000 One every 3 s

aws-ddb-sink

Case Result
CamelAwsDdbOperation: GetItem Fails in the transformation: Unsupported operation 'GetItem'

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root. Before building, the library copies of the four Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regenerated kamelets-index.json.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

… CloudWatch sink can take the metric as options, the CloudTrail source can look back for late events (apache#3170)

New optional options, mapped with optional placeholders, so with them
unset the endpoint URIs are the same as before:

- aws-kinesis-source: maxResultsPerRequest (records per shard and poll),
  shardId (read one shard), shardClosed (ignore, silent or fail when a
  shard is closed by resharding) and applicationName (the KCL
  application, which names its DynamoDB lease table).
- aws-cloudwatch-sink: metricName, metricValue and metricUnit, mapped to
  the name, value and unit of the endpoint, for the messages without the
  metric-name, metric-value and metric-unit headers. metricName goes into
  RAW() so that a + or % in the name is kept.
- aws-cloudtrail-source: lookBackWindow and lookBackInterval (Camel 4.24),
  to read the events that CloudTrail makes visible after a newer one.
- aws-ddb-streams-source: maxResultsPerRequest (changes per poll).

The aws-cloudtrail-source description said it also reads the events from
before it starts, and reads them again after a restart. Since Camel 4.22
it starts at the time it starts, so the description now says so.

aws-ddb-sink does not get consistentRead: the option only applies to
GetItem, Query, Scan and ExecuteStatement, and the JSON transformation of
the Kamelet supports PutItem, UpdateItem and DeleteItem only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… shards (apache#3170)

The test log shows that the records of the new shards are still read;
each poll fails and logs an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit a1b06e2 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-aws-streams-data branch November 18, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant