Repository navigation
chore: the Kinesis and DynamoDB Streams sources can limit a poll, the CloudWatch sink can take the metric as options, the CloudTrail source can look back for late events (#3170) - #3233
Merged
Conversation
… CloudWatch sink can take the metric as options, the CloudTrail source can look back for late events (apache#3170) New optional options, mapped with optional placeholders, so with them unset the endpoint URIs are the same as before: - aws-kinesis-source: maxResultsPerRequest (records per shard and poll), shardId (read one shard), shardClosed (ignore, silent or fail when a shard is closed by resharding) and applicationName (the KCL application, which names its DynamoDB lease table). - aws-cloudwatch-sink: metricName, metricValue and metricUnit, mapped to the name, value and unit of the endpoint, for the messages without the metric-name, metric-value and metric-unit headers. metricName goes into RAW() so that a + or % in the name is kept. - aws-cloudtrail-source: lookBackWindow and lookBackInterval (Camel 4.24), to read the events that CloudTrail makes visible after a newer one. - aws-ddb-streams-source: maxResultsPerRequest (changes per poll). The aws-cloudtrail-source description said it also reads the events from before it starts, and reads them again after a restart. Since Camel 4.22 it starts at the time it starts, so the description now says so. aws-ddb-sink does not get consistentRead: the option only applies to GetItem, Query, Scan and ExecuteStatement, and the JSON transformation of the Kamelet supports PutItem, UpdateItem and DeleteItem only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… shards (apache#3170) The test log shows that the records of the new shards are still read; each poll fails and logs an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170: the Kinesis and DynamoDB Streams sources can read fewer records in one poll, and the Kinesis source can read one shard; the CloudWatch sink can take the metric name, value and unit as options instead of headers; the CloudTrail source can tune how it looks back for events that CloudTrail made visible late.
The options
aws-kinesis-sourcemaxResultsPerRequestmaxResultsPerRequest: "{{?maxResultsPerRequest}}"aws-kinesis-sourceshardIdshardId: "{{?shardId}}"shardId-000000000001.pattern: ^shardId-[0-9]+$.aws-kinesis-sourceshardClosedshardClosed: "{{?shardClosed}}"ignore(component default),silentorfail, once a shard closed by resharding has been read to its end.aws-kinesis-sourceapplicationNameapplicationName: "{{?applicationName}}"useKclConsumers.pattern: ^[A-Za-z0-9_.-]{3,255}$, the rules of a DynamoDB table name.aws-cloudwatch-sinkmetricNamename: "RAW({{?metricName}})"metric-nameheader.pattern: ^[ -~]{1,255}$.aws-cloudwatch-sinkmetricValuevalue: "{{?metricValue}}"metric-value(1 when neither is set).aws-cloudwatch-sinkmetricUnitunit: "{{?metricUnit}}"metric-unit(Countwhen neither is set); anenumof the 27 CloudWatch units.aws-cloudtrail-sourcelookBackWindowlookBackWindow: "{{?lookBackWindow}}"aws-cloudtrail-sourcelookBackIntervallookBackInterval: "{{?lookBackInterval}}"aws-ddb-streams-sourcemaxResultsPerRequestmaxResultsPerRequest: "{{?maxResultsPerRequest}}"default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs are byte-identical tomain's.metricName,metricValueandmetricUnit, asname,valueandunitsay too little next tocwNamespace. None of the names is one thekamelet:component takes (chore: Kamelet options must not use a name the kamelet: component takes, such as location #3201).CamelAwsCwMetricNameand the others only from the headers, and the component falls back on the endpoint options.RAW()onmetricName: without it,Orders Processed+1%arrived in CloudWatch asOrders Processed 1%. With it,Orders Processed+1% (eu)is kept as written.lookBackWindowandlookBackIntervalneed Camel 4.24: they are not in the releasedcamel-aws-cloudtrail4.22.0 and 4.23.0 jars. The descriptions say "Needs Camel 4.24 or later".sequenceNumberandmessageTimestampofaws-kinesis-sourcecame with chore: aws-kinesis-source can choose where it starts reading (#3170) #3172; this PR tests them again, below.aws-ddb-sinkconsistentRead: the component uses it only forGetItem,Query,ScanandExecuteStatement. The JSON transformation of the sink supportsPutItem,UpdateItemandDeleteItemonly: a message withCamelAwsDdbOperation: GetItemfails withUnsupported operation 'GetItem'. Read operations would also need an output that a sink in a Pipe does not have, so neither the read operations norconsistentReadare added.aws-kinesis-source"Good to know": it reads all the shards, up to 1000 records from each in one poll; it goes on with the new shards after a reshard, also withshardClosed: fail, where every poll then fails with an error in the log; the KCL table is named afterapplicationName, or after the stream.shardClosed: failis documented as tested: once a shard is closed, every later poll fails and logs an error with a stack trace, but the records of the other shards are still read.aws-cloudwatch-sink: the summary, Input and "Good to know" say the options are the fallback for the headers, and that an unknown unit fails the message.aws-cloudtrail-source"Good to know" was wrong: it said the Kamelet also reads the events from before it starts, and reads them again after a restart. Since Camel 4.22 (doStartsets the cursor to the start time; checked in the 4.22.0 and 4.23.0 jars and in a run) it reads only the events after it starts, and the events that happen while it is stopped are not read. It now says so, and explains the late-event lookup.aws-ddb-streams-source"Good to know": up to 100 changes in one poll.applicationNamein the KCL example (Kinesis), "The Same Metric for Every Message" (CloudWatch), "Events That Become Visible Late" (CloudTrail), "Fewer Changes in One Poll" (DynamoDB Streams).Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against
floci/floci2.5.0, which has Kinesis (withSplitShard), CloudWatch, CloudTrail, DynamoDB and DynamoDB Streams. Floci's CloudTrail returns no events, so the look-back cases ran against a small fakeLookupEventsserver that logs each request and can add an event with an older event time. Each case ran withcamel run --local-kamelet-dir, andcamel get endpoint --jsongave the endpoint URI.Options unset
aws-ddb-sinktoo), the same parameters,mainand this branchaws-kinesis-source, streamw6swith recordsr1…r5(r1–r3, then a 4 s gap, thenr4,r5), and streamw6split, whose shard 0 was split into shards 1 and 2 before six records were writtenr1…r5AT_SEQUENCE_NUMBER, the sequence number ofr2r2…r5AFTER_SEQUENCE_NUMBER, the samer3…r5AT_TIMESTAMP, a time in the gapr4,r5delay: 3000maxResultsPerRequest: 2,delay: 3000r1,r2at :29.1,r3,r4at :32.1,r5at :35.2: two per poll; the URI hasmaxResultsPerRequest=2w6split, unsetWARN The shard with id=shardId-000000000000 ... reached CLOSE status, onceshardId: shardId-000000000001shardId: shardId-000000000000(the closed shard)shardId: shardId-000000000009WARN Configured shardId ... not found in shard liston every pollshardClosed: ignore, four records written 7 s after the startshardClosed: silentshardClosed: fail, four records written 7 s after the startReachedClosedStatusException(29 times in 15 s), but the four new records are still read from the new shardsshardIdof the closed shard +failReachedClosedStatusExceptionuseKclConsumers+applicationName: w6-orders-app, and a second integration without it (AWS_ENDPOINT_URLpointed the KCL's own DynamoDB client at Floci)w6-orders-appandw6swere created, holding the KCL's worker and leader entries. Floci's KCL support stops there: no shard leases, so no record was read in 150 saws-cloudwatch-sink, data points checked withaws cloudwatch list-metricsandget-metric-statisticsmetric-name: Hdr1,metric-value: 5,metric-unit: Seconds,mainand this branchHdr1, 5,Seconds, as beforemetricName: OrdersProcessed,metricValue: 42.5,metricUnit: MillisecondsOrdersProcessed, 42.5,Milliseconds; URIname=OrdersProcessed&unit=Milliseconds&value=42.5FromHeader/7/Bytesand optionsFromOption/99/SecondsFromHeader, 7,Bytes: the header winsmetricName: TicksTicks, 1,CountmetricName: Rate,metricValue: 2048,metricUnit: Bytes/SecondRate, 2048,Bytes/SecondmetricName: "Orders Processed+1% (eu)"RAW():Orders Processed 1%)mainand this branchThe parameter MetricData.member.1.MetricName is requiredmetric-unit: ParsecsUnknown unit Parsecs: use one of Seconds, ...aws-cloudtrail-source, against the fake: an eventOLDfrom one hour before the start; 20 s after the start the eventNEW1, then 2 s laterLATE, with an event time 5 s ago, so older thanNEW1mainand this branchNEW1at once;LATE60 s after the start, from a lookup withEndTime;OLDneverlookBackWindow: 0NEW1only; no lookup withEndTimelookBackInterval: 5000NEW1, thenLATE2 s after it was addedlookBackWindow: 3000,lookBackInterval: 5000NEW1only:LATEis older than the windowlookBackInterval: 0lookBackInterval must be above 0 unless lookBackWindow is 0, but was 0aws-ddb-streams-source, tablew6twith a stream; five items written in onebatch-write-item15 s after the startmainand this branchINSERTs at oncedelay: 3000maxResultsPerRequest: 2,delay: 3000maxResultsPerRequest=2maxResultsPerRequest: 1,delay: 3000aws-ddb-sinkCamelAwsDdbOperation: GetItemUnsupported operation 'GetItem'./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root. Before building, the library copies of the four Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json.Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code