Skip to content

chore: the S3 sink can upload in parts and refuse to replace a file, the S3 event source can use long polling, the SES sink takes its recipients as options (#3170) - #3231

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-aws-s3-ses
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-aws-s3-ses

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: aws-s3-sink can upload large files in parts and refuse to replace an existing file, aws-s3-event-based-source can use long polling and set how many events it receives and for how long they stay hidden, and aws-ses-sink takes its recipients, subject, Cc, Bcc, return path and configuration set as options and can connect to a local stand-in of AWS.

The problem

  • aws-s3-sink uploaded every file in one request. S3 takes at most 5 GB in one upload, so larger files could not be stored. A file with an existing key was always replaced.
  • aws-s3-event-based-source asked SQS for events every 500 ms with requests that return at once, even on an empty queue, and had no say in the visibility timeout: an event that takes longer to process than the timeout of the queue (30 s by default), such as fetching a large file with getObject, was received a second time.
  • aws-ses-sink took the recipients, the subject and the reply-to addresses only from headers, so every message needed a to header. It had no Cc, Bcc, return path or configuration set, and could not be pointed at a local stand-in of AWS.

The options

aws-s3-sink: appended to the end of the aws2-s3 URI, after the proxy options.

Option Template mapping What it does
multiPartUpload &multiPartUpload={{?multiPartUpload}} A file of partSize bytes or more is uploaded as an S3 multipart upload, one request per part. Smaller files still go in one request.
partSize &partSize={{?partSize}} The size of each part, in bytes; also the size from which a file is uploaded in parts. 25 MB when not set. Only used with multiPartUpload.
conditionalWritesEnabled &conditionalWritesEnabled={{?conditionalWritesEnabled}} Sends If-None-Match: *: the upload of an existing key fails with PreconditionFailed (412), and the file is not changed. Also on the completion of a multipart upload.

aws-s3-event-based-source: added to the parameters of the aws2-sqs consumer.

Option Template mapping What it does
maxMessagesPerPoll maxMessagesPerPoll: "{{?maxMessagesPerPoll}}" The most events in one request, 1 to 10. 10 when not set.
waitTimeSeconds waitTimeSeconds: "{{?waitTimeSeconds}}" Long polling: a request waits up to this many seconds (0 to 20) for an event. The queue's setting when not set.
visibilityTimeout visibilityTimeout: "{{?visibilityTimeout}}" How long a received event stays hidden, in seconds. The queue's setting when not set.

aws-ses-sink: appended to the end of the aws2-ses URI, after the proxy options.

Option Template mapping What it does
to &to=RAW({{?to}}) The recipients of every email, comma-separated. A to header takes precedence; with the option set, the header is no longer required.
cc &cc=RAW({{?cc}}) Cc of every email.
bcc &bcc=RAW({{?bcc}}) Bcc of every email.
subject &subject=RAW({{?subject}}) The subject of every email. A subject header takes precedence.
replyToAddresses &replyToAddresses=RAW({{?replyToAddresses}}) The Reply-To addresses. A reply-to-addresses header takes precedence.
returnPath &returnPath=RAW({{?returnPath}}) Where bounces and complaints go. Must be verified in SES.
configurationSet &configurationSet={{?configurationSet}} The SES configuration set to send with. Pattern ^[A-Za-z0-9_-]+$.
overrideEndpoint &overrideEndpoint={{?overrideEndpoint}} Connect to uriEndpointOverride instead of SES.
uriEndpointOverride &uriEndpointOverride={{?uriEndpointOverride}} The address of the stand-in, such as http://localhost:4566.
  • Defaults: all new options are optional and have no default, as in chore: mail-sink can use STARTTLS (#3170) #3174. When they are unset, the optional placeholders drop them, and the endpoint URIs of the three Kamelets are byte-identical to main's.
  • conditionalWritesEnabled exists at 4.24.0-SNAPSHOT (AWS2S3Configuration). The component sends If-None-Match: * on PutObject and on CompleteMultipartUpload.
  • The subject of SES: on main, a message without a subject header got the exchange ID as its subject, in a header that would have hidden a subject option. That fallback is now skipped, with a precondition choice on ${propertiesExist:!subject} (as aws-s3-sink does for keyName), when the subject option is set. Without the option, the subject is still a generated ID.
  • RAW() in SES: the subject is free text, and addresses can hold a +. Without RAW(), the subject Order (#42) & more + 100% done made the route fail at start (URLDecoder: Illegal hex characters in escape (%) pattern), and a + in an address would turn into a space. With it, both are kept as written. The address options have a pattern for plain, comma-separated addresses (^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$; returnPath a single address), so a ; as separator is caught; a display name such as Jane <jane@example.com> is not allowed in the options.
  • Precedence: a to, subject or reply-to-addresses header takes precedence over the option, as the component does for its own headers. cc, bcc, returnPath and configurationSet have no Kamelet header.
  • Left out: trustAllCertificates of SES (it turns off TLS checks). Kamelet headers for Cc and Bcc.
  • Descriptions:
    • aws-s3-sink: "Good to know" says that conditionalWritesEnabled stops the replacement of a file, and points to multiPartUpload for files over 5 GB.
    • aws-s3-event-based-source: "Good to know" says that an event that fails is received again when its visibility timeout ends.
    • aws-ses-sink: the summary, "Input" and "Good to know" say that to can be an option, that the headers take precedence, that returnPath must be verified and a missing configuration set fails every email, and how to reach a local stand-in. The header descriptions say the same.
  • Partial pages: aws-s3-sink has an "Example with Multipart Uploads and Conditional Writes", aws-s3-event-based-source an "Example with Long Polling and a Longer Visibility Timeout", and aws-ses-sink an "Example with Fixed Recipients and a Configuration Set" and an "Example with a Local Stand-in of AWS". Its "Headers" section says which header is required and that the headers take precedence.
  • From the AWS documentation, not tested here: the 5 GB limit of one upload and the 10,000 parts per file; the 20 s maximum of waitTimeSeconds; that bounces go to returnPath; and what a configuration set publishes (delivery, bounce, open and click events to CloudWatch, SNS or Firehose).

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT against floci/floci 2.5.0 as S3, SQS and SES, with its HTTP access log turned on (QUARKUS_HTTP_ACCESS_LOG_ENABLED=true) as the request log. Floci keeps the emails it was given at /_aws/ses, checks that from and the return path are verified, and checks that a configuration set exists. The upload body was a 12 MiB (12,582,912 bytes) random file; partSize was 5 MiB (5242880). Endpoint URIs were read with camel get endpoint --json.

Unset options

Case Result
All three Kamelets with the new options unset, main vs this branch aws2-s3, aws2-ses and aws2-sqs endpoint URIs identical

aws-s3-sink

Case Result
multiPartUpload + partSize: 5242880, 12 MiB file body Request log: POST ?uploads, PUT ?partNumber=1, 2, 3, POST ?uploadId (200). head-object: 12,582,912 bytes, ETag "96eb…-3" (3 parts); the downloaded file has the MD5 of the source
The same with a byte[] body 3 parts, the same size, ETag and MD5
The same with a stream body 3 parts, ETag "96eb…-3"
multiPartUpload, partSize unset, 12 MiB One PUT (under the default 25 MB); MD5 matches
multiPartUpload + partSize: 5242880, 10-byte body One PUT
partSize: 5242880 without multiPartUpload, 12 MiB One PUT
multiPartUpload + partSize: 1048576, 12 MiB 12 parts sent, then the completion fails (400, Your proposed upload is smaller than the minimum allowed size), followed by DELETE ?uploadId (abort); list-multipart-uploads is empty
conditionalWritesEnabled, new key PUT 200, stored
conditionalWritesEnabled, the same key again PUT 412; the message fails with At least one of the pre-conditions you specified did not hold (Service: S3, Status Code: 412)
Unset, the same key again Replaced (PUT 200)
conditionalWritesEnabled + multiPartUpload, existing key 3 parts sent, completion 412, then DELETE ?uploadId (abort); the message fails
conditionalWritesEnabled + keyName, two messages First PUT 200, second PUT 412; the file keeps the body of the first message

aws-s3-event-based-source

Case Result
maxMessagesPerPoll: 2, delay: 4000, 6 events on the queue 2 events per poll, at +0 s, +4 s and +8 s
Unset, the same All 6 in the first poll
Unset, 12 events, delay: 4000 10 in the first poll, 2 four seconds later
waitTimeSeconds: 10, delay: 100, empty queue for 34 s, one event sent after 18 s 7 SQS requests in all; the event arrived within a second of being sent
Unset, the same 275 SQS requests
visibilityTimeout: 3, a route that fails every event The event is received again every 3 s (5 times in 14 s)
Unset (queue timeout 30 s), the same Received once in 14 s

aws-ses-sink (with overrideEndpoint and uriEndpointOverride pointing at Floci)

Case Result
All options: to: "user+tag@example.com, second@example.com", two cc, a bcc, subject: "Order (#42) & more + 100% done", replyToAddresses, returnPath, configurationSet: cs1 (existing) Floci's stored email has exactly those To (with the +), Cc, Bcc, Reply-To, subject and return path
configurationSet: does-not-exist The message fails: Configuration set <does-not-exist> does not exist (400)
Options to, subject, replyToAddresses together with to, subject, reply-to-addresses headers The email uses the header values
Options to and subject: "Nightly job (done)", no headers Sent to the option's address with that subject
No new options except the endpoint override, to header only The subject is a generated ID, as on main
Neither a to header nor the to option The message fails: At least one destination address is required
Subject mapped without RAW() (first attempt) The route fails at start: URLDecoder: Illegal hex characters in escape (%) pattern

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root. Before building, the library copies of the three Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regenerated kamelets-index.json.

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…the S3 event source can use long polling, the SES sink takes its recipients as options (apache#3170)

aws-s3-sink:
- multiPartUpload and partSize: a file of partSize bytes or more is
  uploaded as an S3 multipart upload, so files over the 5 GB limit of
  one upload can be stored.
- conditionalWritesEnabled: the upload of a key that exists already
  fails with PreconditionFailed (412) instead of replacing the file.

aws-s3-event-based-source: maxMessagesPerPoll, waitTimeSeconds (long
polling) and visibilityTimeout on its SQS consumer.

aws-ses-sink: to, cc, bcc, subject, replyToAddresses, returnPath and
configurationSet as options, and overrideEndpoint and
uriEndpointOverride to reach a local stand-in of AWS. The to, subject
and reply-to-addresses headers take precedence over the options; with
the to option set, the to header is no longer required. The fallback
of the subject to the exchange ID is skipped when the subject option
is set. The subject and the addresses go into the URI inside RAW(), so
that a +, a % or an & is kept as written.

All new options are optional and have no default: with them unset,
the endpoint URIs are the same as before. The descriptions and the
doc partials have examples of the new options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus merged commit 59ae6c1 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-aws-s3-ses branch November 18, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant