Skip to content

chore: the file Kamelets can filter, wait for complete files, stream large files and set timeouts (#3170) - #3221

Merged
davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-files-options
Nov 18, 2026
Merged

davsclaus merged 1 commit into
apache:mainfrom
davsclaus:ci-issue-3170-files-options

Conversation

@davsclaus

Copy link
Copy Markdown
Contributor

Part of #3170: the file Kamelets can read only some files, wait until a file is completely written, stream large files, sort and limit what a poll reads, and set their timeouts. ftps-* can use implicit TLS, sftp-* can check that the server still answers, sftp-sink and scp-sink can set file permissions, scp-sink can write to a directory, and file-watch-source can filter files and stop watching subdirectories.

Earlier PRs already added move, moveFailed and idempotentRepository (#3178), tempPrefix and doneFileName (#3199), and the SSH key login (#3197). scp-sink already had a known hosts file (knownHostsUri, mapped to knownHostsFile), so it is not added again.

The options

ftp-source, ftps-source, sftp-source:

Option Template mapping What it does
include include: "RAW({{?include}})" A regular expression on the file name (not the directory), case-insensitive
exclude exclude: "RAW({{?exclude}})" A regular expression for the names not to read; wins over include and antInclude
antInclude antInclude: "RAW({{?antInclude}})" Comma-separated wildcards on the path below directoryName, case-sensitive
preMove preMove: "{{?preMove}}" Moves the file before it is read; move then acts from there (.inprogress/.done)
readLock readLock: "{{?readLock}}" none, changed or rename (enum: the only values the FTP and SFTP components act on)
readLockMinAge readLockMinAge: "{{?readLockMinAge}}" With changed: the minimum age of a file, in milliseconds
delay delay: "{{?delay}}" Milliseconds between two polls
maxMessagesPerPoll maxMessagesPerPoll: "{{?maxMessagesPerPoll}}" The most files read in one poll
sortBy sortBy: "RAW({{?sortBy}})" file:name, reverse:file:modified, groups separated by ;
streamDownload streamDownload: "{{?streamDownload}}" Reads the file while it is handed on, instead of into memory first
connectTimeout connectTimeout: "{{?connectTimeout}}" Milliseconds to wait for the connection (and, on FTP, the first answer)
socketTimeout soTimeout: "{{?socketTimeout}}" FTP: milliseconds to wait for the answer to a command. SFTP: how long the session may stay without an answer

ftp-source and ftps-source only:

Option Template mapping What it does
dataTimeout timeout: "{{?dataTimeout}}" Milliseconds to wait for data while a file or a listing is transferred
stepwise stepwise: "{{?stepwise}}" Must be false with streamDownload (see below)

Others:

Kamelet Option Template mapping
ftps-source, ftps-sink implicit implicit: "{{?implicit}}"
sftp-source, sftp-sink serverAliveInterval serverAliveInterval: "{{?serverAliveInterval}}"
sftp-sink chmod (pattern ^[0-7]{3}$) chmod: "{{?chmod}}"
scp-sink directoryName (pattern ^[A-Za-z0-9_./~@+=,-]*$) scp:{{serverName}}:{{serverPort}}/{{?directoryName}}?...
scp-sink chmod (pattern ^[0-7]{3}$) ...?chmod={{?chmod}}&...
file-watch-source antInclude antInclude: "RAW({{?antInclude}})"
file-watch-source recursive (default true) recursive: "{{recursive}}"
  • Defaults: every new option is an optional placeholder without a default, as in chore: mail-sink can use STARTTLS (#3170) #3174, so with the options unset the endpoint URIs are byte-identical to main's for ftp-source, ftps-source, sftp-source, ftps-sink and sftp-sink. Two exceptions, both with the same behaviour as before: file-watch-source gains recursive=true (the component's default, and what the task asked for), and the scp-sink URI path gains a trailing / (scp://host:2233/?...), which the component strips, so the files still land in the home directory.
  • Names: timeout is taken by the kamelet: component (chore: Kamelet options must not use a name the kamelet: component takes, such as location #3201), so the FTP data timeout is dataTimeout, and the socket timeout is socketTimeout for consistency.
  • RAW(): for the regular expressions, wildcards and sortBy. A + and a space in a regular expression reached the component as written (a+\.csv matched aaa.csv; a b\.csv|b\.csv matched both).
  • streamDownload needs stepwise false on FTP and FTPS: the FTP component throws The option stepwise is not supported for stream downloading for every file otherwise, so stepwise is added too and the descriptions say so. SFTP has no such restriction.
  • streamDownload alone does not save memory: Camel caches a stream body in memory at the start of the route, so a 400 MB file still ran out of a 256 MB heap. It works with camel.main.streamCachingSpoolEnabled=true (the body is spooled to a file) or with stream caching off. The descriptions and the partial pages say so.
  • readLockMinAge on FTP: FTP servers list the modification time to the minute and in their own time zone. A vsftpd on UTC seen from an integration on CET made a fresh file look one hour old, so it was read at once; with the integration on UTC it was read 28 s after it was written (the minute boundary). The descriptions say so; SFTP is exact. sortBy file:modified is also only to the minute on FTP.
  • maxMessagesPerPoll with sortBy: the limit is applied first, so a poll takes the first files the server lists and sorts only these. Said in both descriptions.
  • sftp-source template fix: its setBody: ${body.getBody()} failed with preMove (the body is then bytes: Method with name: getBody() not found on bean: [B) and with streamDownload (a cached stream). It now unwraps the body only when it is a GenericFile; without the new options the body is bytes as before.
  • scp-sink directoryName: it is not created. When it is missing, SCP writes each file as a file with the name of the directory, which the descriptions say. Subdirectories in CamelFileName are created, and a name that points outside the directory still fails.
  • scp-sink chmod: it applies to new files only, and the server umask applies: the default 664 gave 644, 666 gave 644, 640 gave 640; an existing file kept its 600. sftp-sink chmod is set after the upload, also on an existing file.
  • Left out: the SFTP proxy, which needs a com.jcraft.jsch.Proxy bean. connectTimeout and the socket and data timeouts were not added to the sinks, as the issue lists them for the sources.
  • Descriptions: each new option says what it does, its unit and what happens when it is not set; "Good to know" points to the filters and readLock; ftps-* no longer say implicit TLS is unsupported; scp-sink and file-watch-source say where files go and what is watched. Each partial page has new examples (filters and files still being written, preMove, large files, timeouts, implicit TLS, file permissions, a file pattern).

Verification

Tested with Camel JBang 4.24.0-SNAPSHOT:

  • FTP: delfer/alpine-ftp-server (vsftpd).
  • FTPS explicit: the same image with TLS (require_ssl_reuse=NO); FTPS implicit: the same with implicit_ssl=YES on port 990.
  • SFTP: atmoz/sftp. SCP: linuxserver/openssh-server.
  • Timeouts: a non-routable address for the connect timeout, docker pause of the server for the socket timeout and serverAliveInterval, and a small fake FTP server that accepts the data connection but never sends, for the data timeout.
  • file-watch-source: a local directory.
Case Result
Options unset, ftp-source, ftps-source, sftp-source, ftps-sink, sftp-sink, main and this branch camel get endpoint shows identical URIs
Unset, file-watch-source Branch adds recursive=true; same events as main (CREATE of top.csv, notes.txt, sub, sub/deep.csv, old/old.csv)
Unset, scp-sink Branch URI scp://127.0.0.1:2233/?..., main scp://127.0.0.1:2233?...; both write to the home directory
All options set, every Kamelet Each option is on the endpoint URI under its component name
include: .*\.csv, recursive, FTP / SFTP orders.csv, ORDERS2.CSV, sub/deep.csv; not notes.txt, data.xml, part.csv.tmp
exclude: .*\.(tmp|txt), FTP / SFTP orders.csv, ORDERS2.CSV, data.xml, sub/deep.csv
include: .*\.csv + exclude: orders2.*, FTP orders.csv only (ORDERS2.CSV excluded whatever the case)
antInclude: *.csv,*.xml, recursive, FTP / SFTP orders.csv, data.xml; not ORDERS2.CSV (case) nor sub/deep.csv
antInclude: **/*.csv, recursive, FTP / SFTP orders.csv, sub/deep.csv
antInclude: *.txt, FTPS explicit; include: .*\.csv, FTPS implicit b.txt; a.csv
include: a+\.csv + exclude: a b\.csv; include: a b\.csv|b\.csv aaa.csv; a b.csv and b.csv
preMove: .inprogress + move: .done, FTP / SFTP / FTPS Files read, then in .inprogress/.done/
preMove: .inprogress only, FTP Files read, then in .inprogress/
Five files, unset, FTP Read in the server's (alphabetical) order
sortBy: reverse:file:name, FTP e d c b a
sortBy: reverse:file:modified, files written 1 s apart over two minutes, FTP Newest minute first, alphabetical within a minute (b d e a c)
sortBy: reverse:file:name + maxMessagesPerPoll: 2 + delay: 3000, FTP Polls 3 s apart: b a, d c, e (the limit picks the first listed, then sorts)
sortBy: reverse:file:modified + maxMessagesPerPoll: 3, SFTP d a c, then b e (the same)
delay unset / 3000, idempotent: false, FTP The same file read every ~0.52 s / ~3.02 s
readLock: changed, FTP a.csv read ~6 s after start; the empty file never in 30 s
readLock: rename, FTP File read
readLock: changed + readLockMinAge: 60000, SFTP, fresh.csv and a 5-minute-old old.csv old.csv after the 20 s skip of fresh.csv; fresh.csv exactly 60 s after it was written
The same, FTP (server UTC, integration CET) Both read at once: the listed time is parsed one hour too old
The same, FTP, integration on UTC fresh.csv read 28 s after it was written (listed to the minute)
streamDownload: true, FTP, stepwise unset Every file fails: The option stepwise is not supported for stream downloading
streamDownload + stepwise: false + recursive + move, FTP Both files read and moved; the body is a stream
400 MB file, 256 MB heap, FTP: unset / stream / stream + spool / stream + caching off Out of memory / out of memory / read (FileInputStreamCache) / read (SocketInputStream)
400 MB file, 256 MB heap, SFTP: unset / stream / stream + spool Out of memory / out of memory / read
streamDownload, SFTP Body is a stream; without it, bytes as on main
Many options together, FTPS implicit (exclude, preMove, move, sortBy, maxMessagesPerPoll: 2, streamDownload, stepwise: false, readLock: changed) b a, d c, e, ~5 s apart each; x.tmp left; files in .inprogress/.done
connectTimeout: 2000 / unset, unroutable address, FTP Poll skipped after 10 s / 34 s (3 attempts)
The same, SFTP 10 s / 34 s
Silent server that never greets, FTP Each attempt times out after connectTimeout (Timed out waiting for initial connect reply), not socketTimeout
socketTimeout: 3000 / unset, FTP server paused Read timed out 3 s after the pause / nothing in 45 s
dataTimeout: 3000 / unset, data connection that never sends, FTP Read timed out 3.7 s / 30 s after LIST
socketTimeout: 3000 / serverAliveInterval: 2000 / unset, SFTP server paused Poll fails 6 s / 4.5 s after the pause / nothing in 40 s
serverAliveInterval: 2000 / unset, sftp-sink, two messages 25 s apart, server paused between them Second message: new connection, written / old connection reused, written
implicit: true, port 990, ftps-source / ftps-sink File read / up/orders.csv written
Implicit unset, port 990 Timed out waiting for initial connect reply
implicit: true, explicit server on port 21 Unsupported or unrecognized SSL message
chmod: 640 / unset / 644 over an existing 600 file, sftp-sink -rw-r----- / -rw-r--r-- / -rw-r--r--
scp-sink directoryName: incoming + chmod: 640 incoming/dir.csv, -rw-r-----
scp-sink directoryName: /tmp/abs /tmp/abs/abs.csv
scp-sink directoryName: missing (not there) A file named missing in the home directory, with the content
scp-sink directoryName: incoming, CamelFileName: sub/deep.csv incoming/sub/deep.csv, sub created
scp-sink CamelFileName: ../escape.csv Fails: Cannot write file with name: ../escape.csv; nothing written
scp-sink chmod: 666 / unset / 644 over an existing 600 file -rw-r--r-- (umask 022) / -rw-r--r-- / stays -rw-------
file-watch-source antInclude: *.csv top.csv only
file-watch-source antInclude: **/*.csv top.csv, sub/deep.csv, old/old.csv
file-watch-source recursive: false top.csv, notes.txt, sub; not sub/deep.csv, old/old.csv

./mvnw clean verify -DskipTests and then ./mvnw clean verify passed from the root. Before building, the library copies of the seven Kamelets were refreshed with ./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regenerated kamelets-index.json. KameletsCatalogTest needed no change (no headers were added).

Claude Code on behalf of Claus Ibsen

🤖 Generated with Claude Code

…large files and set timeouts (apache#3170)

ftp-source, ftps-source and sftp-source take include, exclude and
antInclude to read only some files; preMove; readLock (none, changed,
rename) and readLockMinAge to leave a file alone while it is still being
written; delay, maxMessagesPerPoll and sortBy; streamDownload; and
connectTimeout plus a socket timeout. The timeouts are named
socketTimeout (soTimeout) and, on FTP and FTPS, dataTimeout (timeout),
since timeout is taken by the kamelet: component. ftp-source and
ftps-source also take stepwise, because the FTP component refuses
streamDownload while stepwise is true.

ftps-source and ftps-sink take implicit, for servers that use implicit
TLS on port 990. sftp-source and sftp-sink take serverAliveInterval, and
sftp-sink takes chmod. scp-sink takes directoryName and chmod; it already
had knownHostsUri, mapped to knownHostsFile. file-watch-source takes
antInclude and recursive, which defaults to true as before.

All the new options are optional placeholders without a default, so the
endpoints are unchanged when they are not set, except file-watch-source,
whose URI now has recursive=true, and scp-sink, whose URI path now ends
with a slash; both behave as before.

sftp-source called getBody() on the body, which fails with preMove (the
body is then bytes) and with streamDownload (a cached stream); it now
unwraps the body only when it is a remote file.

The SFTP proxy is left out: it needs a bean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davsclaus
davsclaus force-pushed the ci-issue-3170-files-options branch from 6da54d7 to ccfa026 Compare November 18, 2026 20:01
@davsclaus
davsclaus merged commit 6826af9 into apache:main Nov 18, 2026
6 checks passed
@davsclaus
davsclaus deleted the ci-issue-3170-files-options branch November 18, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant