Repository navigation
chore: the file Kamelets can filter, wait for complete files, stream large files and set timeouts (#3170) - #3221
Merged
Conversation
…large files and set timeouts (apache#3170) ftp-source, ftps-source and sftp-source take include, exclude and antInclude to read only some files; preMove; readLock (none, changed, rename) and readLockMinAge to leave a file alone while it is still being written; delay, maxMessagesPerPoll and sortBy; streamDownload; and connectTimeout plus a socket timeout. The timeouts are named socketTimeout (soTimeout) and, on FTP and FTPS, dataTimeout (timeout), since timeout is taken by the kamelet: component. ftp-source and ftps-source also take stepwise, because the FTP component refuses streamDownload while stepwise is true. ftps-source and ftps-sink take implicit, for servers that use implicit TLS on port 990. sftp-source and sftp-sink take serverAliveInterval, and sftp-sink takes chmod. scp-sink takes directoryName and chmod; it already had knownHostsUri, mapped to knownHostsFile. file-watch-source takes antInclude and recursive, which defaults to true as before. All the new options are optional placeholders without a default, so the endpoints are unchanged when they are not set, except file-watch-source, whose URI now has recursive=true, and scp-sink, whose URI path now ends with a slash; both behave as before. sftp-source called getBody() on the body, which fails with preMove (the body is then bytes) and with streamDownload (a cached stream); it now unwraps the body only when it is a remote file. The SFTP proxy is left out: it needs a bean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
davsclaus
force-pushed
the
ci-issue-3170-files-options
branch
from
November 18, 2026 20:01
6da54d7 to
ccfa026
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3170: the file Kamelets can read only some files, wait until a file is completely written, stream large files, sort and limit what a poll reads, and set their timeouts.
ftps-*can use implicit TLS,sftp-*can check that the server still answers,sftp-sinkandscp-sinkcan set file permissions,scp-sinkcan write to a directory, andfile-watch-sourcecan filter files and stop watching subdirectories.Earlier PRs already added
move,moveFailedandidempotentRepository(#3178),tempPrefixanddoneFileName(#3199), and the SSH key login (#3197).scp-sinkalready had a known hosts file (knownHostsUri, mapped toknownHostsFile), so it is not added again.The options
ftp-source,ftps-source,sftp-source:includeinclude: "RAW({{?include}})"excludeexclude: "RAW({{?exclude}})"includeandantIncludeantIncludeantInclude: "RAW({{?antInclude}})"directoryName, case-sensitivepreMovepreMove: "{{?preMove}}"movethen acts from there (.inprogress/.done)readLockreadLock: "{{?readLock}}"none,changedorrename(enum: the only values the FTP and SFTP components act on)readLockMinAgereadLockMinAge: "{{?readLockMinAge}}"changed: the minimum age of a file, in millisecondsdelaydelay: "{{?delay}}"maxMessagesPerPollmaxMessagesPerPoll: "{{?maxMessagesPerPoll}}"sortBysortBy: "RAW({{?sortBy}})"file:name,reverse:file:modified, groups separated by;streamDownloadstreamDownload: "{{?streamDownload}}"connectTimeoutconnectTimeout: "{{?connectTimeout}}"socketTimeoutsoTimeout: "{{?socketTimeout}}"ftp-sourceandftps-sourceonly:dataTimeouttimeout: "{{?dataTimeout}}"stepwisestepwise: "{{?stepwise}}"streamDownload(see below)Others:
ftps-source,ftps-sinkimplicitimplicit: "{{?implicit}}"sftp-source,sftp-sinkserverAliveIntervalserverAliveInterval: "{{?serverAliveInterval}}"sftp-sinkchmod(pattern^[0-7]{3}$)chmod: "{{?chmod}}"scp-sinkdirectoryName(pattern^[A-Za-z0-9_./~@+=,-]*$)scp:{{serverName}}:{{serverPort}}/{{?directoryName}}?...scp-sinkchmod(pattern^[0-7]{3}$)...?chmod={{?chmod}}&...file-watch-sourceantIncludeantInclude: "RAW({{?antInclude}})"file-watch-sourcerecursive(defaulttrue)recursive: "{{recursive}}"default, as in chore: mail-sink can use STARTTLS (#3170) #3174, so with the options unset the endpoint URIs are byte-identical tomain's forftp-source,ftps-source,sftp-source,ftps-sinkandsftp-sink. Two exceptions, both with the same behaviour as before:file-watch-sourcegainsrecursive=true(the component's default, and what the task asked for), and thescp-sinkURI path gains a trailing/(scp://host:2233/?...), which the component strips, so the files still land in the home directory.timeoutis taken by thekamelet:component (chore: Kamelet options must not use a name the kamelet: component takes, such as location #3201), so the FTP data timeout isdataTimeout, and the socket timeout issocketTimeoutfor consistency.RAW(): for the regular expressions, wildcards andsortBy. A+and a space in a regular expression reached the component as written (a+\.csvmatchedaaa.csv;a b\.csv|b\.csvmatched both).streamDownloadneedsstepwisefalse on FTP and FTPS: the FTP component throwsThe option stepwise is not supported for stream downloadingfor every file otherwise, sostepwiseis added too and the descriptions say so. SFTP has no such restriction.streamDownloadalone does not save memory: Camel caches a stream body in memory at the start of the route, so a 400 MB file still ran out of a 256 MB heap. It works withcamel.main.streamCachingSpoolEnabled=true(the body is spooled to a file) or with stream caching off. The descriptions and the partial pages say so.readLockMinAgeon FTP: FTP servers list the modification time to the minute and in their own time zone. A vsftpd on UTC seen from an integration on CET made a fresh file look one hour old, so it was read at once; with the integration on UTC it was read 28 s after it was written (the minute boundary). The descriptions say so; SFTP is exact.sortByfile:modifiedis also only to the minute on FTP.maxMessagesPerPollwithsortBy: the limit is applied first, so a poll takes the first files the server lists and sorts only these. Said in both descriptions.sftp-sourcetemplate fix: itssetBody: ${body.getBody()}failed withpreMove(the body is then bytes:Method with name: getBody() not found on bean: [B) and withstreamDownload(a cached stream). It now unwraps the body only when it is aGenericFile; without the new options the body is bytes as before.scp-sinkdirectoryName: it is not created. When it is missing, SCP writes each file as a file with the name of the directory, which the descriptions say. Subdirectories inCamelFileNameare created, and a name that points outside the directory still fails.scp-sinkchmod: it applies to new files only, and the server umask applies: the default664gave644,666gave644,640gave640; an existing file kept its600.sftp-sinkchmodis set after the upload, also on an existing file.com.jcraft.jsch.Proxybean.connectTimeoutand the socket and data timeouts were not added to the sinks, as the issue lists them for the sources.readLock;ftps-*no longer say implicit TLS is unsupported;scp-sinkandfile-watch-sourcesay where files go and what is watched. Each partial page has new examples (filters and files still being written,preMove, large files, timeouts, implicit TLS, file permissions, a file pattern).Verification
Tested with Camel JBang 4.24.0-SNAPSHOT:
delfer/alpine-ftp-server(vsftpd).require_ssl_reuse=NO); FTPS implicit: the same withimplicit_ssl=YESon port 990.atmoz/sftp. SCP:linuxserver/openssh-server.docker pauseof the server for the socket timeout andserverAliveInterval, and a small fake FTP server that accepts the data connection but never sends, for the data timeout.file-watch-source: a local directory.ftp-source,ftps-source,sftp-source,ftps-sink,sftp-sink,mainand this branchcamel get endpointshows identical URIsfile-watch-sourcerecursive=true; same events asmain(CREATE oftop.csv,notes.txt,sub,sub/deep.csv,old/old.csv)scp-sinkscp://127.0.0.1:2233/?...,mainscp://127.0.0.1:2233?...; both write to the home directoryinclude: .*\.csv, recursive, FTP / SFTPorders.csv,ORDERS2.CSV,sub/deep.csv; notnotes.txt,data.xml,part.csv.tmpexclude: .*\.(tmp|txt), FTP / SFTPorders.csv,ORDERS2.CSV,data.xml,sub/deep.csvinclude: .*\.csv+exclude: orders2.*, FTPorders.csvonly (ORDERS2.CSVexcluded whatever the case)antInclude: *.csv,*.xml, recursive, FTP / SFTPorders.csv,data.xml; notORDERS2.CSV(case) norsub/deep.csvantInclude: **/*.csv, recursive, FTP / SFTPorders.csv,sub/deep.csvantInclude: *.txt, FTPS explicit;include: .*\.csv, FTPS implicitb.txt;a.csvinclude: a+\.csv+exclude: a b\.csv;include: a b\.csv|b\.csvaaa.csv;a b.csvandb.csvpreMove: .inprogress+move: .done, FTP / SFTP / FTPS.inprogress/.done/preMove: .inprogressonly, FTP.inprogress/sortBy: reverse:file:name, FTPe d c b asortBy: reverse:file:modified, files written 1 s apart over two minutes, FTPb d e a c)sortBy: reverse:file:name+maxMessagesPerPoll: 2+delay: 3000, FTPb a,d c,e(the limit picks the first listed, then sorts)sortBy: reverse:file:modified+maxMessagesPerPoll: 3, SFTPd a c, thenb e(the same)delayunset /3000,idempotent: false, FTPreadLock: changed, FTPa.csvread ~6 s after start; the empty file never in 30 sreadLock: rename, FTPreadLock: changed+readLockMinAge: 60000, SFTP,fresh.csvand a 5-minute-oldold.csvold.csvafter the 20 s skip offresh.csv;fresh.csvexactly 60 s after it was writtenfresh.csvread 28 s after it was written (listed to the minute)streamDownload: true, FTP, stepwise unsetThe option stepwise is not supported for stream downloadingstreamDownload+stepwise: false+recursive+move, FTPFileInputStreamCache) / read (SocketInputStream)streamDownload, SFTPmainexclude,preMove,move,sortBy,maxMessagesPerPoll: 2,streamDownload,stepwise: false,readLock: changed)b a,d c,e, ~5 s apart each;x.tmpleft; files in.inprogress/.doneconnectTimeout: 2000/ unset, unroutable address, FTPconnectTimeout(Timed out waiting for initial connect reply), notsocketTimeoutsocketTimeout: 3000/ unset, FTP server pausedRead timed out3 s after the pause / nothing in 45 sdataTimeout: 3000/ unset, data connection that never sends, FTPRead timed out3.7 s / 30 s afterLISTsocketTimeout: 3000/serverAliveInterval: 2000/ unset, SFTP server pausedserverAliveInterval: 2000/ unset,sftp-sink, two messages 25 s apart, server paused between themimplicit: true, port 990,ftps-source/ftps-sinkup/orders.csvwrittenTimed out waiting for initial connect replyimplicit: true, explicit server on port 21Unsupported or unrecognized SSL messagechmod: 640/ unset /644over an existing600file,sftp-sink-rw-r-----/-rw-r--r--/-rw-r--r--scp-sinkdirectoryName: incoming+chmod: 640incoming/dir.csv,-rw-r-----scp-sinkdirectoryName: /tmp/abs/tmp/abs/abs.csvscp-sinkdirectoryName: missing(not there)missingin the home directory, with the contentscp-sinkdirectoryName: incoming,CamelFileName: sub/deep.csvincoming/sub/deep.csv,subcreatedscp-sinkCamelFileName: ../escape.csvCannot write file with name: ../escape.csv; nothing writtenscp-sinkchmod: 666/ unset /644over an existing600file-rw-r--r--(umask 022) /-rw-r--r--/ stays-rw-------file-watch-sourceantInclude: *.csvtop.csvonlyfile-watch-sourceantInclude: **/*.csvtop.csv,sub/deep.csv,old/old.csvfile-watch-sourcerecursive: falsetop.csv,notes.txt,sub; notsub/deep.csv,old/old.csv./mvnw clean verify -DskipTestsand then./mvnw clean verifypassed from the root. Before building, the library copies of the seven Kamelets were refreshed with./mvnw -pl library/camel-kamelets resources:copy-resources@copy-resource-one. The build regeneratedkamelets-index.json.KameletsCatalogTestneeded no change (no headers were added).Claude Code on behalf of Claus Ibsen
🤖 Generated with Claude Code