Skip to content

Add log cache log streaming support - #1338

Open
ZPascal wants to merge 4 commits into
cloudfoundry:5.x.xfrom
ZPascal:add-log-cache-log-streaming-support
Open

ZPascal wants to merge 4 commits into
cloudfoundry:5.x.xfrom
ZPascal:add-log-cache-log-streaming-support

Conversation

@ZPascal

@ZPascal ZPascal commented Mar 3, 2026

Copy link
Copy Markdown

Summary

Adds live log streaming via Log Cache — the Java equivalent of cf tail --follow.

Previously the only Log Cache read path was logsRecent, a single snapshot request. There was no way to continuously receive new log envelopes without polling manually. This change introduces a logsTail API across all three relevant modules.


Motivation

Cloud Foundry dropped the legacy Loggregator Doppler streaming endpoint (DopplerClient.stream()) in Loggregator ≥ 107.0 (CFD ≥ 24.3 / TAS ≥ 4.0). The CF CLI replaced it with a polling loop over the Log Cache /api/v1/read endpoint — that is what cf tail --follow does today. This PR brings the same capability to Java consumers.


Changes

Changes in detail

cloudfoundry-client — new TailLogsRequest and LogCacheClient.logsTail()

New _TailLogsRequest.java (Immutables @Value.Immutable):

Field Type Default Description
sourceId String App / service GUID (required)
startTime Long (nullable) now − 5 s (ns) Cursor start time in UNIX nanoseconds
envelopeTypes List<EnvelopeType> (nullable) all types Envelope type filter
nameFilter String (nullable) none Regex name filter (Log Cache ≥ 2.1.0)
pollInterval Duration 250 ms Back-off between polls when no new data

New method on LogCacheClient:

/**
 * Continuously polls Log Cache /api/v1/read and streams new Envelopes as they appear.
 * Equivalent to the Go logcache.Walk() API and `cf tail --follow`.
 * The Flux never completes on its own – cancel the subscription to stop streaming.
 */
Flux<Envelope> logsTail(TailLogsRequest request);

cloudfoundry-client-reactor — non-blocking polling implementation

ReactorLogCacheEndpoints.logsTail() mirrors the Go logcache.Walk() algorithm, fully non-blocking:

  1. An AtomicLong cursor starts at startTime (or now − 5 s).
  2. Flux.defer builds a fresh ReadRequest from the current cursor on every repetition and calls GET /api/v1/read/{sourceId}?start_time=cursor.
  3. Envelopes are sorted ascending by timestamp; the cursor advances to lastTimestamp + 1; each envelope is emitted individually downstream.
  4. When the batch is empty, repeatWhen inserts a Mono.delay(pollInterval) before the next poll. When envelopes arrive, the next poll starts immediately.
  5. The Flux is infinite; the caller cancels the subscription to stop.

Transient network errors are logged and retried. HTTP errors (4xx/5xx) are propagated to the subscriber.


cloudfoundry-operationsApplications.logsTail()

DefaultApplications delegates to LogCacheClient:

@Override
public Flux<Envelope> logsTail(TailLogsRequest request) {
    return this.logCacheClient
            .flatMapMany(client -> client.logsTail(request))
            .transform(OperationsLogging.log("Tail Application Logs"))
            .checkpoint();
}

Tests

Test What it verifies
logsTailLogCache Happy path: a single LOG/OUT envelope is forwarded correctly
logsTailLogCacheMultipleEnvelopes 3 envelopes with types OUT → ERR → OUT are emitted in order
logsTailLogCacheError A RuntimeException from the client propagates unchanged to the subscriber
logsTailLogCacheOutAndErrEnvelopes Both stdout and stderr envelopes are forwarded without filtering

Usage

TailLogsRequest request = TailLogsRequest.builder()
        .sourceId(applicationGuid)
        .envelopeTypes(List.of(EnvelopeType.LOG))
        .pollInterval(Duration.ofMillis(250))
        .build();

logCacheClient.logsTail(request)
        .filter(e -> e.getLog() != null)
        .map(e -> e.getLog().getPayloadAsText())
        .subscribe(System.out::println);

Or via the Operations API:

cloudFoundryOperations.applications()
        .logsTail(TailLogsRequest.builder().sourceId(appGuid).build())
        .filter(e -> e.getLog() != null)
        .map(e -> e.getLog().getPayloadAsText())
        .subscribe(System.out::println);

Relation to existing API

Method Transport Completes? Use case
DopplerClient.stream() ⚠️ deprecated WebSocket / Doppler Yes (server closes) Legacy streaming (Loggregator < 107.0)
LogCacheClient.recentLogs() HTTP GET (single) Yes Fetch last N log lines
LogCacheClient.logsTail() ✅ new HTTP GET (polling loop) Never Live streaming (Loggregator ≥ 107.0)

Checklist

  • _TailLogsRequest Immutables value object
  • LogCacheClient.logsTail() interface method
  • ReactorLogCacheEndpoints.logsTail() — non-blocking Reactor implementation
  • _ReactorLogCacheClient.logsTail() — delegate override
  • Applications.logsTail() — Operations API method
  • DefaultApplications.logsTail() — implementation
  • Unit tests in DefaultApplicationsTest (120 passing)
  • Integration tests

@ZPascal
ZPascal force-pushed the add-log-cache-log-streaming-support branch from 606817f to 08fb796 Compare March 7, 2026 14:43
@ZPascal
ZPascal changed the base branch from main to 5.x.x March 31, 2026 20:54
@ZPascal
ZPascal force-pushed the add-log-cache-log-streaming-support branch 5 times, most recently from 4e11f4c to b0ce71f Compare June 9, 2026 15:54
@ZPascal
ZPascal marked this pull request as ready for review June 9, 2026 15:55
@ZPascal
ZPascal marked this pull request as draft June 9, 2026 15:55
@ZPascal
ZPascal force-pushed the add-log-cache-log-streaming-support branch 2 times, most recently from 400d808 to 9105a92 Compare June 10, 2026 04:27
@ZPascal
ZPascal force-pushed the add-log-cache-log-streaming-support branch from 9105a92 to 06acac7 Compare June 10, 2026 05:28
Signed-off-by: I539231 <pascal.zimmermann01@sap.com>
@ZPascal
ZPascal marked this pull request as ready for review June 17, 2026 16:00
Comment thread integration-test/src/test/java/org/cloudfoundry/operations/ApplicationsTest.java Outdated
Comment thread integration-test/src/test/java/org/cloudfoundry/operations/ApplicationsTest.java Outdated
Comment thread integration-test/src/test/java/org/cloudfoundry/operations/ApplicationsTest.java Outdated
Comment thread integration-test/src/test/java/org/cloudfoundry/operations/ApplicationsTest.java Outdated
Comment thread integration-test/src/test/java/org/cloudfoundry/operations/ApplicationsTest.java Outdated
Comment thread integration-test/src/test/java/org/cloudfoundry/operations/ApplicationsTest.java Outdated
@ZPascal
ZPascal requested a review from Lokowandtg September 22, 2026 09:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants