Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -2972,10 +2972,14 @@ until `expiresAt`. Only `release: true` ends the paid window early.

#### Description

Stream the service container's stdout/stderr logs live. **Authenticated and owner-scoped**
Stream the service container's stdout/stderr logs. **Authenticated and owner-scoped**
— only the service's owner (`consumerAddress`, proven by signature/nonce or auth token) can
read its logs. Available while the service is `Running` (`40`) or `Error` (`99`) — a crashed
container is kept around until `stop`/`restart`, so its logs remain fetchable for diagnosis.
read its logs. While the service has a container — `Running` (`40`), `Error` (`99`): a
crashed container is kept around until `stop`/`restart`, or `Stopping` until the teardown
removes it — its log is followed live. Once the
container is gone, the logs the node saved when each of the service's containers was removed
are served instead, oldest first, until `storageExpiry` seconds after `expiresAt`. Only the last
16 MiB of each container's log are kept.

#### Query Parameters

Expand All @@ -2989,22 +2993,31 @@ container is kept around until `stop`/`restart`, so its logs remain fetchable fo

#### Response (200)

Raw `stdout`/`stderr` byte stream from the container, connection kept open and followed live.
With `since` set, historical output before that point is skipped — useful for a long-lived
service where fetching the full history would otherwise dump days/weeks of buffered logs
before reaching the live tail (e.g. `since=1h` for just the last hour).
Raw `stdout`/`stderr` byte stream from the container (Docker-multiplexed frames), connection
kept open and followed live. Saved logs come in the same format, and the stream ends after the
last line. With `since` set, historical output before that point is skipped — useful for a
long-lived service where fetching the full history would otherwise dump days/weeks of buffered
logs before reaching the live tail (e.g. `since=1h` for just the last hour).

#### Response (400)

`since` is present but not a valid Unix timestamp or duration (`<number><s|m|h|d>`).
`since` is present but not a valid Unix timestamp or duration (`<number><s|m|h|d>`), or no
service with this `serviceId` belongs to `consumerAddress` (`Service job not found`).

#### Response (404)

Service not found, or not `Running`/`Error`.
The service exists but has no container and no saved logs (never saved, or deleted after
storage expiry).

#### Response (401)

Missing/invalid auth, or `consumerAddress` is not the service owner.
Missing/invalid auth. A service owned by another address gets `400`, as above: the lookup is
owner-scoped, so it is not found.

#### Response (500)

The node could not read the container's live log from Docker. Saved logs are not served in its
place, so a Docker failure is never mistaken for an empty log.

---

Expand Down
39 changes: 34 additions & 5 deletions docs/services.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ and `signature` as query parameters (or an auth-token `Authorization` header).
| `SERVICE_RESTART` | `/api/services/serviceRestart` | POST | Recreate the container (no extra charge); asynchronous like start — returns once the job is `Restarting`, poll `serviceStatus`. Optionally restart on a **new image spec** (bug-fix flow) — see below |
| `SERVICE_STOP` | `/api/services/serviceStop` | POST | Tear down the container; the paid resource reservation (cpu/ram/gpu + host ports) is kept until `expiresAt`, so the service can be restarted anytime on the same endpoints. `release: true` ends the paid window now and frees it instead (no refund, no restart) |
| `SERVICE_GET_TEMPLATES` | `/api/services/serviceTemplates` | GET | List operator-published service templates |
| `SERVICE_GET_STREAMABLE_LOGS` | `/api/services/serviceStreamableLogs` | GET | Stream the container's live stdout/stderr logs — authenticated, owner-scoped; available while `Running` or `Error`; optional `since` to skip history |
| `SERVICE_GET_STREAMABLE_LOGS` | `/api/services/serviceStreamableLogs` | GET | Stream the container's stdout/stderr logs — authenticated, owner-scoped; followed live while the service has a container (`Running`, `Error` before teardown, or `Stopping`), otherwise the logs saved when its containers were removed, until storage expiry; optional `since` to skip history. See [Service logs](#service-logs) |
| `SERVICE_GET_RESULT` | `/api/services/serviceResult` | GET | Download the service's `/data/outputs` as a zip — one of its archives (`index`, resumable with `offset`) or the running container's (`live=true`); authenticated, owner-scoped. See [Service outputs](#service-outputs) |

**Start is asynchronous.** `serviceStart` does only the fast, synchronous validation and then
Expand Down Expand Up @@ -269,6 +269,34 @@ compute job's outputs:
`outputArchives: []`.
- There is no size cap on archives beyond the node's free disk.

## Service logs

`serviceStreamableLogs` follows the container's stdout/stderr live while the service has a
container. Docker deletes a container's logs together with the container, so the node saves them
first, which keeps them available after the session ends:

- **They are saved whenever a container of the service goes away**: `serviceStop`, the expiry
sweep, a `serviceRestart` (the old container), and the crash recovery at node start. This
happens for every service, including one with an output bucket, because the bucket only holds
`/data/outputs`. Each container gives one file, `logs-<n>.log`, next to the output archives
under `<c2d storage>/services/<serviceId>/`. The files are listed on `serviceStatus` as
`logArchives` (`index`, `filename`, `filesize`, `createdAt`, `containerId`, `droppedBytes`
when output was cut, and `cutShort` when reading the log timed out, so its newest output may
be missing). Saving is best-effort: if it fails, teardown continues and the node only logs the
error. A container that logged nothing produces no file.
- **Only the last 16 MiB of each container's log are kept.** When a container logged more, its
oldest output is dropped, and the saved log starts with an `[ocean-node]` line on stderr that
says how much was dropped. The node asks Docker for the last 430,000-odd entries only, which
always add up to more than 16 MiB, so for a longer log the line gives a lower bound ("at
least …").
- **Once the service has no container, `serviceStreamableLogs` serves the saved logs**, oldest
container first, in the same multiplexed format as the live stream. The stream ends after the
last line instead of following. `since` works the same way. The route returns `404` only when
there is neither a container nor a saved log. While a container exists, only its own live log
is returned, so after a `serviceRestart` the previous containers' logs are not included.
- **Saved logs expire with the output archives**: `storageExpiry` seconds after `expiresAt`, by
the same cron. The service record is kept, with `logArchives: []`.

## Configuration

Service-on-demand is configured per Docker connection under `serviceOnDemand`:
Expand Down Expand Up @@ -630,16 +658,17 @@ then `docker network rm t`.
- **`serviceStreamableLogs` is authenticated and owner-scoped, like `serviceStatus`.**
Container stdout/stderr can leak secrets or sensitive request data, so the same
proof-of-`consumerAddress` + ownership check gates log access — a non-owner gets `401`.
Logs are only served while the service is `Running` or `Error` (a crashed container's
logs stay available for diagnosis until `stop`/`restart` tears it down); otherwise the
route returns `404`. By default the full history since container start is returned before
Logs are followed live while the service has a container (`Running`, or `Error` until
`stop`/`restart` tears the crashed container down). After that the logs saved at teardown are
served until storage expiry (see [Service logs](#service-logs)), and the route returns `404`
only when there are none. By default the full history since container start is returned before
the stream switches to following live output — for a service that has been running for
days or weeks that can be a lot of data, so pass `since` (a Unix timestamp, or a relative
duration like `1h`) to skip straight to recent output.

- **`serviceResult` is authenticated and owner-scoped, like `serviceStreamableLogs`.** A
service's outputs are its owner's data; a non-owner gets the same `400` as an unknown
`serviceId`. The node-wide `serviceList` leaves out `outputArchives`.
`serviceId`. The node-wide `serviceList` leaves out `outputArchives` and `logArchives`.

- **`allowImageBuild` runs arbitrary build instructions.** When enabled, a consumer's
inline `dockerfile` is built by the Docker daemon, so its `RUN` steps execute arbitrary
Expand Down
18 changes: 18 additions & 0 deletions src/@types/C2D/ServiceOnDemand.ts
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,22 @@ export interface ServiceOutputArchive {
containerId: string // container it was taken from — a retried teardown must not archive it twice
}

// Docker deletes a container's logs together with the container, so each service container's
// stdout/stderr is saved before it is removed. That keeps the logs readable through
// SERVICE_GET_STREAMABLE_LOGS after a stop or expiry, until expiresAt + storageExpiry.
export interface ServiceLogArchive {
index: number
filename: string // e.g. logs-0.log, under <storage>/services/<serviceId>/
filesize: number // bytes
createdAt: number // Unix ms timestamp
containerId: string // container it was taken from — a retried teardown must not save it twice
// Recorded so the owner can tell a trimmed log from a complete one. A lower bound for a log
// longer than SERVICE_LOGS_TAIL_ENTRIES entries, since Docker sends no more than that.
droppedBytes?: number
// The read timed out before the log ended, so the newest output may be missing.
cutShort?: boolean
}

export interface ServiceEndpoint {
containerPort: number
hostPort: number
Expand Down Expand Up @@ -304,6 +320,8 @@ export interface ServiceJob {
outputBucketId?: string // persistent-storage bucket bind-mounted at /data/outputs
// Archives of /data/outputs, only for services without an outputBucketId.
outputArchives?: ServiceOutputArchive[]
// Kept for services with an outputBucketId too: the bucket holds only /data/outputs.
logArchives?: ServiceLogArchive[]
// Internal, only while a restart is in flight: the stopped old container, kept until its
// /data/outputs has been carried over into the new one (orphan recovery archives and
// removes it if the node dies before that). Never returned to clients.
Expand Down
8 changes: 6 additions & 2 deletions src/components/c2d/compute_engine_base.ts
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,10 @@ export abstract class C2DEngine {
return []
}

// Owners need a service's logs after it stops too, and Docker deletes them with the container:
// so once there is no container, the logs saved at teardown (ServiceJob.logArchives) are served.
// Null when there is neither. Throws when the container can't be read, which is not the same as
// having no logs.
// eslint-disable-next-line require-await
public async getServiceStreamableLogs(
serviceId: string,
Expand All @@ -231,8 +235,8 @@ export abstract class C2DEngine {
return null
}

// Deletes the output archives of expired services whose environment's storageExpiry has
// elapsed since expiresAt. Returns how many services had their archives deleted.
// Deletes the output archives and saved logs of expired services whose environment's
// storageExpiry has elapsed since expiresAt. Returns how many services had them deleted.
// eslint-disable-next-line require-await
public async cleanupExpiredServiceOutputs(): Promise<number> {
return 0
Expand Down
Loading
Loading