Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions benchmarking/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,9 +371,10 @@ deployment fails loudly instead of showing up as OOM-flaky steps.
gap gets ±20% jitter so sessions don't move in lockstep.
* `--resume-mode implicit|explicit` — implicit (default) lets the parked
first request wake the actor; explicit issues ResumeActor before traffic.
* `--lifecycle-mode suspend|pause|none` — durable suspend (default),
node-local pause between steps, or `none` to keep actors resident in memory
between steps without hibernating.
* `--lifecycle-mode suspend|pause|none` — synchronous suspend (`suspend`,
default, uploads the snapshot before returning), asynchronous suspend
(`pause`, uploads the snapshot in the background), or `none` to keep actors
resident in memory between steps without hibernating.
* `--agentsession-snapshot-steps` — comma-separated 1-based step numbers after
which the actor is hibernated (for example, `3,10,20,30,40,50,61`); between
all other steps the actor stays resident.
Expand Down Expand Up @@ -539,34 +540,34 @@ count users × `--actors-per-user`, not `--total-actors`.
### ResumePaused Benchmark

The ResumePaused benchmark drives ate-api-server's cold resume path at a set
rate. A cold resume here is a ResumeActor of a paused actor: ateapi takes the
rate. A cold resume here is a ResumeActor of a suspended actor: ateapi takes the
actor lease, binds a worker, and restores the actor from its snapshot. A
ResumeActor of a running actor does none of that. Each user owns one actor
from the `glutton` template. On its first iteration the user creates the actor
and makes it cold with a resume and a pause; every later iteration resumes the
actor and pauses it again, so every measured resume is cold.
and makes it cold with a resume and a suspend; every later iteration resumes the
actor and suspends it again, so every measured resume is cold.

The warm-up runs in the step that added the user, unpaced, and is recorded
under its own names.

It sends no requests to the actors themselves, so it runs on the fake data
plane, where it measures what a cold resume costs ate-api-server and Postgres
without the router or the restore in the way. The pauses run at the resume
rate, so a capacity it finds is one of resume and pause cycles, not of resumes
without the router or the restore in the way. The suspends run at the resume
rate, so a capacity it finds is one of resume and suspend cycles, not of resumes
alone.

Actors are named `rp-<uuid>` and deleted when boomer exits. If the atespace
cannot be ensured, boomer exits at startup. An actor that fails to warm is
deleted, and its user creates another on its next iteration. When a step lowers
the user count, the actors of the stopped users stay paused, unused, until
the user count, the actors of the stopped users stay suspended, unused, until
boomer exits.

#### ResumePaused Configuration Knobs

* `--target-rps`: Cold resumes per second the whole boomer process paces to
(default `0`, unpaced). A `--ladder` step ending in `@target_rps`, as in
`200:3m@1000`, changes it as the step starts. Keep the users at or above the
rate times the p99 of one resume plus one pause in seconds, or the step falls
rate times the p99 of one resume plus one suspend in seconds, or the step falls
short of its rate.

The web UI shows the same field.
Expand Down Expand Up @@ -792,9 +793,9 @@ actually did, independent of what the load generator reported.
* `snapshots.restore_p50_s` through `restore_p99_s`, `restore_mean_s`, and the
same for `checkpoint_*`: how long the atelet took to resume or suspend an
actor (`ate_actor_{restore,checkpoint}_duration`, `total` phase). Every
snapshot kind counts, as in the sizes: the actor's own snapshot, a
template's first start from its golden snapshot, and a pause. The buckets
reach 60s, so a slower operation reads as 60s.
snapshot kind counts, as in the sizes: the actor's own snapshot and a
template's first start from its golden snapshot. The buckets reach 60s, so a
slower operation reads as 60s.
* `snapshots.checkpoints_in_window`, `checkpoints_cumulative`: checkpoint
volume over the steady-state window, and since the atelet started.
* `snapshots.checkpoint_mb_s`: bytes written per second spent checkpointing,
Expand Down
4 changes: 2 additions & 2 deletions benchmarking/locust/common/lifecycle_mode.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ def add_lifecycle_mode_arguments(parser: LocustArgumentParser) -> None:
type=str,
default="suspend",
choices=["suspend", "pause", "none"],
help="Lifecycle mode: 'suspend' uses SuspendActor (persists snapshot to storage); "
"'pause' uses an asynchronous SuspendActor (keeps snapshot on local node); "
help="Lifecycle mode: 'suspend' uses synchronous SuspendActor (uploads snapshot before returning); "
"'pause' uses asynchronous SuspendActor (uploads snapshot in background); "
"'none' keeps actors resident in memory between steps (default: suspend)",
)
8 changes: 4 additions & 4 deletions cmd/ateapi/internal/controlapi/actor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -361,8 +361,8 @@ func TestUpdateActor_RepointTemplate(t *testing.T) {
t.Fatalf("UpdateActor repointing a running actor = %v, want FailedPrecondition (err: %v)", got, err)
}

// Repointing a paused actor is allowed while suspended, even when it holds
// a node-local snapshot.
// Repointing a suspended actor is allowed even when it holds a node-local
// snapshot.
for _, name := range []string{"paused-actor", "uploaded-paused-actor"} {
paused := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: name},
Expand Down Expand Up @@ -468,10 +468,10 @@ func TestUpdateActor_RepointTemplateStorageLocation(t *testing.T) {
name string
template string
snapshot func(*testing.T, *ateapipb.Actor) string
// pending leaves the snapshot's upload PENDING, as a pause does.
// pending leaves the snapshot's upload PENDING, as an async suspend does.
pending bool
// lastAssignedGen overrides status.LastAssignedGeneration when > 0
// (e.g. after a reverted pause or crashed suspend pruned a higher
// (e.g. after a reverted or crashed suspend pruned a higher
// pending generation).
lastAssignedGen int32
repointTo string
Expand Down
18 changes: 9 additions & 9 deletions cmd/ateapi/internal/controlapi/export_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ type ReleaseWindowWorkflow struct {
// Snapshot, when set, is the actor's latest snapshot.
Snapshot func(t *testing.T) *ateapipb.Snapshot
// Finalize runs the workflow or step that releases the worker and then
// records the actor's new state against st. Pause and suspend reach the
// records the actor's new state against st. Async and sync suspend reach the
// fake atelet newWireCaptureWorkflow serves for the worker
// WireTestAssignment names, so the actor must be assigned that worker
// and run from the template SeedSuspendTemplate stores.
Expand All @@ -64,11 +64,11 @@ type ReleaseWindowWorkflow struct {
WantFinalState ateapipb.ActorState
}

// ReleaseWindowWorkflows returns pause, suspend, crash, and a resume onto an
// ineligible worker: the workflows that release a worker before rewriting the
// actor's status while the actor is in a state HostingNode resolves through
// the worker assignment. Revert and delete release theirs in REVERTING and
// DELETING, which HostingNode never resolves to a node.
// ReleaseWindowWorkflows returns async suspend ("pause"), sync suspend, crash,
// and a resume onto an ineligible worker: the workflows that release a worker
// before rewriting the actor's status while the actor is in a state HostingNode
// resolves through the worker assignment. Revert and delete release theirs in
// REVERTING and DELETING, which HostingNode never resolves to a node.
func ReleaseWindowWorkflows() []ReleaseWindowWorkflow {
// The workers these tests create carry no labels, so a template
// selecting a pool makes any of them ineligible.
Expand All @@ -80,10 +80,10 @@ func ReleaseWindowWorkflows() []ReleaseWindowWorkflow {
Name: "pause",
State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING,
Snapshot: func(t *testing.T) *ateapipb.Snapshot {
// A pause in progress: a PENDING durable copy, not yet captured.
// An async suspend in progress: a PENDING durable copy, not yet captured.
return newInProgressSnapshot(t, 1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "local-snap-1")
},
// The actor is seeded SUSPENDING, so the pause skips its mark step
// The actor is seeded SUSPENDING, so the suspend skips its mark step
// and its first actor update is the finalize commit.
Finalize: func(t *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error {
w, _ := newWireCaptureWorkflow(t, st)
Expand All @@ -99,7 +99,7 @@ func ReleaseWindowWorkflows() []ReleaseWindowWorkflow {
Snapshot: func(t *testing.T) *ateapipb.Snapshot {
return newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snapshot-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING)
},
// As for the pause, the finalize commit is the first actor update.
// As for the async suspend, the finalize commit is the first actor update.
Finalize: func(t *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error {
w, _ := newWireCaptureWorkflow(t, st)
_, err := w.SuspendActor(ctx, actorRef, suspendSync)
Expand Down
Loading
Loading