Repository navigation
Conversation
…up at startup Add characterization tests for start_executor_process ahead of splitting start_executor_process_inner into phases (apache#2597): the single connect attempt with scheduler_connect_timeout_seconds = 0, the memory pool size reported to ExecutorMemoryMetrics for a configured and an unbounded pool, and the shuffle clean-up: with job_data_clean_up_interval_seconds = 0 neither the periodic cleaner nor the clean-up at shutdown runs, and with a positive interval job data is removed on shutdown. The last two run a pull-based executor against an idle in-process scheduler and shut it down by taking its Flight port. Co-authored-by: Megabot <megabot@megabot.com.br>
…s_inner Part of apache#2597. Moves the vcores resolution verbatim; behaviour is unchanged. Co-authored-by: Megabot <megabot@megabot.com.br>
…_process_inner Part of apache#2597. Moves the periodic shuffle cleaner task verbatim; behaviour is unchanged. Adds a test that it removes expired job dirs and stops on shutdown. Co-authored-by: Megabot <megabot@megabot.com.br>
…ocess_inner Part of apache#2597. Moves the service/signal/stop-request select verbatim; behaviour is unchanged. Adds tests for the stop-request and failed-service outcomes. Co-authored-by: Megabot <megabot@megabot.com.br>
…cutor_process_inner Part of apache#2597. Moves the memory pool resolution and its metrics wiring verbatim; behaviour is unchanged. The metrics handle is passed as Option<&ExecutorMemoryMetrics>. Covered by the memory pool metrics tests. Co-authored-by: Megabot <megabot@megabot.com.br>
andygrove
left a comment
There was a problem hiding this comment.
Thanks for splitting this up so carefully. Each extraction reads as a literal move, and the new tests pin the behaviour well. I left a few small optional suggestions on the tests. Happy to see these handled here or in the follow-up PRs.
| .add_service(SchedulerGrpcServer::new(IdleScheduler)) | ||
| .serve_with_incoming(TcpListenerStream::new(scheduler)), | ||
| ); | ||
| let flight_port_holder = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); |
There was a problem hiding this comment.
Nice approach to drive the real shutdown path end to end. One thing I noticed is that create_grpc_server_incoming sets SO_REUSEADDR, and on Windows that can let the Flight server bind the port the test is holding. The executor would then never stop and the test would time out. Would it work to pass an override_arrow_flight_service that returns a handle which fails right away? That would also make the trigger explicit instead of relying on a port collision.
There was a problem hiding this comment.
Good catch on SO_REUSEADDR on Windows. The helper now passes an override_arrow_flight_service whose handle fails right away, so the trigger is explicit and no port is held (renamed to run_executor_until_flight_service_fails). Done in 30c1c7a. I could only run this on Linux, so the Windows benefit follows from the design rather than from a run there.
| /// stops a service right after startup; it then goes through the normal | ||
| /// graceful shutdown path, which runs to the end with the scheduler. | ||
| /// | ||
| /// This sets the process-wide `TERMINATING` flag, which no other unit test |
There was a problem hiding this comment.
Thanks for documenting the TERMINATING side effect. Since heartbeat_params reads it, a future test asserting an Active status could flake depending on test order. Would you mind storing false back at the end of run_executor_until_flight_port_is_taken? It keeps the helper self-contained.
There was a problem hiding this comment.
Done in c311be9. A small drop guard resets the flag, so it also happens if the helper panics.
| } | ||
|
|
||
| /// Reserves a local port and releases it, so nothing listens there. | ||
| fn unused_local_port() -> u16 { |
There was a problem hiding this comment.
Small one. scheduler_connect_timeout_is_honoured inlines the same reserve-and-release logic as unused_local_port. Could it use the new helper?
…is_honoured Co-authored-by: Megabot <megabot@megabot.com.br>
…elper Co-authored-by: Megabot <megabot@megabot.com.br>
…light service Co-authored-by: Megabot <megabot@megabot.com.br>
Which issue does this PR close?
Part of #2597 (first of three PRs; it does not close the issue).
Rationale for this change
start_executor_process_innerinballista/executor/src/executor_process.rsis one longfunction that mixes several unrelated startup and shutdown phases. #2597 asks to split it
into smaller steps so that each one can be read, tested and changed on its own. As described
in the plan on the issue, this is the first PR of three: it extracts the phases that do not
overlap with the open PRs touching the same function. The scheduler connection
(
build_endpoint/connect_to_scheduler) andresolve_work_dirare left for the next twoPRs, once their prerequisites are merged.
This is a behaviour-preserving refactor: the moved code is copied literally, in the same
order, with the same log lines, errors and ordering of side effects.
What changes are included in this PR?
Only
ballista/executor/src/executor_process.rschanges. Four phases move out ofstart_executor_process_innerinto their own functions, one commit per step:resolve_vcores: the "0 means use the available parallelism" rule.spawn_shuffle_cleaner: the periodic shuffle clean-up task and its shutdown handle. Thejob_data_clean_up_interval_seconds > 0decisions stay instart_executor_process_inner.run_until_shutdown: waiting for the service handlers, a stop request or a terminationsignal, and notifying the scheduler on failure.
resolve_memory_pool_policy: choosing the memory pool budget, registering the metrics andwrapping the policy with the pool observer. It takes
Option<&ExecutorMemoryMetrics>. (Theplan on the issue called it
build_memory_pool_policy; it is renamed so it does not readlike the existing
memory_pool_policy(bytes, vcores)constructor.)The first commit only adds tests and does not touch production code. They pin the current
behaviour and pass on
mainbefore any extraction:scheduler_connect_timeout_seconds = 0makes a single attempt;job_data_clean_up_interval_seconds = 0the cleaner is not started and job data iskept on shutdown; with a positive interval job data is removed on shutdown. These two run a
whole executor against a fake idle scheduler and stop it by taking its Flight port.
Each extraction commit adds the unit tests of its own function (
resolve_vcores, the cleaner,and both outcomes of
run_until_shutdown).How it was tested, all on Linux, at the last commit:
-D warnings,cargo check,cargo doc, the workspace tests including the standalone and benchmark ones, toml, proto andconfig-docs checks, machete, RAT, prettier) pass.
fails, then restored the file.
one built from
main: one scheduler, one executor (--job-data-clean-up-interval-seconds 1 --job-data-ttl-seconds 1), aSELECT k, count(*), sum(v) ... GROUP BY k ORDER BY kover a100000-row CSV, then
SIGTERMon the executor. The result is the same, the cleaner empties thework dir, and the shutdown sequence in the logs is the same.
load, with no failure.
Not verified: macOS builds and tests, the h2o and TPC-H/TPC-DS jobs, the Kubernetes chaos job,
the Docker build, the Python bindings, CodeQL and the allowlist check.
SIGINT, the unboundedmemory pool and the automatic
--memory-pool-sizewere checked only by unit tests, not on areal cluster. Under very heavy CPU load I saw the existing
scheduler_connect_timeout_is_honouredandscheduler_connect_timeout_bounds_a_hanging_connecttests fail now and then; they are not touched by this PR.
Are there any user-facing changes?
No. There is no change to the public API, configuration, logs, protocol or behaviour.
Co-developed with Megabot, my AI-assisted development pipeline (Claude Sonnet 5.5; reviewed, tested and owned by me).