Debugging CI failures when parallel frontend is on by default by aerooneqq · Pull Request #165506 · rust-lang/rust

aerooneqq · 2026-11-05T16:08:41Z

Trying to fix tests that fail when we enable two threads in frontend, this is a fix for a test failure when we did not return all tokens for job server. I tested it in top of #162848 with 2 and 222 threads and did not reproduce after 200 launches, while it was certainly reproducing after 20-30 launches without this fix.

There is a second failure which is an infinite hang that I managed to get one or two times locally (with 222 threads). I don't know the reason for it, but maybe this fix will also help as it did not reproduce with it.

cc @Kobzol @petrochenkov

aerooneqq · 2026-11-06T10:06:08Z

@bors try jobs=test-x86_64-gnu-aux

Make `Proxy` weak reference as it seems that it is not dropped at all try-job: test-x86_64-gnu-aux

rust-bors · 2026-11-06T11:09:44Z

💔 Test for 23ce799 failed: CI. Failed job:

try - test-x86_64-gnu-aux (web logs, enhanced plaintext logs)

aerooneqq · 2026-11-06T11:24:44Z

To sum up the current state:

The first idea (the error in tokens calculation and the rewrite of it) didn't work locally as there is no error,
The second idea that Proxy is not dropped and something bad happens because of it fixed the issue with tokens locally, but it encounetered hang after ~300 launches of ./x t src/tools/cargo --stage 1 -- jobserver
Then I changed the command to ./x t src/tools/cargo --stage 1 -- jobserver --test-threads=1 and it survived 609 launches until I stopped it

In CI the failure now differs: locally I had only diff between tokens in test jobserver_and_j, here we have a failure of the code that is in https://github.com/rust-lang/cargo/blob/3c747ea3ead23951c1b43a5edfcbdb449135ddeb/tests/testsuite/jobserver.rs#L42, so it happened when this test code was executed, locally I haven't ever encountered this error:

failures:
2026-11-06T11:09:35.9053866Z 
2026-11-06T11:09:35.9054359Z ---- jobserver::runner_inherits_jobserver stdout ----
2026-11-06T11:09:35.9055125Z running `/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/cargo install --path .`
2026-11-06T11:09:35.9055739Z running `make run -j2`
2026-11-06T11:09:35.9055911Z 
2026-11-06T11:09:35.9056303Z thread 'jobserver::runner_inherits_jobserver' (129959) panicked at src/tools/cargo/tests/testsuite/jobserver.rs:198:10:
2026-11-06T11:09:35.9056783Z 
2026-11-06T11:09:35.9056902Z test failed running `make run -j2`
2026-11-06T11:09:35.9057405Z error: process exited with code 2 (expected 0)
2026-11-06T11:09:35.9058050Z --- stdout
2026-11-06T11:09:35.9058846Z make[1]: Entering directory '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/tmp/cit/src/tools/cargo/testsuite/jobserver/runner_inherits_jobserver/foo'
2026-11-06T11:09:35.9059895Z /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/cargo run
2026-11-06T11:09:35.9060953Z make[1]: Leaving directory '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/tmp/cit/src/tools/cargo/testsuite/jobserver/runner_inherits_jobserver/foo'
2026-11-06T11:09:35.9061905Z 
2026-11-06T11:09:35.9062075Z --- stderr
2026-11-06T11:09:35.9063481Z    Compiling cargo-jobserver-check v0.0.1 (/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/tmp/cit/src/tools/cargo/testsuite/jobserver/runner_inherits_jobserver/foo)
2026-11-06T11:09:35.9065215Z     Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.38s
2026-11-06T11:09:35.9065953Z      Running `target/debug/cargo-jobserver-check`
2026-11-06T11:09:35.9066359Z 
2026-11-06T11:09:35.9066612Z thread 'main' (130227) panicked at src/main.rs:26:44:
2026-11-06T11:09:35.9067826Z called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
2026-11-06T11:09:35.9068962Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2026-11-06T11:09:35.9069642Z make[1]: *** [Makefile:2: run] Error 101
2026-11-06T11:09:35.9070281Z make[1]: INTERNAL: Exiting with 1 jobserver tokens available; should be 2!

If I understand correctly (I haven't debugged it yet) the cause of the missing token is not straight in the compiler but because we failed to write it back in main.

petrochenkov · 2026-11-06T13:23:46Z

            assert_eq!(read.read(&mut buf).unwrap(), 1);
            assert_eq!(write.write(&buf).unwrap(), 1);

WouldBlock during the write means that the pipe is full, but I'm not sure how it's possible if we just read from it.
Perhaps it makes sense to check first whether read and write actually refer to the same pipe.

petrochenkov · 2026-11-06T13:26:41Z

(I think we can merge the Drop fix without blocking on investigation of the next jobserver::runner_inherits_jobserver failure.)

aerooneqq · 2026-11-09T12:28:06Z

@bors try jobs=test-x86_64-gnu-aux

aerooneqq · 2026-11-12T08:50:43Z

@bors try jobs=test-x86_64-gnu-aux-sync

Debugging CI failures when parallel frontend is on by default try-job: test-x86_64-gnu-aux-sync

rust-bors · 2026-11-12T09:39:06Z

☀️ Try build successful (CI)
Build commit: 2560b1b (2560b1b1e43603a24f206d23fd535f7cde9ce853)
Base parent: 6b05f82 (6b05f82b4f2d1309cdbddece388c9e4d26f481fc)

aerooneqq · 2026-11-12T09:42:01Z

@bors try jobs=test-x86_64-gnu-aux-sync

Debugging CI failures when parallel frontend is on by default try-job: test-x86_64-gnu-aux-sync

rust-bors · 2026-11-12T10:42:45Z

☀️ Try build successful (CI)
Build commit: 8fac9fe (8fac9fe1352b02dec539d7ae5f42043a57e48767)
Base parent: 6b05f82 (6b05f82b4f2d1309cdbddece388c9e4d26f481fc)

aerooneqq · 2026-11-12T10:52:28Z

@bors try jobs=test-x86_64-gnu-aux-sync

Debugging CI failures when parallel frontend is on by default try-job: test-x86_64-gnu-aux-sync

rust-bors · 2026-11-12T11:52:03Z

☀️ Try build successful (CI)
Build commit: 01ea506 (01ea50673778eb3ec46f67c37900f8a0682fc1ea)
Base parent: 6b05f82 (6b05f82b4f2d1309cdbddece388c9e4d26f481fc)

@petrochenkov

…hread, r=petrochenkov Explicitly drop job server helper thread An adaptation of an LLM patch that was generated in https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/with/639482582. The original patch consisted from two fixes: explicit drop of a helper thread and additional logic for job server token request counting. This PR implements only the explicit drop part as I don't see how additional requests counting logic should help the issue. This reworks rust-lang#165676, as the [rationale](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/near/639152114) behind explicit drop of a helper thread seems reasonable for me: > From my rough understanding, while helper is handling a token, its upgrade() is often the last strong ref I guess. So, when run_in_thread_pool_with_globals returns and rustc exits, the token may still be in helper's hands. This fix didn't fail after several launches in rust-lang#165506 and locally. r? @petrochenkov

@petrochenkov

…hread, r=petrochenkov Explicitly drop job server helper thread An adaptation of an LLM patch that was generated in https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/with/639482582. The original patch consisted from two fixes: explicit drop of a helper thread and additional logic for job server token request counting. This PR implements only the explicit drop part as I don't see how additional requests counting logic should help the issue. This reworks rust-lang#165676, as the [rationale](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/near/639152114) behind explicit drop of a helper thread seems reasonable for me: > From my rough understanding, while helper is handling a token, its upgrade() is often the last strong ref I guess. So, when run_in_thread_pool_with_globals returns and rustc exits, the token may still be in helper's hands. This fix didn't fail after several launches in rust-lang#165506 and locally. r? @petrochenkov

@petrochenkov

…hread, r=petrochenkov Explicitly drop job server helper thread An adaptation of an LLM patch that was generated in https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/with/639482582. The original patch consisted from two fixes: explicit drop of a helper thread and additional logic for job server token request counting. This PR implements only the explicit drop part as I don't see how additional requests counting logic should help the issue. This reworks rust-lang#165676, as the [rationale](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/near/639152114) behind explicit drop of a helper thread seems reasonable for me: > From my rough understanding, while helper is handling a token, its upgrade() is often the last strong ref I guess. So, when run_in_thread_pool_with_globals returns and rustc exits, the token may still be in helper's hands. This fix didn't fail after several launches in rust-lang#165506 and locally. r? @petrochenkov

@petrochenkov

Rollup merge of #165854 - aerooneqq:explicitly-drop-helper-thread, r=petrochenkov Explicitly drop job server helper thread An adaptation of an LLM patch that was generated in https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/with/639482582. The original patch consisted from two fixes: explicit drop of a helper thread and additional logic for job server token request counting. This PR implements only the explicit drop part as I don't see how additional requests counting logic should help the issue. This reworks #165676, as the [rationale](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Interaction.20of.20Cargo.20jobserver.20tests.20and.20parallel.20frontend/near/639152114) behind explicit drop of a helper thread seems reasonable for me: > From my rough understanding, while helper is handling a token, its upgrade() is often the last strong ref I guess. So, when run_in_thread_pool_with_globals returns and rustc exits, the token may still be in helper's hands. This fix didn't fail after several launches in #165506 and locally. r? @petrochenkov

rust-bors · 2026-11-12T19:04:09Z

☔ The latest upstream changes (presumably #165876) made this pull request unmergeable. Please resolve the merge conflicts by rebasing.

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Nov 5, 2026

bjorn3 reviewed Nov 5, 2026

View reviewed changes

Comment thread compiler/rustc_data_structures/src/jobserver.rs Resolved

aerooneqq changed the title ~~Don't rely on used when deciding whether to release a jobserver token~~ Make Proxy weak reference as it seems that it is not dropped at all Nov 6, 2026

aerooneqq force-pushed the count-external-tokens branch from 387ff40 to 958a88e Compare November 6, 2026 10:05

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Nov 6, 2026

Auto merge of #165506 - aerooneqq:count-external-tokens, r=<try>

23ce799

Make `Proxy` weak reference as it seems that it is not dropped at all try-job: test-x86_64-gnu-aux

This comment has been minimized.

Sign in to view

petrochenkov self-assigned this Nov 6, 2026

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 6, 2026

petrochenkov reviewed Nov 6, 2026

View reviewed changes

Comment thread compiler/rustc_data_structures/src/jobserver.rs Resolved

Comment thread compiler/rustc_data_structures/src/jobserver.rs Resolved

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 6, 2026

This comment has been minimized.

Sign in to view

petrochenkov mentioned this pull request Nov 7, 2026

Use 2 parallel frontend threads by default on the nightly and dev channel #162848

Open

aerooneqq force-pushed the count-external-tokens branch from 958a88e to f652be7 Compare November 8, 2026 16:49

aerooneqq changed the title ~~Make Proxy weak reference as it seems that it is not dropped at all~~ Debugging CI failures when parallel frontend is on by default Nov 9, 2026

aerooneqq mentioned this pull request Nov 9, 2026

Make Proxy weak reference as it seems that it is not dropped at all #165676

Merged

aerooneqq added 6 commits November 12, 2026 11:18

Try LLM patch

e00c71b

Revert LLM patch

8436873

Explicitly drop helper thread

531bdcd

Try to remove not needed logging

1e76f22

Try to remove not needed logging

c2c15e6

Make 300 laucnhes

2100d63

aerooneqq force-pushed the count-external-tokens branch from 49d6cbd to 2100d63 Compare November 12, 2026 08:19

aerooneqq added 2 commits November 12, 2026 11:22

Make 100 launches

2ba6967

Update cargo

af0f5f7

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026

Auto merge of #165506 - aerooneqq:count-external-tokens, r=<try>

2560b1b

Debugging CI failures when parallel frontend is on by default try-job: test-x86_64-gnu-aux-sync

Update cargo

e3bdb8d

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026

Auto merge of #165506 - aerooneqq:count-external-tokens, r=<try>

8fac9fe

Debugging CI failures when parallel frontend is on by default try-job: test-x86_64-gnu-aux-sync

aerooneqq mentioned this pull request Nov 12, 2026

Explicitly drop job server helper thread #165854

Merged

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026

Auto merge of #165506 - aerooneqq:count-external-tokens, r=<try>

01ea506

Debugging CI failures when parallel frontend is on by default try-job: test-x86_64-gnu-aux-sync

This comment has been minimized.

Sign in to view

aerooneqq closed this Nov 12, 2026

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Nov 12, 2026

Conversation

aerooneqq commented Nov 5, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aerooneqq commented Nov 6, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Nov 6, 2026

Uh oh!

This comment has been minimized.

aerooneqq commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Nov 6, 2026

Uh oh!

petrochenkov commented Nov 6, 2026

Uh oh!

This comment has been minimized.

aerooneqq commented Nov 9, 2026

Uh oh!

aerooneqq commented Nov 12, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Nov 12, 2026

Uh oh!

aerooneqq commented Nov 12, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Nov 12, 2026

Uh oh!

aerooneqq commented Nov 12, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Nov 12, 2026

Uh oh!

rust-bors Bot commented Nov 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aerooneqq commented Nov 5, 2026 •

edited by rustbot

Loading

aerooneqq commented Nov 6, 2026 •

edited

Loading