Skip to content

Avoid computing layout of enums with non-int discriminants - #157562

Merged
rust-bors[bot] merged 1 commit into
rust-lang:mainfrom
sjwang05:fix-138660-enum-discr-layout
Sep 25, 2026
Merged

rust-bors[bot] merged 1 commit into
rust-lang:mainfrom
sjwang05:fix-138660-enum-discr-layout

Conversation

@sjwang05

@sjwang05 sjwang05 commented Jun 7, 2026 •

Copy link
Copy Markdown
Contributor

View all comments

Currently, enums with explicit non-int discriminants, such as 1..=10, correctly error with E0308 and E0732 during typeck. However, layout_of never sees this error, since AdtDef::discriminants() falls back to a default value when eval_explicit_discr returns an Err, causing the layout of the enum to be broken. If this enum then appears in a promoted, we expect CTFE to fail; however, since layout_of technically "succeeded," just with a broken layout, our enum isn't in allowed_in_infallible, so CTFE expects it to succeed. CTFE failing then causes us to ICE with "interpret const eval failure of...which is not in required_consts".

This PR modifies layout_of_uncached in compiler/rustc_ty_utils/src/layout.rs to typeck all explicit enum discriminants before computing layout, and early-returns with a LayoutError::ReferncesError if eval fails. CTFE then sees this LayoutError as allowed_in_infallible, avoiding the ICE.

fixes #138660

@rustbot

rustbot commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jun 7, 2026
@rustbot

rustbot commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

r? @Kivooeo

rustbot has assigned @Kivooeo.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: compiler
  • compiler expanded to 73 candidates
  • Random selection from 20 candidates

@sjwang05

sjwang05 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

I don't expect the loop to impact perf significantly, since later on in the same function we construct an iterator by calling eval_explicit_discr via AdtDef::discriminants(), which later on in layout_of_enum gets eagerly consumed into a BTreeSet. Calling this loop now just means those const_eval_poly results are cached, hence we avoid duplicating too much work. A perf run may still be warranted though?

@Kivooeo

Kivooeo commented Jun 7, 2026

Copy link
Copy Markdown
Member

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

@rust-bors

This comment has been minimized.

@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 7, 2026
rust-bors Bot pushed a commit that referenced this pull request Jun 7, 2026
Avoid computing layout of enums with non-int discriminants
@rust-log-analyzer

This comment has been minimized.

@sjwang05
sjwang05 force-pushed the fix-138660-enum-discr-layout branch from 183e882 to 0ba35a5 Compare June 7, 2026 07:24
@theemathas

This comment was marked as resolved.

@sjwang05
sjwang05 force-pushed the fix-138660-enum-discr-layout branch from 0ba35a5 to 88d314a Compare June 7, 2026 08:51
@sjwang05

sjwang05 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for pointing that out, the previous version caused a cycle error, so now we just check the cached typeck results to see if those have errors instead of trying to const-eval the discriminant.

@rust-bors

rust-bors Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: 023d36e (023d36e8abdd31106a2c96f6803147c3dbac7b1c, parent: 61d7280f3c4c63fa24c56bdaa9a446151b5a30dc)

@rust-timer

This comment has been minimized.

@theemathas

Copy link
Copy Markdown
Contributor

This PR is still technically a breaking change, since now the following code fails to compile:

pub trait Trait {
    type Assoc;
}

impl Trait for [u8; 0] {
    type Assoc = isize;
}

pub enum Foo {
    One = unsafe { std::mem::zeroed::<<[u8; size_of::<Foo>()] as Trait>::Assoc>() },
}
Error with this PR
error[E0391]: cycle detected when type-checking `Foo::One::{constant#0}`
  --> src/lib.rs:10:45
   |
10 |     One = unsafe { std::mem::zeroed::<<[u8; size_of::<Foo>()] as Tra...
   |                                             ^^^^^^^^^^^^^^^^
   |
note: ...which requires evaluating type-level constant...
  --> src/lib.rs:10:45
   |
10 |     One = unsafe { std::mem::zeroed::<<[u8; size_of::<Foo>()] as Tra...
   |                                             ^^^^^^^^^^^^^^^^
note: ...which requires const-evaluating + checking `Foo::One::{constant#0}::{constant#0}`...
  --> src/lib.rs:10:45
   |
10 |     One = unsafe { std::mem::zeroed::<<[u8; size_of::<Foo>()] as Tra...
   |                                             ^^^^^^^^^^^^^^^^
note: ...which requires const-evaluating + checking `Foo::One::{constant#0}::{constant#0}`...
  --> library/core/src/mem/mod.rs:374:4
note: ...which requires simplifying constant for the type system `core::mem::SizedTypeProperties::SIZE`...
  --> library/core/src/mem/mod.rs:1379:4
note: ...which requires const-evaluating + checking `core::mem::SizedTypeProperties::SIZE`...
  --> library/core/src/mem/mod.rs:1379:4
note: ...which requires computing layout of `Foo`...
  --> src/lib.rs:9:1
   |
 9 | pub enum Foo {
   | ^^^^^^^^^^^^
   = note: ...which again requires type-checking `Foo::One::{constant#0}`, completing the cycle
note: cycle used when match-checking `Foo::One::{constant#0}`
  --> src/lib.rs:10:11
   |
10 | ... = unsafe { std::mem::zeroed::<<[u8; size_of::<Foo>()] as Trait>::Assoc>() },
   |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: see https://rustc-dev-guide.rust-lang.org/overview.html#queries and https://rustc-dev-guide.rust-lang.org/query.html for more information

For more information about this error, try `rustc --explain E0391`.
error: could not compile `foo` (lib) due to 1 previous error

@theemathas

Copy link
Copy Markdown
Contributor

Might be an acceptable breaking change? Not sure. Would require a crater run if we go ahead with this.

@rust-timer

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (023d36e): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This perf run didn't have relevant results for this metric.

Max RSS (memory usage)

This perf run didn't have relevant results for this metric.

Cycles

Results (primary -2.2%, secondary -3.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
-2.2% [-2.2%, -2.2%] 1
Improvements ✅
(secondary)
-3.3% [-3.3%, -3.3%] 1
All ❌✅ (primary) -2.2% [-2.2%, -2.2%] 1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 514.705s -> 515.476s (0.15%)
Artifact size: 401.25 MiB -> 400.75 MiB (-0.13%)

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 7, 2026
@sjwang05

sjwang05 commented Jun 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks, it turns out that this simpler version causes the same error with this PR for the same reason:

use std::mem::size_of;

pub enum Foo {
    One = size_of::<[u8; size_of::<Foo>()]>() as isize,
}

When type-checking One = size_of::<[u8; size_of::<Foo>()]>() as isize, we first need to resolve the type [u8; size_of::<Foo>()], which requires computing the layout of Foo in order to evaluate size_of, which triggers typeck on Foo's discriminants, which...etc.

I looked into why this doesn't cause the same query cycles on stable/nightly, and it turns out this code, with 2 variants instead of 1, doesn't compile on both current stable and nightly:

use std::mem::size_of;

pub trait Trait {
    type Assoc;
}

impl Trait for [u8; 0] {
    type Assoc = isize;
}

pub enum Foo {
    One = unsafe { std::mem::zeroed::<<[u8; size_of::<Foo>()] as Trait>::Assoc>() },
    Two,
}

stable playground link
nightly

It seems like the reason why this happens for 2+ variants but not 1 is because the discriminants() iterator only gets consumed when there are 2+ variants; when there is only 1 variant, the enum gets treated like a struct by layout computation, so its discriminants are never evaluated during layout. So looks like this kind of cycle error is a preexisting issue, just that this PR broadens the impact to univariant enums as well and probably makes it harder to fix in the future.

I do definitely think a crater run is still warranted.

@theemathas

Copy link
Copy Markdown
Contributor

Preparing for crater run.

@bors try

@rust-bors

This comment has been minimized.

rust-bors Bot pushed a commit that referenced this pull request Jun 8, 2026
Avoid computing layout of enums with non-int discriminants
@rust-bors rust-bors Bot added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 8, 2026
@rust-bors

rust-bors Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

💥 Test timed out after 21600s

@rust-bors rust-bors Bot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 8, 2026
@rust-rfcbot rust-rfcbot removed the proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. label Jul 15, 2026
@rust-rfcbot

Copy link
Copy Markdown
Collaborator

🔔 This is now entering its final comment period, as per the review above. 🔔

@traviscross traviscross removed P-lang-drag-1 Lang team prioritization drag level 1. https://rust-lang.zulipchat.com/#narrow/channel/410516-t-lang I-lang-nominated Nominated for discussion during a lang team meeting. labels Jul 15, 2026
@nikomatsakis

nikomatsakis commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

I can't imagine that which queries are triggered when and exactly which things trigger cycle errors is something that we're writing into the spec?

So I agree that queries are not part of the spec. I do think that "what requires what" at some level should be part of the spec, in so far as it accepts what programs are accepted.

@joshtriplett

Copy link
Copy Markdown
Member

@rfcbot reviewed

@tmandry

tmandry commented Jul 15, 2026 •

Copy link
Copy Markdown
Member

The way I see the FCP is that we are approving a change to the language spec, even though that section is currently imaginary (the language is underspecified by the reference). I think the simplest model is one where lang, or one of its delegated subteams, approve those changes.

This is consistent with how the types team operates, because they have a dual lang/compiler role. On the lang side, they manage a jagged subset of the design space where the only specification we have is the implementation. In creating the team, lang delegated to their best judgment how to adapt that implementation toward a smoother language over time, one we can better reason about and specify in the long run. I expect that as that continues, they will naturally start to own parts of the reference text.

@lcnr

lcnr commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

I would expect that we wind up introducing new query edges all the time in ways that might cause additional query cycles (though I'm not sure...) and doing a lang FCP every time that happens seems like a lot

I wouldn't expect to happen too often. New query edges which may cycle mean there's a cyclic dependency going on, which is valuable to look into as e.g. the fact that there's cyclic reasoning between trait solving and check_item_well_formed is actually the root of an unsoundness: #135246

It does feel more T-types than T-lang to me. Or well, more accurately, it's enough of an implementation detail that I feel kinda iffy about having T-lang be responsible here and would like it to be delegated to a subteam. Due to a lack of better choices, I guess that subteam is T-types here 🤣

@RalfJung

Copy link
Copy Markdown
Member

When I change code in const-eval I am usually not aware if whatever I call adds a new query edge, let alone if that can cause cycles.

@rust-rfcbot rust-rfcbot added finished-final-comment-period The final comment period is finished for this PR / Issue. to-announce Announce this issue on triage meeting and removed final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. labels Jul 25, 2026
@rust-rfcbot

Copy link
Copy Markdown
Collaborator

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

@sjwang05

Copy link
Copy Markdown
Contributor Author

@Kivooeo when you have the time, could you take a look at this PR again? I think it should be ready to merge, though I'm not sure. sorry for the ping, and thanks! :D

@Kivooeo

Kivooeo commented Sep 24, 2026

Copy link
Copy Markdown
Member

@bors r+ rollup

@rust-bors

rust-bors Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

📌 Commit 88d314a has been approved by Kivooeo

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 24, 2026
jhpratt added a commit to jhpratt/rust that referenced this pull request Sep 24, 2026
…out, r=Kivooeo

Avoid computing layout of enums with non-int discriminants

Currently, enums with explicit non-int discriminants, such as `1..=10`, correctly error with E0308 and E0732 during typeck. However, `layout_of` never sees this error, since `AdtDef::discriminants()` falls back to a default value when `eval_explicit_discr` returns an `Err`, causing the layout of the enum to be broken. If this enum then appears in a promoted, we expect CTFE to fail; however, since `layout_of` technically "succeeded," just with a broken layout, our enum isn't in `allowed_in_infallible`, so CTFE expects it to succeed. CTFE failing then causes us to ICE with `"interpret const eval failure of...which is not in required_consts"`.

This PR modifies `layout_of_uncached` in `compiler/rustc_ty_utils/src/layout.rs` to typeck all explicit enum discriminants before computing layout, and early-returns with a `LayoutError::ReferncesError` if eval fails. CTFE then sees this `LayoutError` as `allowed_in_infallible`, avoiding the ICE.

fixes rust-lang#138660
rust-bors Bot pushed a commit that referenced this pull request Sep 25, 2026
Rollup of 9 pull requests

Successful merges:

 - #162228 (tests: Run more pauth tests in CI and make them pass)
 - #162493 (Add support for -Zsanitizer-cfi-minimal-runtime)
 - #163271 (add a leak check test)
 - #163282 (cleanup: clean up more dependencies that are unused)
 - #157562 (Avoid computing layout of enums with non-int discriminants)
 - #162275 (Do not increase recursion depth for coroutine witness and rigid opaques when proving auto traits)
 - #163037 (fix and test `va_arg` on `f128` on `x86`)
 - #163114 (Add regression test for hang on mutually recursive trait impls)
 - #163255 (Check the entire library and cg_clif workspaces for permitted deps in tidy)
rust-bors Bot pushed a commit that referenced this pull request Sep 25, 2026
Rollup of 8 pull requests

Successful merges:

 - #162228 (tests: Run more pauth tests in CI and make them pass)
 - #163271 (add a leak check test)
 - #163282 (cleanup: clean up more dependencies that are unused)
 - #157562 (Avoid computing layout of enums with non-int discriminants)
 - #162275 (Do not increase recursion depth for coroutine witness and rigid opaques when proving auto traits)
 - #163037 (fix and test `va_arg` on `f128` on `x86`)
 - #163114 (Add regression test for hang on mutually recursive trait impls)
 - #163255 (Check the entire library and cg_clif workspaces for permitted deps in tidy)
@rust-bors
rust-bors Bot merged commit 734de5c into rust-lang:main Sep 25, 2026
13 checks passed
@rustbot rustbot added this to the 1.100.0 milestone Sep 25, 2026
rust-bors Bot pushed a commit that referenced this pull request Sep 25, 2026
Rollup merge of #157562 - sjwang05:fix-138660-enum-discr-layout, r=Kivooeo

Avoid computing layout of enums with non-int discriminants

Currently, enums with explicit non-int discriminants, such as `1..=10`, correctly error with E0308 and E0732 during typeck. However, `layout_of` never sees this error, since `AdtDef::discriminants()` falls back to a default value when `eval_explicit_discr` returns an `Err`, causing the layout of the enum to be broken. If this enum then appears in a promoted, we expect CTFE to fail; however, since `layout_of` technically "succeeded," just with a broken layout, our enum isn't in `allowed_in_infallible`, so CTFE expects it to succeed. CTFE failing then causes us to ICE with `"interpret const eval failure of...which is not in required_consts"`.

This PR modifies `layout_of_uncached` in `compiler/rustc_ty_utils/src/layout.rs` to typeck all explicit enum discriminants before computing layout, and early-returns with a `LayoutError::ReferncesError` if eval fails. CTFE then sees this `LayoutError` as `allowed_in_infallible`, avoiding the ICE.

fixes #138660
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. finished-final-comment-period The final comment period is finished for this PR / Issue. I-lang-radar Items that are on lang's radar and will need eventual work or consideration. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-lang Relevant to the language team to-announce Announce this issue on triage meeting waived-reference-pr This language change does not need a Reference PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ICE:rustc panicked at compiler\rustc_const_eval\src\interpret\eval_context.rs:555:33