Skip to content

m3front: define the static link on both arms of a closure call; cm3: pass debug/optimize to the C back end (fixes #1243) - #1244

Open
mikan-cerebras wants to merge 2 commits into
modula3:masterfrom
mikan-cerebras:fix-undefined-static-link
Open

mikan-cerebras wants to merge 2 commits into
modula3:masterfrom
mikan-cerebras:fix-undefined-static-link

Conversation

@mikan-cerebras

@mikan-cerebras mikan-cerebras commented Oct 28, 2026 •

Copy link
Copy Markdown

GenClosureCall emits Pop_static_link only inside the arm taken when the procedure value is a closure, but the indirect call after the join reads the static link on both arms. When the value is a top-level procedure the static link is therefore read having never been written.

m3-sys/m3front/src/types/UserProc.m3, before:

CG.If_closure (p_temp, CG.No_label, skip, CG.Maybe);
CG.Push (p_temp);
CG.Closure_frame ();
CG.Pop_static_link ();      (* the only write -- inside the branch *)
CG.Push (p_temp);
CG.Closure_proc ();
CG.Store_temp (p_temp);
CG.Set_label (skip);        (* join *)
CG.Push (p_temp);
...
CG.Gen_Call_indirect (result, cc);   (* reads the static link on both arms *)

The value really is a don't-care for a top-level procedure, since the callee ignores it. But don't care and undefined are not the same thing to an optimizer.

What goes wrong

With the C back end and clang -O2, the never-written local becomes undef:

%20 = phi ptr [ undef, %3 ], [ %15, %13 ]

undef means "any value I like", so the two arms become interchangeable and SimplifyCFGPass folds the branch, keeping the CL_marker comparison only as an assumption:

%12 = icmp eq i64 %11, -1
call void @llvm.assume(i1 %12)     ; <- was a branch
; CL_proc / CL_frame loads now unconditional

After that, every suitably aligned non-NIL procedure value takes the closure path, loads CL_proc and CL_frame out of the target function's own instruction stream, and branches to the result. The resulting pc decodes to a real instruction — in our case stp x29, x30, [sp, #0x16], the prologue of the very function that should have been called.

Bisected with -opt-bisect-limit: pass #132, SimplifyCFGPass.

Why it is intermittent

The "not 8-aligned, so not a closure" pre-check survives the fold, so whether a program crashes depends on where the linker puts the target procedure. Four variants of one test differing only in a record's field count:

int fields target proc address & 7 result
1 0x100001244 4 clean
2 0x100001288 0 crash
4 0x1000012f0 0 crash
18 0x1000015bc 4 clean

Adding an unrelated field can move a program between the two.

Why the fix is here and not in the C back end

M3C.m3's loadBeforeStore analysis would normally emit ={0} for a read-before-write temp, but it is branch-insensitive by design — its own comment says "This knowingly imagines the IR runs in order, without branches around." Patching it would fix one back end. M3CG_LLVM and M3x86 have no equivalent mitigation at all, and the undefined read originates in the front end regardless.

Note also that Pop_static_link spills the stack top to a fresh temporary on each call, so simply calling it again before the branch does not work — it allocates a second temp and the first store is dead. The fix uses one temp, written on both arms, popped once after the join.

Testing

ARM64_DARWIN, C back end, Apple clang.

  • A 27-line reproducer that passes a procedure as a parameter crashes at -O2 before and prints the correct answer after, with the target procedure forced to 8-byte alignment so the failing arm is the one exercised.
  • Nested procedures still get the correct static link. A closure reading a captured local returns identical values under patched -O2 and stock -O1 (1407, 407), so the environment is still delivered correctly — this was the obvious regression risk.
  • A ~20k-line numerical model builds at -O2 and reproduces its reference output bit-for-bit (99,999 rows, matching SHA-256), and a checkpoint-restart that segfaulted at -O2 before now completes.
  • -O1 behaviour is unchanged before and after.

Related, but not this bug

The CL_marker test itself dereferences a function pointer as data, which is undefined in C (a function is not an object, so there is nothing there to read). That is worth addressing separately, but it is not what is happening here: under opaque pointers LLVM cannot tell the pointer designates a function, and a hand transcription of the dispatch sequence into plain C does not reproduce at any optimization level. The undef is the whole trigger.


Second commit: pass the unit's debug/optimize flags to the C back end

Added after the above, because it is what makes -O2 reachable for the C back end in the first place — and therefore what exposes the bug fixed in the first commit.

Builder.m3 hardcoded TRUE, FALSE for debug, optimize at the DoRunC call site:

IF ok AND DoRunC THEN
  ok := RunCC (s, CCodeName, u.object, TRUE, FALSE, s.include_path_empty);
END;

so with the C back end, m3_optimize in an m3makefile and -O on the command line were silently ignored — every build was -O0 with debug forced on. Every other back end on the same path already passes u.debug, u.optimize, as do the other two RunCC call sites in the same file. This fixes #1243.

Measured on ARM64_DARWIN with a config carrying the current Unix.common:

flag seen by compile_c args to cc Callee_m.o
before optimize= -g -O0 f4e0ea4a…
after optimize=TRUE -g -O2 ec98460b…

_m3main.cpp is unaffected — it goes through a separate call at the bottom of the file that passes debug := TRUE, optimize := FALSE deliberately, and still reports optimize= after the change.

Please take these together, or this one second

Landing this commit alone turns on -O2 for the C back end and will produce programs that call procedure parameters through a word read out of their own instruction stream, on whichever procedures the linker happens to place at a multiple of 8. The static-link fix is what makes -O2 safe here. With both commits applied, the same test program compiles at -O2 and produces correct results for both top-level and nested procedures.

Happy to split this into two PRs if you'd prefer to review them separately — the ordering constraint is the only reason they are together.

GenClosureCall emitted Pop_static_link only inside the arm taken when
the procedure value is a closure, while the indirect call after the
join reads the static link on both arms.  For a top-level procedure
the static link was therefore read having never been written.

The value is genuinely a don't-care there -- the callee ignores it --
but "don't care" and "undefined" are not the same thing to an
optimizer.  With the C back end and clang -O2 the never-written local
becomes undef, which lets SimplifyCFG treat the two arms as
interchangeable and fold the closure test away: the CL_marker
comparison is demoted to llvm.assume, and every suitably aligned
non-NIL procedure value then takes the closure path.  It loads
CL_proc and CL_frame out of the target function's own instruction
stream and branches to the result.

Whether a given program crashes depends on where the linker places
the target procedure, because the "not 8-aligned, so not a closure"
pre-check survives the fold.  A procedure landing at 4 mod 8 escapes
through it and is called correctly; one landing at 0 mod 8 is called
through its own code.  Adding an unrelated field to a record can move
a program between the two.

Pop_static_link spills the stack top to a fresh temporary each time
it is called, so calling it a second time before the branch does not
help -- it allocates a second temp and the first store is dead.  This
uses one temp, written on both arms, popped once after the join.

Fixed in the front end rather than in the C back end's
loadBeforeStore analysis.  That analysis is branch-insensitive by
design ("This knowingly imagines the IR runs in order, without
branches around") and so does not mark this temp, but more
importantly M3CG_LLVM and M3x86 have no equivalent mitigation at all,
and the undefined read is the front end's to begin with.

Tested on ARM64_DARWIN, C back end:

  - a 27-line reproducer passing a procedure as a parameter crashes
    at -O2 before, prints the right answer after, with the target
    procedure forced to 8-byte alignment so the arm is the failing one
  - nested procedures still receive the correct static link: a closure
    reading a captured local returns identical values under patched
    -O2 and stock -O1
  - a 20k-line numerical model builds at -O2 and reproduces its
    reference output bit-for-bit, and a checkpoint-restart that
    segfaulted at -O2 before now completes
  - -O1 was unaffected throughout, before and after

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builder.m3 hardcoded TRUE, FALSE for debug and optimize at the DoRunC
call site, so with the C back end m3_optimize in an m3makefile and -O
on the command line were silently ignored: every build was compiled
-O0 with debug forced on, whatever the m3makefile asked for.

Every other back end on the same path already passes u.debug and
u.optimize, as do the other two RunCC call sites in the same file.
Only this one did not.

The flag does reach compile_c once this is fixed, measured on
ARM64_DARWIN with a config carrying the current Unix.common:

  before:  optimize=      args=-g -O0
  after:   optimize=TRUE  args=-g -O2

and the resulting objects differ, as expected.

_m3main.cpp is unaffected: it is compiled through a separate call at
the bottom of this file that passes debug := TRUE, optimize := FALSE
deliberately, and still reports optimize= after this change.

ORDERING NOTE: this change is what exposes the undefined static link
fixed in the preceding commit.  Taking this one alone turns on -O2
for the C back end and will produce programs that call procedure
parameters through a word read out of their own instruction stream,
depending on where the linker places the target.  The two belong
together, or this one belongs after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mikan-cerebras mikan-cerebras changed the title m3front: define the static link on both arms of a closure call m3front: define the static link on both arms of a closure call; cm3: pass debug/optimize to the C back end (fixes #1243) Oct 28, 2026
@VictorMiasnikov

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C backend has code generation optimization disabled

2 participants