Skip to content

Integrate negotiated Heft child reporting - #6000

Open
Sean Larkin (TheLarkInn) wants to merge 175 commits into
copilot/reporter-r5b-demo-reportersfrom
copilot/reporter-r7a-heft-reporting
Open

Integrate negotiated Heft child reporting#6000
Sean Larkin (TheLarkInn) wants to merge 175 commits into
copilot/reporter-r5b-demo-reportersfrom
copilot/reporter-r7a-heft-reporting

Conversation

@TheLarkInn

Copy link
Copy Markdown
Member

Part of #5980

Stack

Parent: #5997 (copilot/reporter-r5b-demo-reporters)

This PR is based directly on the latest R5B demo tip and intentionally excludes the parallel R6 #5993 bootstrap work. Auto-merge remains disabled while stack ancestors are open.

Negotiated Heft reporting

Rush now allocates a pair of private inherited descriptors for operation children: one carries child-to-parent NDJSON events and the other returns the parent hello acknowledgement. The handshake negotiates protocol major, optional capabilities, required features, and parent-owned reporter context (reporter, log level, color, and terminal width).

Compatible Heft versions:

  • advertise heft-child-events-v1 and reporter-context-v1;
  • emit bounded externalOutput and structured diagnostics without selecting or constructing the parent's reporters;
  • preserve child session, parent request/session/operation correlation, privacy classification, and child sourceSequence;
  • propagate structured warnings back into Rush operation status calculation.

The Heft-side bootstrap uses a node-only frozen wire client rather than a runtime dependency on @rushstack/rush-reporter, avoiding a Heft/self-hosting dependency cycle. It activates only when Rush supplies both private descriptors; direct standalone Heft behavior is unchanged.

Fallback and problem matchers

Older Heft and opaque external children continue through ordered raw stdout/stderr. Missing or capability-incompatible negotiation falls back without losing output. Unsupported protocol majors or required features produce RUSH_PROTOCOL_UPDATE_REQUIRED; malformed, oversized, or truncated streams produce RUSH_PROTOCOL_INVALID_CHILD_STREAM.

Raw output is published before matcher diagnostics. Matchers consume ANSI-normalized copies keyed independently by operation, child record, and stdout/stderr stream, retain bounded partial lines, cap duplicates, and never mutate, suppress, or reorder the source output. They are disabled only after heft-child-events-v1 is actually accepted, preventing duplicate recovery when structured diagnostics exist.

Validation

  • rush build --only @rushstack/heft --only @rushstack/rush-reporter --only @microsoft/rush-lib --only @microsoft/rush --verbose
  • reporter protocol/matcher/Heft integration suite: 317 tests
  • Rush child-process cross-version fixtures: 5 tests
  • Rush operation stream/sharding/raw-order suite: 15 tests
  • Heft child bootstrap fixtures: 3 tests
  • apps/rush frontend suite: 28 tests
  • node apps/rush/src/test/sandbox/reporter-demo/run.mjs
  • rush check
  • rush change --verify --no-fetch

Coverage includes new Rush + new Heft, old/capability-less child fallback, unsupported major/required features, unknown optional and required events, malformed/truncated/oversized records, descriptor close and child crash, stdout/stderr interleaving, UTF-8 chunk limits, sourceSequence, request/session/operation correlation, privacy, matcher recovery and duplicate caps, sharded operations, warning status propagation, and parent context.

Non-goals

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Create the authoritative frontend reporter host before version selection, register global reporter controls, and preserve legacy output unless a non-legacy reporter is explicitly selected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Consume the repository experiment before Rush version selection, keep agent detection out of pre-major defaults, strip frontend-only controls before engine handoff, and preserve legacy verbosity compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn

Copy link
Copy Markdown
Member Author

Independent /pr-agentmerge review completed.

Findings fixed in 1f910c2

  • Structured Heft externalOutput bypassed Rush project logs and build-cache log chunks. Structured output now enters the archival pipeline through a no-reemit terminal provider, preserving child correlation while avoiding duplicate reporter output and problem matching.
  • Heft trusted inherited descriptor numbers and could write to or close unrelated descriptors when an intermediate launcher preserved environment variables but dropped fd mappings. Heft now requires pipe/socket descriptors before negotiation; Windows lifecycle commands explicitly retain safe raw stdout/stderr fallback because the shell launch path does not preserve Node fd mappings.
  • The parent host now rejects child-session changes, non-monotonic source sequence, malformed externalOutput payloads, and chunks above the 64 KiB protocol limit.
  • The reporter demo now accepts the valid fromCache operation terminal state when comparing rendered output with raw events.

Validation

  • rush test --only @rushstack/rush-reporter --only @microsoft/rush-lib --only @microsoft/rush: 6/6 operations passed.
  • Focused Heft HeftChildReporter suite: 4/4 passed.
  • Reporter demo passed, covering negotiated current-Heft events, parent correlation/sourceSequence order, chunk bounds, legacy rollback parity, raw grouping, and duplicate suppression.
  • rush check, rush change --verify, API Extractor, lint, and the focused post-fix review passed.
  • The full Heft suite reached four unrelated pre-existing GitUtilities.checkIgnoreAsync failures caused by this hidden worktree path; the changed Heft suite passes directly.

There are no review threads to resolve. #6000 remains stacked on open #5997 with auto-merge disabled. Merge is blocked until the ancestor chain reaches main, the PR is retargeted/rebased one step at a time with an R7-only diff, and full required CI/review gates pass.

@TheLarkInn

Copy link
Copy Markdown
Member Author

Combined deep review of current head 1f910c26e8.

  1. HIGH extra reporter descriptors are attached to every Unix shell operation. Forked grandchildren can retain the pipes and hang operation completion. Allocate descriptors only for negotiated children.
  2. MEDIUM the acknowledgement stream lacks an error handler. A closed child pipe can raise an unhandled EPIPE.
  3. MEDIUM forwarded child envelopes retain child-chosen source and privacy. Override both from trusted parent context before telemetry or reporter fan-out.
  4. MEDIUM handshake/context exceptions can escape the stream data handler and terminate Rush. Validate once and reject attachAsync() instead.

Stop reporter control scans at the pass-through separator and add an exactly-once frontend close contract across success, failure, and termination paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Keep reporter controls out of ts-command-line globals, gate incompatible engines before initialization, and enforce bounded signal and close-error behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Preserve unsupported custom reporter values until frontend ownership is unambiguous, and narrow emergency legacy stripping to the reporter selection flag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Probe reporter ownership without requiring a value, then enforce strict reporter parsing only after frontend ownership is established.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Sean Larkin (TheLarkInn) force-pushed the copilot/reporter-r5b-demo-reporters branch from e30aa36 to 7159901 Compare August 28, 2026 20:08
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Track explicit operation graph iteration identities across lifecycle events and isolate reporter totals, diagnostics, and output spools by watch cycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Track explicit operation graph iteration identities across lifecycle events and isolate reporter totals, diagnostics, and output spools by watch cycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Preserve R5B emergency sanitization, file-primary stderr ownership, and typed-sink flush behavior while retaining the four foundation regression cases.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Reporter-only subset of 1198341f54 for the posted presentation fixes. No Heft transport, demo, timers, output-observation guard, or manager drain changes are included.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve R7 protocol and child-reporting behavior while merging the corrected foundation host/parser parent without replaying historical changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Adapt a4cb979172 onto staged R5B 06bc7c5 after the shared-only human formatter lift. Preserve original host/parser changes and exclude downstream timers, wasRendered filtering, protected drain, R7 transport and initializer disposal.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Replay only the remaining R7 subset of 1198341 after corrected R5B. Preserve Windows fallback/pipe behavior and add explicit local-version/baseEnv setup to the actual old driver. The real driver and descriptor/readability regressions pass without copying integrated files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Prevent explicitly secret parameter values from reappearing through source labels, source paths or lower-classified template parameters. Preserve the original diagnostic and public context. The unchanged combined R8 corpus now passes all gates without AI, fixture or threshold changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Inherit the narrowly verified shared R5B human-context correction without changing the transport, fallback, or demo semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Sean Larkin (TheLarkInn) force-pushed the copilot/reporter-r5b-demo-reporters branch from 26c2ac0 to 26a469f Compare September 9, 2026 16:03
Sean Larkin (TheLarkInn) added a commit that referenced this pull request Sep 9, 2026
Address #6000 review 5154313652: reject corrupt accepted streams, render privacy-aware compiler context, and validate intentional Windows raw fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve published R7 cd97510 and merge only qualified R5B 8ae2115 from pinned main 67ca939. Reconcile native Heft/Rushlib private members and retained child reporting, preserve the 44-path side scope and API deltas, and exercise native scoped logging through the real child descriptor.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Sean Larkin (TheLarkInn) force-pushed the copilot/reporter-r5b-demo-reporters branch from 26a469f to 8ae2115 Compare September 10, 2026 01:30
Recover the persisted, conflict-resolved merge index into an isolated workspace after previous agent handles became unavailable. Preserve both exact parents and all prior side-specific changes; validation follows on this recovered tree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Retain the exact 44-path Heft side slice and APIs while incorporating the qualified post-main core and native Windows watch fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

1 participant