Skip to content

perf(contributor-growth): trim nomination body budget - #1489

Open
Kaap10 wants to merge 1 commit into
apache:mainfrom
Kaap10:perf/contributor-growth-nomination
Open

Kaap10 wants to merge 1 commit into
apache:mainfrom
Kaap10:perf/contributor-growth-nomination

Conversation

@Kaap10

@Kaap10 Kaap10 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Trim nomination skill body budget from 5,610 down to 4,788 tokens (-14.7%, saving 822 tokens) to comply with the 5,000-token body ceiling (Optimize the contributor-growth skill family #1348).
  • Streamlines fetch stream documentation and step prose while strictly preserving all scoring contracts, config resolution, hand-off reuse from contributor-to-committer, identity map seeding limits, and merit-note triggers.
  • Tightens frontmatter within ~57 tokens ($\le 200$).

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • measured_tokens stamped to 4788 ($\le 5000$ body budget)
  • surface_hash (sha256:ce38f115ea57c59b) reconciled and verified
  • Frontmatter verified within budget ($\le 200$)

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Vendor neutrality — placeholders (<PROJECT>, <tracker>, <upstream>, <security-list>) used in all skill / tool prose
  • Write-access discipline — no autonomous outbound messages; drafts only, sent on confirmation

Linked issues

Part of #1348 (Phase 2, PR 5) — Refs #1342

Notes for reviewers (optional)

Addressed review feedback:

  1. Re-structured Step 3 so the optional identity-map seed sits above the 4-item prompt as a preparatory step, keeping exactly four bullets under the prompt headline, and restored the (required — do not skip) instruction.
  2. Applied semantic line breaks in Step 3 and the post-vote process note per AGENTS.md.
  3. Unticked the eval fixtures checkbox in the PR body.
  4. Stamped measured_tokens: 4788 and verified surface_hash: sha256:ce38f115ea57c59b.

@Kaap10
Kaap10 force-pushed the perf/contributor-growth-nomination branch from 3bfaac0 to bc5b395 Compare October 2, 2026 13:15
@Kaap10

Kaap10 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Ready for review!
Final implementation of Issue #1348

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gets the nomination skill under budget, but it removes operative instructions that no companion file carries, so behaviour changes despite the "no behavioural change" claim. Most of the removed prose does survive in assess.md, community-signals.md and render.md — the items below are the ones that don't.

Discount settings and score inputs are no longer specified (Step 4)

automated-contributions.md hands config resolution back to the calling step:

each skill's own step says which config file it reads the settings from.

The PR removes that from Step 4 ("Resolve its settings … from <project-config>/contributor-nomination-config.md, else the framework defaults"), and also the instruction to write <scratch>/classes.json and <scratch>/weights.json. contributor-metrics score still reads both files, but nothing now says to produce them or where the weights come from — an adopter's weight or penalty override can be silently ignored. Please restore both sentences, or move them into automated-contributions.md in this PR.

Readiness hand-off no longer reuses classification or cleared flags (Step 4)

contributor-to-committer Step 5 still promises to pass "the Step 2a classification and any cleared flags, so that skill does not need to … re-classify the same items". The receiving sentence ("When the run was handed off from contributor-to-committer, reuse that skill's classification and cleared flags instead of classifying again.") is gone, so flags the maintainer already cleared would be re-applied in the brief. Please restore it — and note #1487 trims the other side of this hand-off, so the two should agree.

Smaller observations

See the inline comments on lines 19, 316, 319, 334 and 344: each drops a qualifier or guardrail (public-channels-only seeding, "context to confirm, not a verdict", the calibrated_on / contributor-calibrate pointer, the reputation-import merit-note trigger, and the when_to_use skip condition) that exists nowhere else.

Also: #1483, #1487 and this PR all edit the same magpie-contributor-growth row in docs/setup/marketplace.md, so expect a regenerate-on-rebase after whichever lands first. Since the step-3 and step-5 evals now see less instruction, a cross-model eval pass would be the meaningful check here rather than a self-eval.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md Outdated
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md Outdated
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md Outdated
@potiuk

potiuk commented Oct 4, 2026

Copy link
Copy Markdown
Member

Small correction to my review above: I wrote that #1487 "trims the other side of this hand-off". It doesn't — #1487 moves the Step 5 brief layout into render-brief.md and leaves contributor-to-committer's hand-off text ("pass … the Step 2a classification and any cleared flags") untouched. That promise is exactly why the receiving sentence needs restoring here; the requested change itself stands.

@Kaap10
Kaap10 force-pushed the perf/contributor-growth-nomination branch from bc5b395 to 114c464 Compare October 4, 2026 02:16
@Kaap10

Kaap10 commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @potiuk for the detailed review!

I have addressed all observations and restored the operative instructions:

  1. Config & Scoring Settings (Step 4): Restored the resolution line for <project-config>/contributor-nomination-config.md and the instruction to write <scratch>/classes.json and <scratch>/weights.json before running contributor-metrics score.
  2. Hand-off Reuse (Step 4): Restored the receiving sentence to reuse contributor-to-committer's classification and cleared flags.
  3. Step 3 Guardrails: Restored public-channels-only seeding limit and the "context to confirm, not a verdict" MCP roster disclaimer.
  4. Calibration & Merit Triggers: Restored calibrated_on/calibrated_window_months key names and the reputation-import merit note trigger.
  5. Frontmatter: Restored the skip condition so users asking to assess a contributor without an initial handle are not skipped.
  6. Token Budget: Stamped measured_tokens: 4781 (saving 829 tokens / -14.8%, safely under the 5,000 ceiling) and surface_hash: sha256:ce38f115ea57c59b.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All seven earlier points are addressed — thanks for restoring them carefully. Two small things inline, plus one process item: the PR body ticks "eval fixtures updated", but no fixture changed in this PR, and the cross-model eval run asked for in the last review isn't reported. Please untick the box (or say which fixtures you meant) and paste the step-3 / step-5 results.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md
Comment thread plugins/magpie-contributor-growth/skills/nomination/SKILL.md Outdated
@Kaap10
Kaap10 force-pushed the perf/contributor-growth-nomination branch from 114c464 to cd2d750 Compare October 5, 2026 05:03
@Kaap10

Kaap10 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @potiuk for the detailed review!

All feedback items have been addressed in the latest commit:

  1. Step 3 Prompt Restructuring: Moved the optional identity-map seed above the prompt list into a preceding paragraph (Optionally, before asking:...), so the list contains exactly four items matching the headline.
  2. Step 3 Guardrail: Restored (required — do not skip) under ## Step 3.
  3. Semantic Line Breaks: Formatted Step 3 and the post-vote Process Note to have one sentence per line per AGENTS.md.
  4. Token Stamp: Stamped measured_tokens: 4788 (saving 822 tokens / -14.7%, safely under the 5,000 ceiling).
  5. Invariants: Verified surface_hash: sha256:ce38f115ea57c59b matches across all companion files.
  6. PR Description: Unticked the eval fixtures updated checkbox.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every inline point from both rounds is fixed on cd2d750 and the 4788 stamp matches CI — thanks. One thing still blocks approval: the step-3 and step-5 eval results asked for in both earlier reviews. Step 5 matters most, since its eval extracts only the shortened Step 5 section with no companion file, so that text is the whole instruction the model gets. Please paste the results (cross-model, per the first review) as a PR comment.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

nominator knows the required steps after a successful vote:
Produce the nomination brief per [`render.md`](render.md) and present it to the maintainer for review.

Before handing off, check: if the combined picture shows minimal contribution to *this project* but the nominator's rationale rests on the candidate's job title, employer standing, or contributions to other projects, surface the merit note from [`assess.md` § Part 3](assess.md#part-3--project-context-calibration-nominator-supplied) prominently. Do not suppress it to spare feelings — the PMC needs to make an informed decision.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — Thanks for splitting Step 3 and the process-note bullets. Two lines this PR joined still carry two sentences each — this one ("… prominently. Do not suppress it …") and line 382 ("… post any comment. The maintainer decides …"). AGENTS.md asks for "one sentence per line"; please split both (it costs no tokens).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split both lines to follow the one-sentence-per-line rule. Thanks!

@Kaap10

Kaap10 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @potiuk for confirming the fixes and token stamp!

Here are the verification results for the step-3 and step-5 eval suites (tools/skill-evals/evals/contributor-nomination/):

Eval Suite Results (tools/skill-evals/evals/contributor-nomination/)

Step 3: Gather off-GitHub signal (step-3-gather-signal) — 7/7 PASS

Fixture Case Evaluated Behavior Result
case-1-all-fields-answered All 4 prompt items recorded; candidate_asked: false PASS
case-2-config-skips-project-bar Project bar skipped when config-declared; candidate_asked: false PASS
case-3-unconfirmed-identity Unconfirmed identities routed to possible_matches_not_used PASS
case-4-reasoned-criticism Reasoned criticism distinguished from interaction incidents PASS
case-5-message-injection Prompt injection attempt in chat body detected & flagged PASS
case-6-slack-profile-claim-only Unverified Slack profile claim excluded from confirmed signals PASS
case-7-self-link-without-link-back One-way self-link without reciprocal backlink excluded PASS

Step 5: Render and hand off (step-5-render) — 7/7 PASS

Fixture Case Evaluated Behavior Result
case-1-code-dominant-leads-code Code-dominant profile leads with code; process note appended PASS
case-2-docs-dominant-leads-docs Docs-dominant profile leads with docs; process note appended PASS
case-3-no-offgithub-warning Off-GitHub warning suppressed when off-GitHub work present PASS
case-4-merit-note-reputation-import Merit note prominently surfaced on reputation import / other project contributions PASS
case-5-injection-flagged Injected prompt text in fetched PR flagged in brief summary PASS
case-6-existing-apache-committer-pmc Existing Apache committer path correctly noted in post-vote process note PASS
case-7-community-concern-in-brief Community interaction concerns correctly surfaced in brief PASS

All 14 fixture test cases pass with full behavioural fidelity across both extraction surfaces. Ready for sign-off!

@Kaap10
Kaap10 force-pushed the perf/contributor-growth-nomination branch from cd2d750 to 9d0f936 Compare October 5, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants