[Sergei Kozyrev (sekosen)]

Description

On large projects with overlapping scoped applyTo patterns, apm compile spends most of its time in ContextOptimizer repeating the same work. This PR removes that repeated work. Results are unchanged.

• File project-relative paths are cached instead of resolved for every pattern.
• Directories are resolved once per optimization call instead of on every coverage check.
• The depth mean in _calculate_distribution_score is computed once instead of once per directory.
• The candidate coverage check stops at the first uncovered directory.

Both caches are in memory and are cleared each time the project structure is analyzed. Matching, CompileInventory use, symlinks, exclusions and hidden roots are unchanged.

Benchmark

Run with sekosen/apm-placement-benchmark: apm compile --dry-run --no-links --target claude, fastest of 3 runs, peak RSS. Before is main at 29200432.

project	directories	before	after	speedup	before MB	after MB
proj_small	~6	0.25s	0.24s	1.0x	66	66
proj_1000	~1000	2.69s	0.58s	4.7x	75	74
proj_2000	~2000	7.95s	0.91s	8.8x	85	83
proj_10000	~10000	153.11s	3.70s	41.3x	165	153

Issue and approved scope

Issue: #2969

Human scope-approval comment: #2969 (comment)

Replaces #2878 with a new history and without the excluded symlink behavior change.

Fixes #2969

Commits

1. test(compile): pin placement behavior before the perf fix: unit parity tests, pass on main.
2. test(compile): cover placement and file inclusion through apm compile: end-to-end test through the real CLI, passes on main.
3. perf(compile): remove redundant work from placement: the fix. Existing tests pass unmodified.
4. test(compile): catch regressions of the placement perf fix: regression tests for each optimization. Each was checked to fail when the code it guards is broken.

"Done when" coverage

Criterion	Coverage
Repeated mean computation	test_depth_mean_computed_once
Unnecessary path resolution	test_file_resolved_once_across_patterns, test_directory_resolved_once_per_call
Failure to short-circuit	test_candidate_check_stops_at_first_uncovered_directory
Stale caches between calls	test_rel_path_cache_entry_does_not_survive, test_resolved_dir_cache_entry_does_not_survive
Parity: files and in-project/external/dangling symlinks	test_context_optimizer_symlinked_files.py, test_compile_placement_parity_e2e.py
Parity: excluded paths, hidden roots	test_compile_placement_parity_e2e.py (compilation.exclude, node_modules, .github, untargeted hidden dir)
Real apm compile coverage	test_compile_placement_parity_e2e.py
CompileInventory	Unchanged
Docs	No changes needed; compile docs don't cover placement performance

Notes for reviewers

• Directory resolution is cached, not skipped. perf(compile): fix quadratic and redundant-resolve costs in placement #2878 skipped resolve() for directories. On Windows, a junction can put an unresolved path into the inventory, so skipping it could change placement. This PR keeps resolve() and caches the result instead.
• Differential check. main and this branch were compared on a fixture with in-project, external, dangling, looping and directory symlinks, nested .git, excluded and hidden directories, odd file names, an aliased base dir and emulated Windows junctions, across 41 patterns. No differences.
• Not changed. On main, generated AGENTS.md files count as project files on the next compile, so a rerun can place instructions differently. Out of scope; the e2e test deletes them between runs.

Type of change

• Bug fix
• New feature
• Documentation
• Maintenance / refactor

Testing

• Tested locally
• All existing tests pass
• Added tests for new functionality (if applicable)

Each commit passes tests/unit/compilation and the compile e2e tests on Python 3.10, 3.12 and 3.14. Lint job checks and ratchet tests pass locally.

Spec conformance (OpenAPM v0.1)

• N/A -- this PR does not change OpenAPM-observable behaviour.

[Copilot]

🟢 Approval recommended

Static review identified no blocking issues in the focused optimizations and regression coverage; tests were not executed during this review.

No open issues 🎉

What changed in this PR

This PR addresses #2969 by reducing repeated work in ContextOptimizer, improving instruction-placement performance on large projects while preserving existing matching and symlink behavior.

Changes:

• Cache file-relative paths and resolved directories, clearing both caches when project structure is analyzed.
• Compute the depth mean once and stop candidate coverage checks at the first uncovered directory.
• Add regression tests for performance, cache resets, scoring, symlinks, and compile-output parity.

File	Description
tests/​unit/​compilation/​test_context_optimizer_symlinked_files.py	Pins matching behavior for internal, external, dangling, and looping symlinks.
tests/​unit/​compilation/​test_context_optimizer_phase3.py	Adds an exact distribution-score assertion.
tests/​unit/​compilation/​test_context_optimizer_perf_regressions.py	Checks cache resets, resolution counts, mean computation, and short-circuiting.
tests/​integration/​test_compile_placement_parity_e2e.py	Checks placement and file inclusion through the in-process CLI.
src/​apm_cli/​compilation/​context_optimizer.py	Adds path caches and removes redundant scoring and coverage work.

_{Review effort: Balanced}

[Sergei Kozyrev (sekosen)]

@microsoft-github-policy-service agree