perf(grpo): skip the pre-update policy forward when the rollout fits in one step

[RerankerGuo]

Fixes #10358.

Problem

grpo_trainer.py always runs an extra policy forward to fill old_per_token_logps. Under the default setup (num_iterations == 1, the rollout batch consumed in a single optimizer step) the pre-update policy and the current policy are the same weights, so the result is bit-for-bit identical to per_token_logps.detach() and the PPO ratio is 1 — the clip term is a no-op and the forward is pure overhead. Cost on an H200 (Qwen2.5-0.5B, one forward over the completion batch plus its logps):

batch x seqlen	tokens	time	peak
8 x 1024	8192	65.2 ms	5.80 GiB
16 x 2044	32768	109.2 ms	20.31 GiB

Fix

Skip the pre-update forward when no downstream consumer needs the real pre-update logps:

• OPD-RL (teacher KL reads old_per_token_logps),
• kl_in_reward,
• the rollout importance-sampling correction,
• the rollout off-policy diagnostic metrics.

In the skipped path old_per_token_logps stays None; the loss site already falls back to per_token_logps.detach() (which matches old_per_token_logps to the bit), so all logged metrics, the PPO clip term and the peak memory are unchanged. The ref forward at beta == 0 also short-circuits since it lives in the same with no_grad block.

The decision is factored into a module-level predicate _should_skip_pre_update_policy_forward so it can be unit-tested without pulling in the full GRPO trainer stack.

Verification

tests/general/test_grpo_skip_pre_update_forward.py (7 tests):

• default setup (num_iterations=1, no teacher, no IS) → skip;
• each of num_iterations != 1, _has_teacher, kl_in_reward, rollout_importance_sampling_mode != None, log_rollout_offpolicy_metrics → do not skip;
• combined: any single guard is sufficient to disable the skip.

The suite also reads grpo_trainer.py and asserts the normalized return expression matches the inline copy, so a maintainer who changes the predicate without updating the test will trip setUpClass before any subtest runs. All 7 pass via python -m unittest with the repo's reme-venv. flake8 / isort / yapf are clean per the repo pins.

The reporter's microbenchmarks stand: the savings match the measured pre-update forward cost exactly. The loss site fallback was already present in the loss-side reader (per_token_logps.detach() if grpo_batch.old_per_token_logps is None else grpo_batch.old_per_token_logps).