ProAS: A Progress-Aware Advantage Shaping Framework For Multi-Turn Agentic Reinforcement Learning

Calibrating the Credit Assignment of Group-base RL Algorithms for Effective and Efficient Agentic Training.

📑 Paper   

---

📚 Contents

• Introduction
  • Motivation
  • Method
• Getting Started
  • ALFWorld
  • WebShop
  • Verl
• Usage
  • Data
  • Models
  • Codes
• Results
  • Main Results
  • Ablation Study
• Acknowledgements

---

📖 Introduction

Group-based RL algorithms such as GRPO and DAPO assign the same advantage to every step of a trajectory. This uniform credit assignment reinforces unreasonable actions in successful trajectories and penalizes every action in a failed trajectory equally, regardless of how much it contributed to the failure. ProAS (Progress-Aware Advantage Shaping) calibrates this credit with step-level progress signals from a dynamically evolving Agentic Reference Graph (ARG).

Motivation

Two findings on GRPO with Qwen2.5-1.5B-Instruct:

• Unreasonable actions get reinforced, and they hurt. Unreasonable actions, i.e., actions that make an invalid state transition such as an explicit error, make up 41.6% / 39.6% of successful trajectories on ALFWorld / WebShop. Injecting a 4-step unreasonable segment into a clean ALFWorld context drops the expected success rate (ESR) from 82% to 58% (panel (b) below).
• Uniform penalties overlook how much each step drifts the trajectory. Non-advancing (NA) actions, i.e., actions that fail to reach a new valid state, dominate failed trajectories (93.1% / 76.1%), and late ones are often fallout from an already polluted context. We measure this with Future-Drift (FD), the number of NA actions from step $t$ to the end of trajectory $\tau_i$: $\mathcal{D}{i,t} = \sum{t'=t}^{\lvert\tau_i\rvert} (1 - \text{PA}{i,t'})$, where $\text{PA}{i,t'} = 0$ marks an NA step. Suppressing actions with FD = 47 / 23 / 11 / 4 recovers ESR of 24% / 16% / 10% / 0% (panel (c) below), so high-FD actions should be suppressed first.

_{(a) The demonstration of ARG. Diverse successful trajectories are unified into a DAG that serves as a progress reference. (b) Probing unreasonable actions. From an identical state, the counterpart injected with unreasonable actions yields a lower Expected Success Rate (ESR) on 50 sampled continuations. (c) Suppressing by Future-Drift (FD). Forcibly suppressing the original action at different trajectory points and resampling 50 continuations shows recovered ESR rises with the suppressed action's FD.}

Method

ProAS reshapes the trajectory-level advantage $A_i$ into a step-level advantage $\tilde{A}_{i,t}$:

$$\tilde{A}_{i,t} = \begin{cases} \text{Rea}_{t} \cdot A_i, & \text{if } A_i > 0 \quad \text{(MUI)}, \\\ \dfrac{\mathcal{D}_{i,t}}{\bar{\mathcal{D}}_i + \epsilon} \cdot A_i, & \text{if } A_i < 0 \text{ and } t \in \mathcal{T} \quad \text{(APR)}, \\\ A_i, & \text{otherwise.} \end{cases}$$

• Masking Unreasonable Incentives (MUI): in preferred trajectories ($A_i &gt; 0$), unreasonable steps ($\text{Rea}_t = 0$) get zero advantage, so they are no longer reinforced.
• Asymmetric Penalty Redistribution (APR): in unpreferred trajectories ($A_i &lt; 0$), each step's penalty is scaled by its Future-Drift $\mathcal{D}_{i,t}$ relative to the mean $\bar{\mathcal{D}}_i$ over the uniformly credited steps $\mathcal{T}$, so early actions are penalized more than late, context-induced ones.

---

✨ Getting Started

ProAS is built on top of verl-agent. Before training, set up the two agent environments below, ALFWorld and WebShop.

⚠️ Important: Each environment must be installed and configured before an agent can run in it. To avoid package version conflicts, we recommend giving each environment its own conda environment.

1. ALFWorld

Install the dependencies:

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip install alfworld

Fetch the PDDL and game files together with the pre-trained MaskRCNN detector, which are saved to ~/.cache/alfworld/. Add --extra if you also need the pre-trained checkpoints and seq2seq data.

alfworld-download -f

To check the installation, launch an interactive TextWorld game:

alfworld-play-tw

2. WebShop

WebShop needs Python 3.10 or lower, so create a dedicated environment first:

conda create -n verl-agent-webshop python==3.10 -y
conda activate verl-agent-webshop

Run the WebShop setup script from the repository root:

cd verl-agent/agent_system/environments/env_package/webshop/webshop
./setup.sh -d all

If gdown fails to download the data, copy your Google Drive cookie from https://drive.google.com/ into ~/.cache/gdown/cookies.txt, or download the files manually.

3. Verl

Then go back to verl-agent/ and install verl together with its dependencies:

cd ../../../../..   # back to verl-agent/
pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation
pip3 install -e .
pip3 install vllm==0.8.2

pip may report that spacy and weasel require typer<0.10.0 while typer 0.15.2 is installed. These conflicts are harmless and can be ignored.

---

🔧 Usage

We adopt the non-intrusive development on the verl-agent for ProAS. Data, models, codes, and scripts are saved in seperate folders.

Data

All fixed task data lives under verl-agent/recipe/graph_guide/data/. We train for 150 steps with 16 tasks per step for group-based methods and 128 task slots per step for PPO, and validate on 128 tasks every 10 steps. To make runs reproducible, both the training tasks and the validation tasks are fixed in advance:

• verl-agent/recipe/graph_guide/data/fixed_train_set/ holds training schedules for seed{0,1,2}/{alfworld,webshop}. Each schedule.jsonl lists the 16 tasks for each of the 150 steps, taken from the official GRPO rollouts in the same order. Each manifest.json records where the schedule came from, its file hashes, and a check that it shares no tasks with the validation set. Enable a schedule with env.fixed_train_set.path=recipe/graph_guide/data/fixed_train_set/seed0/alfworld; if the option is left unset, training samples tasks online as before.
• verl-agent/recipe/graph_guide/data/fix_train_set_ppo/ holds the PPO schedules for the same seeds and environments, with 150 batches of 128 task slots. Enable a PPO schedule with env.fixed_train_set.path=recipe/graph_guide/data/fix_train_set_ppo/seed0/alfworld.
• verl-agent/recipe/graph_guide/data/fixed_val_set/ holds 128 unique validation tasks per environment, none of which appear in training. ALFWorld tasks come from the official valid_seen split (alfworld.jsonl, alfworld_gamefiles.txt); WebShop tasks come from the benchmark's evaluation range [500, 1500) (webshop.jsonl). Enable it with env.fixed_val_set.enabled=true and keep data.val_batch_size=128. The default validation root is recipe/graph_guide/data/fixed_val_set, relative to verl-agent/.

Models

Our experiments are performed on Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. Download these models to verl-agent/recipe/graph_guide/models from huggingface:

• Qwen2.5-1.5B-Instruct

• Qwen2.5-7B-Instuct

Codes

ProAS is implemented as a single recipe, verl-agent/recipe/graph_guide/. It hooks into the verl-agent trainer without changing the PPO loop: the base estimator computes advantages as usual, and ProAS then reshapes them with ARG annotations. Each part of the recipe has its own README:

• graph_guide/: Training entry points and the core credit module, graph_guide_credit.py, which annotates every step against the ARG and applies MUI and APR. The README also maps the paper's notation to the code.
• graph_guide/oracle_build/: ARG construction and environment adapters for ALFWorld and WebShop, plus the prebuilt graphs, including the Incomplete and Distill variants.
• graph_guide/run_sh_fair/: Launchers for ProAS, the RL baselines, and every ablation, organized by model size, seed, and environment.
• graph_guide/sft_baseline/: SFT baseline trained on oracle trajectories of the same training tasks.
• graph_guide/react_infer/: ReAct prompting baseline with a proprietary API model on the fixed validation set.

Outside the recipe, the fixed training and validation sets are loaded by verl-agent/agent_system/environments/fixed_training.py and fixed_validation.py, and the MiGPO baseline lives in verl-agent/recipe/migpo/.

📈 Results

All results are the mean ± std over seeds 0/1/2, taken from the best validation checkpoint within 150 RL steps. ALFWorld reports success rate (SR, %) per task type and overall; WebShop reports task score (TS) and SR. Spd.@K is the speedup of ProAS over its base algorithm, measured as the ratio of RL training steps each needs to first reach K% SR (K = 70 for ALFWorld, 50 for WebShop).

Main Results

ProAS improves the overall ALFWorld SR and the WebShop TS of GRPO, DAPO, and GiGPO at both model sizes, and reaches the target SR 1.1×–2.4× faster. The best results are 94.3% SR on ALFWorld (GiGPO+ProAS, 7B) and 70.8% SR on WebShop (GRPO+ProAS, 7B).

Method	ALFWorld								WebShop
	Pick	Look	Clean	Heat	Cool	Pick2	All	Spd.@70	TS	SR	Spd.@50
Proprietary Model
GPT-5.6-Sol (ReAct)	90.9	83.3	68.0	83.3	79.2	68.2	78.9	–	27.6	21.9	–
Qwen2.5-1.5B-Instruct
Direct Prompt	24.2±0.0	25.0±0.0	4.0±0.0	8.3±0.0	0.0±0.0	4.5±0.0	10.9±0.0	–	25.3±0.0	4.7±0.0	–
SFT	99.0±1.7*	91.7±8.3	94.7±2.3	100.0±0.0*	93.1±2.4*	1.5±2.6	79.7±0.8	–	78.2±1.7	64.3±3.0	–
PPO	82.8±7.6	58.3±8.3	80.0±4.0	75.0±8.3	76.4±6.4	54.5±9.1	73.2±4.3	–	75.3±11.1	55.2±6.6	–
MiGPO	97.0±3.0	91.7±0.0	98.7±2.3*	100.0±0.0*	84.7±6.4	75.8±2.6	91.1±1.6	–	79.5±1.2	53.6±1.8	–
GRPO	80.8±4.6	69.4±4.8	70.7±6.1	75.0±8.3	63.9±9.6	59.1±9.1	70.3±2.8	–	82.4±1.3	57.0±3.1	–
GRPO+ProAS	90.9±5.2	86.1±4.8	90.7±2.3	91.7±8.3	83.3±4.2	78.8±5.2	87.0±2.3	1.59×	85.3±2.4	64.6±1.2	1.79×*
DAPO	89.9±1.7	83.3±0.0	89.3±12.2	88.9±4.8	75.0±8.3	66.7±2.6	82.3±5.0	–	79.2±2.5	57.3±4.6	–
DAPO+ProAS	93.9±3.0	86.1±9.6	84.0±6.9	94.4±4.8	83.3±4.2	77.3±4.5	86.5±1.2	1.27×	87.2±0.7	66.7±1.2*	1.61×
GiGPO	91.9±1.7	83.3±0.0	94.7±4.6	94.4±4.8	84.7±2.4	89.4±2.6*	90.1±0.5	–	83.9±0.4	64.3±3.9	–
GiGPO+ProAS	94.9±1.7	94.4±4.8*	93.3±2.3	97.2±4.8	88.9±2.4	84.8±2.6	91.9±0.5*	1.61×*	87.5±1.7*	66.7±3.2*	1.12×
Qwen2.5-7B-Instruct
Direct Prompt	35.4±3.5	33.3±8.3	13.3±6.1	5.6±4.8	9.7±2.4	3.0±5.2	17.7±0.9	–	3.4±1.3	0.5±0.5	–
SFT	100.0±0.0*	88.9±4.8	96.0±4.0	100.0±0.0*	93.1±2.4*	1.5±2.6	79.9±0.5	–	81.9±0.7	70.1±0.9	–
PPO	97.0±3.0	72.2±4.8	89.3±8.3	94.4±4.8	77.8±6.4	63.6±12.0	83.6±4.1	–	84.1±4.1	66.7±4.3	–
MiGPO	100.0±0.0*	94.4±4.8	96.0±4.0	80.6±4.8	84.7±2.4	87.9±2.6	91.9±1.2	–	88.0±2.2	67.2±3.1	–
GRPO	90.9±0.0	72.2±9.6	92.0±6.9	77.8±17.3	76.4±4.8	63.6±4.5	80.7±3.9	–	85.6±5.2	67.7±5.5	–
GRPO+ProAS	97.0±3.0	91.7±8.3	94.7±2.3	86.1±4.8	77.8±2.4	83.3±6.9	89.1±2.3	2.20×	89.0±0.6	70.8±0.9*	1.91×*
DAPO	97.0±3.0	83.3±14.4	96.0±4.0	88.9±9.6	76.4±6.4	84.8±6.9	88.8±5.2	–	87.1±3.6	68.8±4.7	–
DAPO+ProAS	97.0±3.0	86.1±12.7	97.3±4.6	88.9±9.6	83.3±8.3	78.8±5.2	89.6±2.0	2.05×	88.2±2.8	70.6±0.9	1.62×
GiGPO	97.0±3.0	100.0±0.0*	97.3±2.3	97.2±4.8	87.5±0.0	86.4±9.1	93.8±2.1	–	88.3±1.8	70.3±1.6	–
GiGPO+ProAS	99.0±1.7	88.9±4.8	98.7±2.3*	97.2±4.8	86.1±10.5	92.4±2.6*	94.3±2.4*	2.44×*	89.1±2.7*	70.1±2.3	1.58×

_{Bold: ProAS improves over its base algorithm. ^*: best value within each model-size block.}

Ablation Study

All ablations use Qwen2.5-1.5B-Instruct with GRPO as the base algorithm.

• (a) Advantage shaping: PR adds a progress reward to GRPO without ProAS advantage shaping. ProAS₋ disables MUI, ProAS₊ disables APR, and ProAS_msk± replaces APR with plain masking of reasonable actions in negatively advantaged trajectories. ProAS_suffix replaces Future-Drift in APR with the remaining suffix length, retaining MUI and ARG evolution. All five alternatives fall below full ProAS on overall ALFWorld SR, and none exceeds it on WebShop SR.

• (b) ARG: Random replaces ARG annotations with random labels, Static disables ARG evolution, Incomplete prunes 40% of the ARG's state-action pairs, and Distill builds the ARG from rollouts generated by the 7B model. Random labels collapse performance, while ProAS still clearly beats GRPO with an incomplete or distilled ARG. Disabling evolution on an incomplete ARG costs about 5 SR points on both environments.

Method	ALFWorld							WebShop
	Pick	Look	Clean	Heat	Cool	Pick2	All	TS	SR
GRPO	80.8±4.6	69.4±4.8	70.7±6.1	75.0±8.3	63.9±9.6	59.1±9.1	70.3±2.8	82.4±1.3	57.0±3.1
GRPO + ProAS	90.9±5.2*	86.1±4.8*	90.7±2.3	91.7±8.3*	83.3±4.2*	78.8±5.2	87.0±2.3*	85.3±2.4	64.6±1.2
(a) Advantage Shaping Ablation
GRPO + ProgressReward	85.9±4.6	44.4±4.8	86.7±4.6	83.3±8.3	77.8±6.4	42.4±5.2	72.9±2.3	83.8±2.6	61.2±2.5
GRPO + ProAS−	83.8±9.7	52.8±19.2	78.7±8.3	77.8±9.6	63.9±19.2	60.6±5.2	71.6±10.4	86.0±2.4*	63.8±5.9
GRPO + ProAS+	86.9±1.7	83.3±8.3	85.3±6.1	88.9±12.7	76.4±2.4	80.3±2.6*	83.3±1.8	84.1±2.1	64.6±1.8
GRPO + ProASmsk±	80.8±4.6	72.2±4.8	74.7±12.2	83.3±8.3	70.8±4.2	56.1±2.6	72.9±2.4	83.5±2.0	63.8±2.5
GRPO + ProASsuffix	87.9±3.0	83.3±8.3	85.3±2.3	88.9±4.8	76.4±6.4	68.2±4.5	81.5±2.0	85.6±1.1	63.0±5.5
(b) ARG Ablation
Random	67.7±7.0	38.9±9.6	56.0±10.6	44.4±4.8	34.7±19.7	28.8±6.9	47.7±4.8	78.6±2.1	49.2±3.6
Static	87.9±6.1	83.3±0.0	94.7±2.3*	86.1±4.8	70.8±7.2	72.7±4.5	82.8±2.8	85.2±1.7	65.9±2.7*
Incomplete	85.9±1.7	75.0±8.3	77.3±4.6	77.8±4.8	76.4±2.4	66.7±10.5	77.3±2.1	85.0±2.4	63.0±3.2
Incomplete & Static	82.8±7.6	75.0±8.3	76.0±4.0	77.8±4.8	62.5±12.5	57.6±6.9	72.1±5.2	82.3±1.4	57.6±2.4
Distill	85.9±9.3	80.5±4.8	89.3±2.3	88.9±4.8	80.5±8.7	75.7±2.6	83.6±0.8	84.8±1.0	64.8±1.4

_{Bold: higher than GRPO. ^*: best mean in each column.}

---

🙏 Acknowledgements

ProAS is built on verl-agent at commit 796ed310 (27 Feb 2026, Update HGPO readme). Our GRPO, DAPO, and GiGPO experiments use the implementations from this version of verl-agent. Our MiGPO experiments use the implementation from BEACON. Furthermore, We thank the developers and maintainers of ALFWorld and WebShop.

📚 Bibliography

If you find this repository helpful for your project, please consider citing our work:

@misc{ding2026proasprogressawareadvantageshaping,
      title={ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning}, 
      author={Bowen Ding and Mingda Wan and Yuhan Chen and Jiyao Yuan and Qi Zhu and Futing Wang and Heyuan Deng and Fei Mi and Lifeng Shang and Tao Lin},
      year={2026},
      eprint={2610.18683},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.18683}, 
}