PolyZero is a headless Python engine for The Battle of Polytopia.
It runs the game logic (movement, combat, economy) with no graphics and no UI, so reinforcement learning agents can play thousands of matches without waiting on a renderer.
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txtThe engine only needs numpy and gymnasium. Torch and Stable-Baselines3 are for training,
pygame for the viewer. Install those when you need them — training runs headless without pygame.
Uses MaskablePPO from sb3-contrib, so the agent never samples an illegal action. Output goes to
a timestamped folder under runs/.
python train.py # defaults: heuristic opponent, 200k steps
python train.py --opponent random # warm-up against random play
python train.py --opponent self # self-play against a pool of snapshots
python train.py --timesteps 500000 # longer run
python train.py --resume runs/<run>/best_model.zip--resume continues a run rather than starting one that happens to share weights: the step count
carries over, so --timesteps is what this run adds, and the discounts, clip range and rollout
settings you pass are applied to the loaded model rather than left at whatever it was saved with.
A self-play pool beside the checkpoint is copied into the new run, so the agent keeps the league
it earned.
Stable-Baselines3 measures progress against the current call, so each resumed segment anneals its
own learning rate and entropy from partway down and starts over at the next one. Pass
--anneal-over with the whole budget on every segment to get one descent across all of them:
python train.py --timesteps 2000000 --anneal-over 20000000--opponent heuristic plays the scripted bot in engine/bot.py. Its ceiling is that bot: past a
point the agent stops learning Polytopia and starts learning the script's habits. The bot reads its
targets through its own fog map, so it does not march at a capital it has never seen while the
agent plays blind.
--opponent self is the setting without that ceiling. Snapshots of the run are written to
runs/<run>/pool/ every --snapshot-every steps, and each episode draws one. Half the draws come
from the newest few and half from anywhere in the run, so the agent keeps facing something that can
still beat it without forgetting the strategies it has already answered.
A snapshot is about 11 MB, so the pool is capped at --pool-max. The oldest and newest are always
kept and the closest-spaced ones in between are dropped, which bounds the disk without turning the
league into recent history.
Evaluation always faces --eval-opponent (the scripted bot by default) on a fixed set of maps, so
the number stays comparable across a run even while the training opponent moves.
That number stops saying anything once the agent can beat the bot, so a self-play run also scores
itself against its own pool. pool_eval/vs_first plays the oldest snapshot and should climb all
run; pool_eval/vs_latest plays the newest one written before the current step and should sit
just above zero, since seat 0 moves first. A vs_latest that keeps rising means the pool has gone
stale — lower --snapshot-every.
Self-play costs a policy forward pass for every action the opponent takes, which measured about nine
times slower per environment step than the scripted bot — less than that end to end, since the
learner's own update is the larger cost either way. Each worker also holds its own copies of the
snapshots it has drawn on the CPU, pinned to one thread so eight of them do not each try to use
every core. Budget for both when picking --n-envs.
Run python train.py --help for the full flag list.
A step is one action, not one turn, so a game that runs to the 50-turn limit is about 500 steps at
the ten or so actions a turn takes. Only a policy being eliminated in the opening finishes inside a
hundred. The 200k default is therefore a few hundred full games — enough to watch the agent stop
losing on purpose, nowhere near enough to play well, against an 83,294-slot action space, stochastic
combat and a fresh map every episode. Plan in millions of steps, and use --opponent random for a
short warm-up before switching.
The reward is the change in a potential made of three things: the real score
against the strongest surviving rival, the cities held over that rival, and the
rivals already knocked out. Score alone treats a quiet builder and a conqueror
the same, so --city-reward pays for every city held over the best rival and
--elim-reward pays for removing one, on top of the win bonus.
python train.py --city-reward 0.5 --elim-reward 2.0 # take the map
python train.py --city-reward 0 --elim-reward 0 # play for pointsTaking a city is worth about 1.5 at the defaults, against a whole game's score difference of roughly 3, so capturing is the strongest single move available. The shaping is still potential based, so an episode's rewards sum to the change in that potential and nothing is double counted.
Two numbers set this and only the second one binds.
gamma discounts per action, so a discount of g lets a return reach about 1/(1-g) steps: the
default 0.999 is around 1,000 steps, or 100 turns at 10 actions each. That is the ceiling.
The advantage the policy gradient is actually built from decays at gamma * gae_lambda, and
reaches about 1/(1 - gamma*lambda) steps. This is the one that bounds credit assignment, and it
is dominated by lambda: at any gamma you like, the usual 0.95 reaches 20 steps, which is two
turns. The default here is --gae-lambda 0.98, about 50 steps or five turns. Raise it to plan
further and pay for it in variance.
The run prints the horizon on startup. Watch polyzero/actions_per_turn in TensorBoard and divide
by it to read the figure in turns.
The learner is the bottleneck, not the game. A single environment plays about 1,000 steps a second;
a run measured 100. The update is where the time goes, and most of that used to be the softmax:
a mean of 21 actions are legal out of 83,294, and forming all of them to describe 21 choices cost
more than the trunk did. The policy gathers the legal logits instead, which measured 2.4x on the
update and 1.7x end to end. What is left is real work, so budget hours per million steps and use
--n-envs to fill the cores the update leaves idle rather than to go faster.
At the defaults a run measured 187 steps a second on eight CPU workers, or about thirty hours for
20M, with the update taking 83% of it and the whole stack sitting under 6 GB. Evaluation is charged
on top: twenty episodes take about ten seconds, so the default --eval-freq 10000 adds a further
19% over a run that long. Raise it for anything measured in millions.
Handy for checking that the engine still behaves.
python train.py --watch # 2 bots, no training
python train.py --watch --num-players 4 # 4-way free-for-allPrompts you for the number of players, whether each one is a trained model or a scripted bot, and which run folder to load each model from.
python play.pyBoth viewers share the same controls: drag with the left mouse button to pan, scroll to zoom,
R to reset the camera, SPACE to pause, +/- to change speed, 1-4 to show a player's
tech tree, ESC or Q to quit.
--tribe takes one name or a comma-separated list, one per seat:
python train.py --tribe bardur # every seat plays Bardur
python train.py --tribe bardur,kickoo # asymmetric matchupTribes decide starting tech, starting stars and the opening unit, and the agent can read each seat's tribe from the observation. Terrain generation still keys on seat 0's tribe.
tensorboard --logdir runs/python -m pytest tests/The suite needs only numpy, gymnasium and pytest, and runs in a few seconds. Install torch
and Stable-Baselines3 as well and it also covers the policy and the self-play opponent, which skip
without them.
CI runs it both ways on every push and pull request: once with the engine's dependencies alone, and
once with everything in requirements.txt, followed by a short train.py run so a broken training
script fails here rather than on your machine.
| Path | Contents |
|---|---|
engine/ |
The engine. Board, units, cities, buildings, tech tree, movement and combat. |
engine/env.py |
Gymnasium environment wrapping the engine for RL. |
engine/observation.py |
The observation layout. Both the env and the live bridge build against it. |
engine/bot.py |
The scripted bot, shared by the training opponent and both viewers. |
engine/visualizer.py |
Optional pygame viewer. The engine runs headless without it. |
engine/bridge.py, engine/live_client.py |
Talk to the real game over TCP (see bridge/). |
bridge/ |
C# plugin, loaded into Polytopia by PolyMod, that the two modules above connect to. |
heuristic.py |
Tech-tree overlay for the pygame viewers. |
assets/ |
Tribe sprites used by the viewer. |
tests/ |
pytest suite, organised by subsystem. |
Training output lands in runs/, which is not versioned. Checkpoints are large, and you can
regenerate them from the training command.
A checkpoint is tied to the observation layout it trained on. play.py and capture.py check the
shape and refuse a mismatch rather than failing somewhere inside the policy.
engine/observation.py owns that layout, including every normalisation divisor, because the live
bridge builds the same tensor from the real game and the two have to agree exactly. The turn
channel is a fraction of MAX_TURNS, so --max-turns past it is refused rather than quietly
flattening the only channel that says how much game is left.
MIT. See LICENSE.