Featured · Final Year Project
Teaching a game agent not to forget
In standard self-play, reinforcement-learning agents develop strong strategies then overwrite them through continued training and collapse back to baseline. My final year project, The Effect of Historical Checkpoints on a Real-Time Strategy Game Agent Training, forces the agent to keep competing against frozen copies of its past selves, weighted towards the checkpoints it struggles with the most.
Over 80 million training steps in MicroRTS, historical self-play prevented the prolonged performance collapses seen in standard self-play and held consistently higher TrueSkill ratings. The video shows the trained agent (blue) playing against competition bot CoacAI.
Results at a glance
Two identical training runs the same agent, same parameters, ~80 million steps. The only difference was that one agent also trains against frozen copies of its past selves and one does not.
Without historical checkpoints: the rating plateaus around 27–31 and repeatedly dips as the agent overwrites its own best strategies. Peak: TrueSkill 34.8.
With historical checkpoints: the rating keeps climbing and holds consistently higher through the second half of training with a peak of 36.7, closing in on the competition bot CoacAI (top dashed line).
Evaluation charts taken directly from the report with TrueSkill rating vs training steps, 83 evaluations per run. Dashed lines mark fixed reference bots.
