Authors: Abdulaziz Alyahya (IMSIU), Abdallah Al Siyabi (Monash University), Markus R. Ernst (UNSW Sydney), Luke Yang (Monash University), Levin Kuhlmann (Monash University / brAIn Lab), and Gideon Kowadlo (Cerenaut).
The Problem: AI That Forgets Everything It Learned Yesterday
Imagine hiring a new employee who is brilliant at their current task — but every time you give them a new project, they completely forget how to do the old one. That is essentially what happens to most AI agents today. This phenomenon is called catastrophic forgetting. When an AI system learns a new skill or task, it overwrites what it previously learned. For a robot in a warehouse, a self-driving car, or a healthcare AI, this is a serious problem. The real world does not pause while the AI retrains from scratch. Our new work, ARROW (Augmented Replay for RObust World models), tackles this problem head-on — and does so without demanding extra memory or compute.
Why Is This Hard?
Teaching an AI to learn multiple tasks in sequence — without forgetting earlier ones — is known as continual reinforcement learning (CRL). It sits at the intersection of two tensions:
- Stability vs. Plasticity: The agent must remember old skills (stability) while staying flexible enough to learn new ones (plasticity). Too much of either breaks the balance.
- Memory vs. Performance: The obvious fix — store everything — is expensive. Real-world agents cannot carry an ever-growing archive of past experiences.
Most existing solutions either use huge memory buffers that do not scale, or require the agent to know explicitly which task it is performing (a luxury rarely available in practice).
Our Solution: Borrow a Page from Neuroscience
The human brain does not forget everything when it learns something new. Why? Neuroscientists believe the brain has two complementary memory systems:
- The hippocampus — a fast system that records recent experiences.
- The neocortex — a slow system that builds long-term, structured knowledge.
While we sleep, the hippocampus “replays” the day’s experiences to the neocortex, gradually weaving them into long-term memory. This is why a good night’s sleep before an exam actually helps. ARROW mirrors this design inside an AI agent.
How ARROW Works
ARROW is built on top of DreamerV3, a state-of-the-art model-based reinforcement learning algorithm. Instead of the agent directly interacting with the real environment at every step, DreamerV3 builds a “World Model” — an internal simulation of how the world works — and practices inside that simulation. Think of it as the agent dreaming up experiences rather than always acting in reality. Our key contribution is a smarter memory system called the Augmented Replay Buffer, which has two compartments working in parallel:
| Buffer | Brain Analogy | What It Stores |
|---|---|---|
| Short-Term FIFO Buffer | Hippocampus | The most recent ~262,000 experiences. Keeps the agent sharp on the current task. |
| Long-Term LTDM Buffer | Neocortex | A carefully maintained sample of experiences spread across all tasks seen so far. Prevents forgetting. |
The “LTDM” stands for Long-Term global Distribution Matching. Essentially, it uses a statistical technique called reservoir sampling to ensure the long-term buffer stays representative of the full range of past experiences — not just the most recent ones.Every training update mixes samples from both buffers. The result: the World Model is always being reminded of past tasks, even as it learns new ones. Crucially, the total memory used by ARROW is the same size as the standard DreamerV3 buffer — we just split it more intelligently.
What We Tested
We evaluated ARROW on two challenging continual learning scenarios:
1. Tasks Without Shared Structure — Atari Games
We trained the agent on six very different Atari games in sequence: Ms. Pac-Man, Boxing, Crazy Climber, Frostbite, Seaquest, and Enduro. These games look and feel completely different from one another. There is no shared skill or visual feature to transfer. This is the hardest case for continual learning.
2. Tasks With Shared Structure — CoinRun (Procgen)
We also tested on six progressive variations of the CoinRun game, where each version introduces a new visual or behavioural change. Because the games share an underlying structure, there is real potential for the agent to transfer knowledge from one version to the next. We compared ARROW against two baselines, using the exact same memory budget:
- DreamerV3 — the same World Model architecture, but with a standard single FIFO buffer.
- TES-SAC — a strong model-free agent (no World Model) as an alternative approach.
The Results: ARROW Nearly Eliminates Forgetting
On Atari (no shared structure)
This is where the difference was most dramatic. DreamerV3 suffered catastrophic forgetting every single time a new game was introduced— its performance on previous games collapsed almost immediately. ARROW reduced forgetting by nearly twenty-fold compared to DreamerV3. TES-SAC appeared to have low forgetting too — but for the wrong reason. It never learned the games well enough in the first place to have anything to forget. This is an important reminder: low forgetting only matters if the agent was actually performing well to begin with.
On CoinRun (shared structure)
When tasks share visual features, all methods forget less. But ARROW still came out on top, achieving the best balance between learning new tasks (plasticity) and remembering old ones (stability) in every single configuration tested.
The Two-Cycle Setting: The Most Practical Test
We also ran a “two-cycle” experiment, where the agent goes through all tasks twice — closer to how real agents would be deployed, cycling through a set of responsibilities repeatedly. This is where ARROW’s advantage was most striking. Between the first and second exposure to each task, ARROW’s maximum forgetting was essentially zero on CoinRun, and an order of magnitude lower than DreamerV3 on Atari. When tasks were revisited, ARROW not only recovered its prior performance — it often surpassed it.
Key Takeaways
- ✅ No extra memory needed. ARROW uses the same memory budget as standard DreamerV3, just allocated more wisely.
- ✅ No task labels required. ARROW does not need to be told which task it is currently on — making it practical for real-world deployment.
- ✅ Bio-inspired and principled. The dual-buffer design maps directly onto Complementary Learning Systems theory from neuroscience.
- ✅ Works across different scenarios. Effective whether tasks share structure or not, and under different task orderings.
- ⚖️ A trade-off exists. ARROW sometimes learns a new task slightly slower than DreamerV3 (lower peak per-task performance within a task’s training window) — because part of its memory is always reserved for past tasks. This is a reasonable price to pay for dramatically better retention.
What Comes Next
This work opens several exciting directions:
- Testing ARROW on robotics and continuous control environments (e.g. MuJoCo).
- Exploring dynamic buffer allocation — letting the agent automatically decide how much memory to dedicate to the short-term vs. long-term buffer based on how fast the environment is changing.
- Combining ARROW with other continual learning techniques, such as Elastic Weight Consolidation (EWC) or the behaviour cloning used in CLEAR.
- Testing on natural curricula — graded difficulty levels where simpler skills genuinely compose into more complex ones.
About This Work
ARROW was developed by Abdulaziz Alyahya (IMSIU), Abdallah Al Siyabi (Monash University), Markus R. Ernst (UNSW Sydney), Luke Yang (Monash University), Levin Kuhlmann (Monash University / brAIn Lab), and Gideon Kowadlo (Cerenaut).
The code is available at github.com/Cerenaut/ARROW. Experiments were run on the Monash eResearch M3 HPC cluster.
