Platform trains and tests AI that operates phones and computers
MAI-Gym supported ten established tests and 20,000 simultaneous software environments, using a shared process to run agents during training and testing.
Edited by Boris Djordjevic · Paperfoot
The researchers built MAI-Gym, an open platform for testing, collecting data and training agents—programs that operate graphical interfaces—across mobile, computer and web environments. It supported ten established benchmarks and scaled to 20,000 environments running at once. It also supported generating imitation apps and tasks with verified completion checks.
Using reinforcement learning, training through rewards, the researchers trained MAI-Gym-9B, a model with 9 billion parameters, the numerical settings learned during training. It scored 78.4% on MobileWorld and 76.2% on OSWorld-Verified, two benchmarks for interface-using agents. The authors reported state-of-the-art performance among open agents below 10 billion parameters. Training in an imitation train-ticketing app also improved task success in its real counterpart on a physical phone, from 28.8% to 48.1%.
Why it matters
The system brought training and testing across different software environments into one platform. The released framework, model and 2,000 verified tasks provide shared materials for reproducing and extending the work.
Caveats
The physical-phone transfer test concerned a train-ticketing app; the abstract did not name comparison agents or report variation across runs for the benchmark results. The work is an arXiv preprint, not yet peer-reviewed.
In this paper
GUI agent evaluation and online reinforcement learning
- Methods
- MAI-Gym (this paper)MAI-Gym-9B (this paper)Qwen3.5-9BGRPO
- Datasets
- MobileWorld (evaluated)OSWorld-Verified (evaluated)OSWorld (evaluated)
Results
- MAI-Gym-9B, performance, on MobileWorld: 78.4%; MAI-Gym-9B, performance, on OSWorld-Verified: 76.2%
Using MAI-Gym, we train MAI-Gym-9B, a unified 9B agent for mobile and computer use, achieving state-of-the-art performance among open GUI agents under 10B parameters: 78.4% on MobileWorld and 76.2% on OSWorld-Verified.
Abstract
- MAI-Gym, median reset time, Android: 165 seconds; MAI-Gym, median reset time, desktop virtual machines: 37 seconds; MAI-Gym, median reset time, the browser: 6 seconds
With these adaptations, a reset, including queueing and task preparation, takes a median of 165, 37, and 6 seconds on Android, desktop virtual machines, and the browser, respectively (Section 4.3.1).
Reliable Task Resets.
- MAI-Gym, provisioning time for 1,000 fresh instances, mobile: 328.4 seconds; MAI-Gym, provisioning time for 1,000 fresh instances, computer: 88.9 seconds
We have tested MAI-Gym with 20,000 instances running in parallel, and 1,000 fresh instances come up in 328.4 seconds for mobile and 88.9 seconds for computer (Section 4.3.1).
Fast Provisioning at Scale.
What it shows
- A unified 9B agent trained with online RL reaches state-of-the-art performance among open GUI agents under 10B parameters on MobileWorld and OSWorld-Verified.
Using MAI-Gym, we train MAI-Gym-9B, a unified 9B agent for mobile and computer use, achieving state-of-the-art performance among open GUI agents under 10B parameters: 78.4% on MobileWorld and 76.2% on OSWorld-Verified.
Abstract
- Reinforcement learning in a mocked train-ticketing app transfers to its real-world counterpart on a physical phone.
Importantly, RL in a mocked train-ticketing app transfers to its real-world counterpart, improving task success on a phys- ical phone from 28.8% to 48.1%.
Abstract
- Joint reinforcement learning training outperforms merging per-domain experts by weighted parameter averaging.
Our analyses indicate that joint training outperforms merging per-domain experts, larger synthesized task pools strengthen the policy, and skills learned in a real-world clone transfer to the real app, raising success on a physical phone from 28.8% to 48.1%.
6 Conclusion
Quoted word for word from the full text on arXiv.
The paper
arXiv, Oct 14, 2026, Open access, CC BY-NC-SA, Preprint, not peer-reviewed