Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios

This paper evaluates seven frontier AI models (Aug 2024–Feb 2026) on two bespoke cyber ranges—a 32-step corporate-network attack and a 7-step industrial control system (ICS) attack—measuring how many sequential attack steps an autonomous agent can complete. Key findings: performance scales roughly logarithmically with inference-time token spend (no plateau up to 100M tokens), newer model releases outperform predecessors at fixed token budgets, and the best run completed 22/32 corporate steps (Opus 4.6 at 100M tokens); ICS progress remains limited. The work highlights capability growth, key bottlenecks (reverse engineering, cryptography, CI/CD supply-chain phases), experimental limitations (no active defenders, token budgets, minimal scaffolding), and implications for threat proliferation and evaluation methods.