Microsoft introduces ExCyTIn-Bench: A new benchmark for AI in cybersecurity
Microsoft has launched ExCyTIn-Bench, a groundbreaking open-source benchmarking tool designed to evaluate how effectively AI systems perform in real-world cybersecurity investigations. Unlike traditional benchmarks that focus on static knowledge or threat intelligence trivia, ExCyTIn-Bench simulates realistic, multi-stage cyberattack scenarios within a Microsoft Azure-based Security Operations Center (SOC). It draws from 57 log tables across Microsoft Sentinel and related services to mirror the complexity and scale of actual SOC environments.
For CISOs and IT leaders, ExCyTIn-Bench offers a practical and transparent way to assess AI’s true capabilities in detecting, investigating, and responding to advanced cyber threats. Microsoft uses this framework internally to enhance the AI-driven security features of Microsoft Defender, Sentinel, and Security Copilot, ensuring that models can withstand sophisticated, real-world attack patterns.
Key innovations that set ExCyTIn-Bench apart include:
- Realistic SOC simulations that replicate real analyst workflows and multistep investigations.
- Transparent metrics that track not only what a model finds but how it reasons, helping organizations build trust and ensure compliance.
- Open-source collaboration, enabling researchers and security vendors to test, compare, and improve AI models globally.
Recent results highlight that GPT-5 (High Reasoning) achieved the highest performance in security investigations, emphasizing that deep, structured reasoning is critical to effective cyber defense. Smaller reasoning-optimized models are also catching up, showing the rapid pace of innovation in AI for cybersecurity.
Organizations and security professionals can access and contribute to ExCyTIn-Bench via Microsoft’s official GitHub repository, joining a global effort to drive progress in AI-powered threat detection and response.
To stay informed, Microsoft encourages participation in Microsoft Ignite (November 17–21, 2025).