AWS wants to know whether AI agents can actually do real cloud jobs, and it has released an open-source benchmark to find out.
The project, called aws-bench, skips static question sets. Every scenario builds actual AWS resources in throwaway accounts, laid out as CDK stacks. The agent under evaluation gets a sandboxed container with scoped credentials, finishes its task, and then an automated verifier grades the outcome, either through an LLM judge or a programmatic check against the live AWS environment.
Bundled datasets span basic and advanced work: observability, compute and data, databases and storage, multi-region EC2 setups, serverless, streaming and IoT, reference architectures, and multi-service troubleshooting.
The stated goal is a truer read of how agents handle the chores developers actually perform, such as diagnosing misconfigurations, provisioning infrastructure, and operating production-like environments. The initial release targets AI researchers and model providers, but engineering teams can extend the scenarios to match their own systems.
The launch joins a fast-growing field of agent evaluation tools, as organizations hunt for reliable ways to decide when autonomous assistants are safe to trust with production work.
