Shypd.aiThe largest open index of AI toolsterminal-bench is an open-source benchmark for testing AI agents in real terminal environments. It evaluates LLMs on complicated tasks, providing a reproducible task suite and execution harness for real-world evaluation.
Trending
Also listed in
About
terminal-bench is an open-source benchmark designed to evaluate the performance of AI agents, specifically Large Language Models (LLMs), in realistic terminal environments. It provides a comprehensive suite of tasks that challenge agents with complex, end-to-end scenarios, ranging from compiling code to training models and setting up servers. The tool consists of a dataset of tasks, each with an English instruction, a test script for verification, and a reference solution, along with an execution harness that connects the language model to a sandboxed terminal environment. This setup ensures reproducible and practical evaluation of system-level reasoning. It is currently in beta with approximately 100 tasks, with plans for significant expansion, and welcomes community contributions for new and challenging tasks.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
