Shypd.aiThe largest open index of AI toolssql-eval is an open-source tool for evaluating the accuracy of LLM-generated SQL outputs. It compares generated SQL queries against 'gold' queries to assess exact and subset matches, logging metrics like tokens used and latency.
Trending
About
sql-eval is an open-source evaluation framework designed to assess the accuracy of SQL queries generated by Large Language Models (LLMs). It operates by taking a question/query pair, generating a SQL query (potentially from an LLM), and then running both the 'gold' query and the generated query on their respective databases. The tool then compares the resulting dataframes using both 'exact' and 'subset' matching criteria. Beyond accuracy, sql-eval logs other critical metrics such as tokens used and latency, providing a comprehensive view of LLM performance. It supports various database types including Postgres, Snowflake, BigQuery, MySQL, SQLite, and SQL Server, and offers runners for popular LLM APIs like OpenAI and Anthropic, as well as local Hugging Face models and vLLM.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
