Scored on a Real Backend, Not a Multiple-Choice Test
The scenarios are drawn from actual support tickets, bug reports, and GitHub issues, covering tasks such as building a schema, debugging a failing Edge Function, and fixing a broken RLS policy. They are organized along products, topics, and four stages: build, deploy, investigate, and resolve. Every scenario runs in a real containerized environment where the agent calls the actual MCP server and CLI. Scoring combines deterministic checks — SQL assertions against project state, real client calls made as a simulated user, and inspection of the files the agent created — with an LLM judge, and each agent gets one retry before grading. The published benchmark suite covers breadth, while a separate regression suite refreshes daily against known failure modes and does not affect public scores.
Skills Move the Numbers More Than Model Choice
In the first snapshot Supabase published, Opus 5 and Kimi K3 already passed the build stage at 100% with no skill loaded. Sonnet 5 went from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89% once Supabase's own context files were available — the remaining gap was closed by context, not by a bigger model. Documentation habits differ just as sharply: Codex and GPT-5.6 read roughly 8 doc pages per scenario against about 2 for Claude Code, which consults the docs in under 40% of scenarios even with skills loaded. Supabase also notes that rewriting the description of its Postgres best-practices skill lifted that skill's activation rate from roughly 1 in 10 sessions to 60%.
How to Read the Leaderboard
This is a vendor benchmark built around the vendor's own product: it measures whether an agent knows how to use Supabase, not general coding ability, and the scores are a snapshot taken at publication. Its more durable value is that the prompts, scorers, runtime, and experiment configuration all ship with the repo. Teams shipping an SDK or an API can copy the setup to build an agent regression suite for their own product, and check the same way whether their skill files and documentation edits actually make agents fail less often.
via: Supabase announcement: Introducing Supabase Evals, supabase/evals repository, MarkTechPost report; verified 2026-08-03