Yardstick 600 generated SQL queries, all executed for real

Can you tell which SQL is wrong?

Aakash Mehta Email LinkedIn Code and data Full write-up

Why this exists

Teams decide an AI is safe to put near their data using one number: how often it gets the answer right. That number cannot tell you whether anyone would notice when it is wrong. A model that fails loudly at 80 percent accuracy is safer than one that fails silently at 90.

So this study measures the second thing, with the hypothesis and the statistics committed to version control before any data was collected.

1

What this is

step 1 of 3

Companies now let AI write . When it gets one wrong, it rarely breaks. The query runs fine and quietly returns wrong numbers, and those numbers go into somebody's report.

So I had AI write 600 queries, ran every one against a real database, and checked them against human answers. Then I counted how many of the wrong ones anybody could have spotted.

150 questions · 4 configurations · 600 runs Llama 3.1 8B vs 3.3 70B zero-shot vs few-shot Spider corpus, 93 databases pre-registered McNemar exact + bootstrap CI
↓ Now try it yourself  six queries, about a minute Start →

One term you need before the quiz

A silent failure is a query that runs perfectly, reports no error, and gives you the wrong numbers anyway. Nothing looks broken. There is no red text, no warning, no crash. The numbers just go into somebody's report.

That is the kind of mistake this study is about, and it is the kind you are about to try to catch by eye. 93 of the 104 mistakes here were this kind.

2

Judge it yourself

step 2 of 3

Six real queries

not started
loading 600 runs…
—of the best model's mistakes ran with no error at all and handed back wrong numbers
0difficulty levels where showing the AI worked examples first helped at all
+28points more accurate from the bigger model on the hardest questions. Odds this is a fluke: about 1 in 750
84%of all mistakes (87 of 104) caught by running four model setups (two models, two prompts each) and checking only where they disagree

swipe the chart sideways →

Bar chart. Share of each setup's errors that were silent: the 8B model with no examples 89 percent, 8B with examples 81 percent, 70B with examples 95 percent, 70B with no examples 100 percent. Accuracy rises from 77 to 89 percent across the same four rows.
Read the bars, then read the numbers on the right. Accuracy climbs as you go down, from 77 percent to 89 percent. The bars climb too, and those measure the share of each setup's mistakes that nobody could see. The best model made the fewest mistakes, and every single one of them ran perfectly and returned wrong numbers.

That is the whole finding. A query that crashes gets investigated. A query that returns believable wrong numbers goes into a dashboard. The second kind is a , and it accounted for 82 to 97 percent of every mistake in this study. The share went up as the models got better, which is the part worth worrying about.

Everything below is the raw evidence. Filter all 600 runs by , difficulty or outcome, click any row to see the human answer next to the machine one, or write your own SQL against the results.

3

Dig through the data yourself

step 3 of 3

Every run

—
Setup

Variant Tier Database Question Outcome Conf Rows
Hover a row to preview the generated SQL. Click to open it against the human answer.

Query the results

table: runs  ·  600 rows

A study about writing SQL ought to let you write some. This runs a real query engine in your browser, hand-written for this page, over all 600 runs.

Ctrl or Cmd + Enter

So what do you do about it

Accuracy scores cannot help you here, because the failures they miss are exactly the ones that look fine. You need a check that runs on every query and does not get tired.

Send the same question to more than one model setup and compare the rows they come back with. When they all agree, move on. When they disagree, have a person look.

This study used four model setups (two models, two prompts each). They disagreed on 28 percent of questions (42 of 150), and that 28 percent contained 84% of every mistake made (87 of 104). Four calls per question instead of one, still about a tenth of a cent at list prices, and roughly three quarters of questions never need a human at all.

The honest caveat: this was measured on one benchmark, at one point in time, and it was never deployed anywhere. A cheaper version with only two setups was not measured separately. Test it on your own queries before you trust it.