ABAP SQL fine-tune - the training data failed the audit

[kovakamo]

This post is about the following: Can you teach a small local Model ABAP SQL with training data written by bigger Models, on a budget of 35 Euros and one weekend?

I never got as far as the training.

The plan

Take a Model small enough to run in Ollama on a Mac mini, and teach it to turn plain-English questions into ABAP SQL. Graded by running the SQL on my A4H system and comparing the rows that come back, not by the SQL text.

The setup:

• Student: Qwen3-4B-Instruct-2507, 4 billion parameters, q8_0 in Ollama,
• data: the /DMO/ flight reference model on A4H, client 001, 13 Database Tables,
• grading: each query runs through the ADT data preview, and the result rows are compared with the reference result,
• budget: 35 Euros, one weekend,
• success, decided before any training: the fine-tuned Model on its own beats the base Model with a syntax fixer, and not by luck (McNemar test, p < 0.05).

Claude Code (Opus 5.5) built everything, OpenAI Codex reviewed.

The test set

Rule number one: whoever writes the training data must not write the exam.

So OpenAI Codex drafted the questions, every reference query ran on A4H, a second Codex pass answered each question without seeing the first answer, and I reviewed the disputes. 308 questions in the end:

• 93 easy,
• 106 medium,
• 77 hard,
• 32 the database simply cannot answer (pilot names, weather, that sort of thing).

Frozen, with a fingerprint of the data so nobody could change it behind the scenes.

Then the baselines. % of questions answered correctly:

Little Qwen out of the box: 14.6%. And most of the misses weren't even wrong logic - 93.5% of his SQL didn't run at all. ABAP SQL wants AS before every table alias, a tilde instead of a dot, and blanks inside the brackets of SUM( ). So Claude wrote a fixer with exactly those three rules, and Qwen jumped to 52.9%. That was the bar to beat. Claude got all 308 right, as one does.

Choosing a teacher

The idea of the fine-tune: a big Model writes a few thousand question-and-SQL pairs, we check them, and little Qwen learns from the survivors. But which big Model?

• Not OpenAI: Codex wrote the test set, and the same family mustn't write the training data,
• not Claude, Gemini or Grok: their API terms restrict using their output to build other Models (xAI's terms forbid training any AI system on Grok's output, full stop),
• so an open-weight Model it was.

The teacher: DeepSeek-V4-Pro, MIT licence, and DeepSeek's terms explicitly allow distillation. His checker: GLM-5.2 from Z.ai, a different lab, who answers every question again without seeing the teacher's SQL. Both through OpenRouter, pinned to one provider (DeepInfra), with thinking switched off. The original plan was to rent a GPU on Runpod and host a smaller teacher myself, but OpenRouter was simpler.

Every pair had to survive the same obstacle course:

1. The teacher drafts a question and its SQL.
2. The SQL runs on A4H and returns rows.
3. The checker answers the same question blind, and his rows must match.
4. The question must not be too close to any test question.

Plus three gates, written down before the first paid call:

1. at least 40% of all drafts must survive,
2. at least 25% of the hard ones,
3. I audit 50 survivors myself, with at most 2 wrong (at most 1 among the hard ones).

Fail a gate after the one allowed restart, and we stop.

Pilot one: DeepSeek + ABAP SQL

A pilot of 300 drafts:

• 90 easy,
• 110 medium,
• 70 hard,
• 30 unanswerable.

DeepSeek wrote them in a few minutes, for 27 cents. Then A4H had its say.

Turns out the big Model writes ABAP SQL the way little Qwen does, only with more confidence. FROM /dmo/flight f  without the AS, ROUND(SUM(x), 2) without the blanks ABAP wants, customer~name with no alias called customer anywhere, subqueries in FROM, subtracting one date from another. 57 queries didn't even get an error message, they ended in short DUMPs. In fairness, the dialect notes in his prompt mentioned only the tilde and the column aliases. The rest he was supposed to just know, like Qwen.

The checker had his own moment. The prompt told him to answer with sql: null when a question can't be answered. GLM took that literally and sent back the string "sql: null". A4H refused to run it (fair enough), so 28 of the 30 perfectly good unanswerable questions were marked as disputed. That one was on us, not on him.

The result: 64 of 300 pairs agreed, and 5 of 70 hard ones. Way under both gates. Pilot one, 29 cents, archived with all its evidence.

Pilot two, and one amendment

The one restart we had allowed ourselves came with three changes:

• a checklist of ABAP SQL rules in both prompts, each rule tested on A4H first, so as not to teach the teacher anything wrong,
• "set sql to JSON null, not a string" for the checker,
• Qwen's three-rule syntax fixer applied to both Models' SQL, with the raw SQL kept next to the repaired one, so nothing is hidden.

Much better: 136 of 300 pairs agreed. But only 11 of 70 hard ones, 15.7%, under the 25% bar.

Stop.

When we looked at why, something odd turned up. The teacher had marked 140 of 180 answerable pairs as "order matters", including questions that never ask for any order. The checker returned the same rows in a different order, and the comparison said no.

Fixing that after seeing the failures is exactly the kind of thing that makes a result worthless if you're not careful. My first proposal still let the teacher's flag cast a vote. Codex shot it down: under my rule, he pointed out, not a single order dispute could ever be rescued, so the only way to pass would have been me fixing things by hand in the pilot, and the full run wouldn't do that. Fair.

The version that went through:

• two fresh judgments from DeepSeek and GLM on each question alone - no SQL, no results, no teacher flag,
• both say "order matters" or both say "it doesn't", or the pair is thrown out,
• the same rule for the pilot and the full run,
• disclosed as an amendment made after the failure,
• and if this fails a gate, the experiment stops, no more fiddling.

The two judges agreed on 175 of 180 pairs. Then I labelled 80 pairs of similar-looking training and test questions, which set the threshold for "too close to the test". 25 pairs went out for that. What was left: 132 of 300 pairs (44.0%), and 18 of 70 hard ones (25.7%). Passed. By one hard pair.

Then it was my turn

The last gate was me: 50 surviving pairs, 12 of them hard, each with its question, its SQL and its result. Does the SQL answer the question?

42 did. 8 didn't, 5 of them hard. The limit was 2.

Seven of the eight added up or compared money in different currencies as if it were one number. Trip fees and flight prices in /DMO/ come in EUR, JPY, SGD and USD. Two pairs summed the booking fees per trip status: for booked trips that is 9,920 EUR + 3,700 JPY + 5,980 SGD + 14,530 USD, which the label proudly reported as 34,130.00. Others compared fees to a cutoff of 50 or 500, or sorted flights into price bands of 300, 1,000 and 1,200, in whatever currency happened to be there. In January 2027 every EUR, JPY and SGD flight already costs at least 1,200, so those carriers came out as entirely "premium". The eighth pair was asked for the average number of flights per carrier, and returned each carrier's flight count without ever computing an average.

The worst thing: All eight had run fine on A4H. And for all eight, GLM, from a different lab, answering blind, had written SQL that returned the very same rows. Both Models made the same mistake, so the cross-check waved it through. The teacher prompt even said, in so many words, not to sum or compare amounts across currencies. Two Models agreeing is not the same as two Models being right.

So, as agreed before the reassessment, the experiment stops here. No training, no GPU, no little Qwen with a diploma.

So was it all for nothing?

Yes.

OK, not quite. The automatic checks did their job on syntax: the execution on A4H threw out everything that didn't run, and the cross-check threw out plenty that ran but answered differently. What they couldn't catch was meaning, where both Models were wrong in the same way. 50 pairs read by a human caught that. With a few thousand pairs and no audit, little Qwen would have learned, very diligently, to add yen to dollars.

The fine print

In fairness:

• One teacher and one checker, and both are open-weight. Claude, Gemini, Grok or GPT might have done better, but GPT wrote the test set, and the terms of the other three ruled them out.
• 50 audited pairs is a small sample, and the labels are mine.
• The ordering amendment came after we had seen pilot two fail. It's disclosed, it applied to everything, and it got exactly one go.
• The test set and the baselines stand, and they don't depend on any of this.

P.S.: The whole thing cost 64 cents in API calls: 29 for pilot one, 31 for pilot two, 4 for the order judgments. No GPU was rented. Plus Voyage embeddings for the "too close to the test" check. The setup: a Mac mini, A4H in Docker on a VPS, Claude Code (Opus 5.5) doing the building, Codex doing the reviewing. The test set, the code, both failed pilots and every API call are on GitHub: https://github.com/Kamilkov/abap-sql-bench.