Kaidris Notes · ABAP
How a 7B local model scores on ABAP: results on an open-source ABAP runtime
Disclosure: Kaidris-ABAP-7B is our product. It is free and open (Apache-2.0); the paid edition, Kaidris Forge, is in development. Every number below is on the public model card.
Most AI coding assistants are trained on languages with millions of public repositories. ABAP is not one of them, and it shows: general code models write plausible-looking ABAP that does not compile, or compiles and gets the business logic wrong. We wanted to know how far a small model that runs on one consumer GPU can get if every example it learns from has been compiled and tested first.
How it was measured
We did not score on an SAP system. Every answer is compiled and its ABAP Unit tests are run by an open-source ABAP runtime — the abaplint transpiler (opens in new tab) and open-abap (opens in new tab) (MIT) — with ABAP SQL running on SQLite, plus our own lint for rules the runtime does not enforce (for example @ escaping and 255-character lines). The runtime is close to a real system but not identical: we found 8 constructs where it rejects valid ABAP and documented each one.
Every exam was validated before use: reference solutions must pass, empty or gutted solutions must fail, and planted subtle bugs must be caught. Scoring is pass@1, greedy, one answer per task.
The results
| Model | General ABAP 158 tasks | ABAP SQL, unseen schemas 165 tasks | ABAP SQL Dev 44 tasks | ABAP SQL Exam A 44 tasks |
|---|---|---|---|---|
| Qwen2.5-Coder-7B-Instruct (base) | 22 | 53 | 13 | 12 |
| Qwen3.5-9B | 17 | — | — | — |
| Seed-Coder-8B-Instruct | 18 | — | — | — |
| oisee abap-v6 (7B) | 24 | 46 | 9 | 13 |
| Kaidris-ABAP-7B (Q8_0) | 51 | 149 (90%) | 35 | 38 |
Scored on an open-source ABAP runtime (abaplint transpiler + open-abap), not on an SAP system. Tasks solved, pass@1, greedy, one answer per task; every answer is compiled and its ABAP Unit tests are run, plus a strictness lint. Source: the model card (opens in new tab).
On 158 general ABAP tasks the fine-tune solves 51, against 22 for its base model, Qwen2.5-Coder-7B-Instruct. On 165 ABAP SQL tasks over two business schemas it never saw in training, it solves 149 (90%), against 53 for the base. The quantised files hold up: the 4.4 GB Q4_K_M file still solves 147 of those 165.
Where it is weak
On our Novel exam — problems that need techniques the model was never shown, like recursion, date arithmetic or remainder-exact allocation — it is no better than the base model. Whole multi-method classes often contain one wrong method; on a business-app exam of 16 methods it got 8 right in one shot. And it has not been validated on a licensed SAP system, so run your own ATC checks and unit tests on anything it writes.
Those weaknesses are exactly what Kaidris Forge is for: an agent that compiles and tests each answer, reads the failures and repairs the code. Forge will be released only once it scores at least twice the free model on the hardest exams, and the numbers will be published with it.
Where the data came from
No SAP software, no SAP system output and no closed-model output were used for training. The general ABAP examples start from MBPP (CC-BY-4.0), LeetCode-style problems and self-oss-instruct (ODC-BY); the SQL examples use invented business tables. Solutions were written by an open Apache-2.0 model (Qwen3.8-27B) and kept only if they compiled and passed every test on the open-source runtime. The training and evaluation tooling was built with Claude Code (Anthropic).
Try it
The model runs with llama.cpp, LM Studio or Ollama:
llama-server -m kaidris-abap-7b-Q8_0.gguf -ngl 99 -c 8192 --jinja
Give it the table columns and the exact class definition — that matters most for good results. Download Kaidris-ABAP-7B (opens in new tab), or join the Forge waitlist.