PrismaCoder-3B: A Specialized Code-Generation Model for Prisma ORM

Abstract

PrismaCoder-3B is a 3-billion-parameter language model fine-tuned from Qwen2.5-Coder-3B-Instruct for the narrow domain of Prisma ORM schema and client-code generation. It was trained through a three-stage pipeline — supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO, an online reinforcement-learning method) — entirely on a single free-tier NVIDIA T4 GPU (16GB) on Kaggle, subject to a 12-hour session limit requiring cross-session checkpoint resumption. On a 1,200-task benchmark (PrismaBench) built for this project, the model achieves 0.9992 pass@1 (Wilson 95% CI [0.9953, 0.9999]), and outperforms several larger open coding models on an identical 200-task subset under matched evaluation conditions.

This report also documents a material limitation discovered through manual testing after publication: the benchmark's automated static checker does not validate relation-clause (@relation) syntax correctness in generated Prisma schemas, a structural gap confirmed by direct inspection of the checker's source code. This report presents the training methodology, comparative results, and the limitation finding together, on the view that a benchmark result and the boundaries of what it measures are inseparable parts of the same claim.

1. Introduction

Prisma is a widely used TypeScript/JavaScript ORM. Schema definition and client-code authoring for Prisma follow enough regular patterns that they are a reasonable target for a specialized, narrow-domain code-generation model, as an alternative to relying on general-purpose coding assistants for this task.

This project asks a narrow, practical question: how far can a small (3B-parameter) model be pushed toward strong performance on this single domain, when training is constrained to consumer-accessible, free-tier infrastructure rather than institutional compute? The answer required solving two separate classes of problem: the machine-learning problem (what training recipe produces good Prisma-specific generation) and the systems-engineering problem (how to run a multi-stage RLHF-style pipeline reliably on a GPU that gets reclaimed every 12 hours).

2. Methodology

2.1 Base Model and Training Stages

• Base model: Qwen/Qwen2.5-Coder-3B-Instruct
• Adaptation method: QLoRA (4-bit NF4 quantization, LoRA rank 16, alpha 32, applied to all linear projections), with LoRA weights merged into the base model between stages.

All stages used a fixed seed (20260830) for reproducibility.

2.2 Training Data

The training corpus (PrismaForge) combines two tracks: a small mined track (real schema.prisma files and call sites crawled from permissively licensed public repositories) and a larger synthetic track (mechanically generated task/schema pairs spanning task families, domains, and difficulty levels). A contamination firewall excludes any training record whose provenance hash matches a PrismaBench benchmark task, applied before the training splits are finalized.

The model reported here was trained on approximately 3,000 of the ~17,500 available finalized training records — below the project's full-scale design target of 120,000 synthetic records. Results should be interpreted as a proof of concept at this data scale rather than an upper bound on the method. The DPO stage's curated (non-mined) pair supply was likewise small in this run (2 static gate-differential pairs); most of DPO's training signal came from on-policy mining rather than the curated set the pipeline is designed to draw on at full scale.

2.3 Infrastructure and Engineering Constraints

Training ran on a single Kaggle notebook session type (NVIDIA T4 ×2 accelerator, though only one GPU was used — see below), with a hard 12-hour session limit. Two infrastructure issues were identified and resolved during development, both worth recording as they generalize beyond this project:

• Unintended multi-GPU data parallelism. With both provisioned GPUs visible to the process, Hugging Face's Trainer automatically wraps the model in torch.nn.DataParallel, regardless of an explicit single-device device_map. This produces a slow, cumulative memory leak from repeated activation-gathering onto one device, eventually causing an out-of-memory failure many steps into a run. Resolved by setting CUDA_VISIBLE_DEVICES=0 before any CUDA-touching import, which removes the second GPU from consideration entirely.
• Cumulative memory fragmentation during generation-heavy RL. GRPO performs three passes per step (sampling generation, policy forward, reference-model forward), and memory fragmentation compounded over many steps until the allocator could not satisfy an allocation despite nominal headroom. Mitigated with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and a periodic (every_n_steps=5) cache-clearing training callback, alongside reduced per-step generation footprint (num_generations, max_completion_length).

Step-level checkpointing (every 10–25 steps depending on stage) combined with an explicit, user-specified checkpoint-path resume mechanism allowed each training stage to span an arbitrary number of 12-hour sessions.

3. Benchmark Design

PrismaBench is a 1,200-task benchmark spanning twelve task categories (T1–T12: schema authoring, schema editing, migration planning, client CRUD, complex reads, nested writes/transactions, aggregations, repository-scale completion, bug fixing, cross-file reasoning, performance-oriented tasks, and security-relevant tasks) across three difficulty levels. Each task carries a canary string and provenance hash used for the training-data contamination firewall described above.

Evaluation mode used throughout this report is the static ("V0") gate: a lightweight checker (LightValidator) that verifies generated output parses, that code fences are balanced, and that model/field names referenced in the output are grounded in the task's provided schema. A more thorough gate chain (V1–V5: actually invoking Prisma's own schema validator, generating a client, type-checking, and executing against a live database) is designed into the project but requires a Node.js ≥18 environment that was not available in the training/evaluation environment used for this report. This distinction is material and is discussed further in Section 5.

4. Results

4.1 Full-Benchmark Result (PrismaCoder-3B, N=1,200)

Per-category pass@1 was at or near ceiling across all twelve categories (11 of 12 at 120/120, 60/60, etc.; T5_complex_reads at 139/140). Error taxonomy across the full run: a single failure classified E4; no other error classes observed. The low exact-match figure alongside near-ceiling pass@1 is expected rather than anomalous: pass@1 measures gate-level acceptability, not verbatim agreement with one reference solution, and correct-but-differently-phrased output scores low on exact-match by construction.

4.2 Comparative Evaluation Against Other Open Models

To assess whether specialization provides a measurable advantage over general-purpose coding models of comparable or larger scale, five additional models were evaluated under identical conditions: the same 200-task benchmark subset, the same prompt construction (system prompt + templated user instruction via each model's own chat template), and the same greedy decoding settings (max_new_tokens=768, do_sample=False).

Two additional candidate models were queued for comparison and did not produce results: microsoft/Phi-3.5-mini-instruct failed to load under this environment's fixed attn_implementation="sdpa" setting (the model architecture does not support scaled-dot-product attention in the transformers version used, and requires attn_implementation="eager"), and google/codegemma-7b-it is a gated model that was not accessible with the token used. Both are disclosed rather than silently omitted, on the principle that a fair comparative claim should state what was attempted and did not run, not only what succeeded.

The 200-task result for PrismaCoder-3B (1.0000) and the full 1,200-task result (0.9992) are drawn from different (overlapping) samples and are consistent with each other, not contradictory; the full-benchmark figure is the authoritative number for this model.

5. A Documented Evaluation Limitation

5.1 Motivating Observation

Following publication, the model was tested manually with a simple, representative prompt: "Create a Prisma schema for a User model with an id, an email string, and a Post model with a title and a relation to User," issued with the model's correct chat template and system prompt. The model produced syntactically clean, non-degenerate text containing the following:

This is not valid Prisma schema syntax. The correct form of the relation declaration is author User @relation(fields: [authorId], references: [id]); the string-arrow notation used has no meaning in the Prisma schema language. Additionally, the reciprocal posts Post[] field — required on the User side of a valid one-to-many relation in Prisma — is absent entirely. This schema would be rejected by Prisma's own validate command.

An earlier, separate test using a mismatched prompt template (an Alpaca-style ### Instruction: / ### Response: format rather than the model's actual ChatML template) produced substantially worse output — degenerate token repetition, a hallucinated continuation of the conversation, and code wrapped in an unrelated JavaScript module export. That failure mode did not reproduce once the correct chat template was used, and is attributed to prompt-format mismatch rather than a model defect. The @relation syntax issue described above, by contrast, persists under correct prompting and is treated as a genuine model limitation.

5.2 Root Cause, Verified Against Source

The static gate (LightValidator, in prismacoder/verification/verifier.py) was inspected directly to determine why this failure was not reflected in the pass@1 figures reported in Section 4. Its first check (G1) parses the task's input context schema, not the model's generated output. Its second check (G2) validates the generated output only for (a) balanced code fences, and (b) whether tokens matching a prisma.<model>. call-site pattern, or a generic key: value-shaped token, reference model or field names that exist in the schema. This design targets validation of Prisma Client call sites — the kind of check meaningful for tasks like "write a query" — and does not parse or validate generated schema text as Prisma schema language, and in particular never inspects whether a relation field carries a correctly formed @relation attribute, or whether both sides of a relation are declared.

Consequently, a generated schema with a missing, malformed, or entirely invented relation clause can pass every check the static gate performs, provided referenced identifiers happen to be grounded in the schema. This is a structural limitation of the evaluation methodology, not an isolated scoring error, and applies to every pass@1 figure reported under the static gate throughout this report and the associated model card.

5.3 Additional Disclosed Limitations

• RL round 2 was not the benchmarked checkpoint. The training run that produced the published model completed a second, self-play-based RL round successfully, matching the GRPO stage's proxy evaluation score. The checkpoint reported in Sections 4.1–4.2, however, is the GRPO checkpoint; RL round 2's checkpoint has not yet been scored on the full benchmark.
• Full gate-chain (V1–V5) verification has not been run against this model at any point, for the reasons given in Section 3. All pass@1 figures in this report should be read as "well-formed and schema-grounded, under static analysis," not as a claim of verified compile-and-execute correctness.
• Training scale, as noted in Section 2.2, is well below the project's full-scale design target.

6. Discussion

The comparative results in Section 4.2 support the claim that targeted fine-tuning on a narrow, well-specified domain produces a measurable advantage over general-purpose models of comparable or larger parameter count, evaluated under matched conditions. This is consistent with prior findings in the broader literature on domain-specialized code models.

The finding in Section 5, however, is treated in this report as equally significant to the headline comparison. A high pass@1 score under a static, syntax-and-grounding-oriented gate is a real, reproducible, and useful signal — it is not, and should not be represented as, a claim about functional correctness of the generated code. The fact that this gap was identified through direct, low-cost manual testing rather than through the benchmark itself is itself informative: automated evaluation at scale is valuable for catching regressions and enabling comparison, but it is not a substitute for spot-checking actual model output against the real downstream tool the code is meant to serve.

7. Future Work

1. Run the full V1–V5 gate chain (Prisma validate, client generation, type-check, execution against a live database) in a Node-enabled environment, and report pass@1 under that stricter standard alongside the static-gate figures already published.
2. Extend the static gate (or add a dedicated check) to parse generated schema artifacts as Prisma schema language and verify relation-clause completeness and correctness specifically, closing the gap identified in Section 5.2.
3. Re-run training at the project's originally targeted data scale (120,000 synthetic records, full curated DPO pair set) and re-evaluate.
4. Separately benchmark the RL-round-2 checkpoint against the published GRPO checkpoint on the full 1,200-task benchmark.

8. Conclusion

PrismaCoder-3B demonstrates that a small, specialized model trained entirely on free-tier infrastructure can substantially outperform larger general-purpose coding models on a narrow, well-defined domain, under a fast, low-infrastructure evaluation methodology. That same evaluation methodology has a documented, source-verified blind spot with respect to relation-syntax correctness, discovered through direct testing after publication and disclosed alongside the results it qualifies. Both facts are published together, and are intended to be read together.

Citation

Published by TIMPS. For questions about this model or the TIMPS coding agent ecosystem, contact timps.ai090@gmail.com .