Pramana: Open-Source Decision Models
Small, calibrated models for structured decisions. This report covers how the models were trained, the calibration method at their core, how they were evaluated, and what did not work.
confidence output.All Pramana numbers are computed from our published prediction files with the benchmark's metric code. Every other number is as listed on the benchmark leaderboard unless stated.
In short
- What it is. A decision model reads one piece of unstructured state (a support thread, an invoice, a security alert) and answers typed questions about it, returning a probability for every option in a single forward pass.
- Where it leads. Distributional quality: Pramana-2B has the lowest KL and Brier among fine-tuned models listed on the benchmark.
- Where it does not. Accuracy. Pramana-2B reaches 0.8005, level with small fine-tuned models and below the best 4B entries. Across every recipe we tried, single models landed between 0.784 and 0.803.
- The idea. On this benchmark, KL and top-label ECE want different temperatures. Pramana returns two calibrations of the same logits instead of forcing one distribution to serve both.
Results
Typed Decisions has 400 test cases across four workflows (agent-trace review, customer service, invoice processing, security incidents), each with five typed questions: 600 yes/no (noul), 600 one-of-N (choice) and 800 ordered-rubric (score) decisions. The gold label for each decision is a soft distribution g, the mean of three teacher samples; the model outputs a distribution p over the same options.
| Metric | Definition (per decision, averaged over 2,000) | What it rewards |
|---|---|---|
| Accuracy ↑ | argmax p = argmax g | picking the teacher's most likely option |
| KL ↓ | Σ g·log(g / p) | matching the full soft distribution, heavily punishing near-zero p where g has mass |
| Brier ↓ | Σ (p − g)² | matching the distribution, with bounded penalties |
| ECE ↓ | 10 equal-width bins of max p; Σ (n_b/N)·|acc_b − conf_b| | top probability equal to the chance of being right |
The benchmark keeps fine-tuned models and zero-shot models in separate tables, and so do we.
Like for like: models fine-tuned on the train split
| Model | Params | Accuracy ↑ | KL ↓ | Brier ↓ | ECE ↓ |
|---|---|---|---|---|---|
| Pramana-2B | 2B | 0.8005 | 0.0755 | 0.0415 | 0.168 · 0.038† |
| Pramana-0.8B | 0.8B | 0.7970 | 0.0819 | 0.0446 | 0.165 · 0.025† |
| Decider-4B | 4B | 0.8205 | 0.301 | 0.132 | 0.031 |
| Arbiter-4B | 4B | 0.817 | 0.092 | 0.048 | 0.184 |
| Darwin-27B-ZTC-v5 | 27B | 0.812 | – | – | – |
| ministral3-3b-reasoning | 3B | 0.808 | 0.077 | 0.042 | 0.182 |
| sysone-0.4B | 0.4B | 0.8025 | 0.080 | 0.0438 | 0.174 |
| OpenDecider-large-td | 80B-A3B | 0.801 | 0.081 | 0.044 | – |
| od1-typed-decisions | 4B | 0.797 | 0.082 | 0.045 | 0.156 |
| OpenDecider-nano | 0.4B | 0.796 | 0.079 | 0.043 | – |
† ECE of probabilities · ECE of the separate confidence output. The benchmark computes everyone's ECE from probabilities, so the first number is the like-for-like one.
Zero-shot systems, for scale only
These general systems answer any question schema and have never seen these workflows. They are not comparable with fine-tuned models; Pramana is trained for its four workflows and transfers poorly outside them (an earlier adapter scored 0.572 on CLINC150 and 0.255 on Banking77).
| Model | Access | Accuracy | KL | Brier | ECE |
|---|---|---|---|---|---|
| Quyet-1.0-Large (31B) | open weights | 0.791 | 0.281 | 0.114 | 0.093 |
| meraGPT Decider 1 | API | 0.768 | 0.096 | 0.052 | 0.180 |
| Baseten Mammoth 1 (27B) | API | 0.768 | 0.122 | 0.063 | 0.153 |
| Liquid AI d1 | API | 0.742 | 0.475 | 0.155 | 0.124 |
| TypeSafe Jev 1.13.0 | API | 0.727 | 1.442 | 0.148 | 0.144 |
Detailed evaluation
The tables below break both Pramana models down by question type, workflow and operating point. Three fine-tuned references that publish per-decision predictions are shown for context, recomputed with the same code, so their figures come from public predictions rather than leaderboard rows. Best value per column in bold.
By question type
| Model | noul (600) | choice (600) | score (800) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Acc | KL | Brier | Acc | KL | Brier | Acc | KL | Brier | |
| Pramana-2B | 0.858 | 0.0365 | 0.0287 | 0.767 | 0.1037 | 0.0546 | 0.782 | 0.0835 | 0.0413 |
| Pramana-0.8B | 0.865 | 0.0391 | 0.0297 | 0.762 | 0.1115 | 0.0584 | 0.772 | 0.0919 | 0.0454 |
| Decider-4B | 0.880 | 0.0410 | 0.0300 | 0.800 | 0.1067 | 0.0548 | 0.807 | 0.0900 | 0.0438 |
| Arbiter-4B | 0.865 | 0.1054 | 0.0572 | 0.780 | 0.0997 | 0.0520 | 0.801 | 0.0767 | 0.0379 |
| sysone-0.4B | 0.862 | 0.0376 | 0.0292 | 0.752 | 0.1164 | 0.0607 | 0.796 | 0.0844 | 0.0421 |
Pramana's KL and Brier advantage comes from yes/no questions, where its distributions are the closest to gold in the table (KL 0.0365). On choice and score questions its accuracy is lower, which accounts for its overall accuracy level.
By workflow (accuracy)
| Model | Agent traces | Customer service | Invoices | Security incidents |
|---|---|---|---|---|
| Pramana-2B | 0.750 | 0.830 | 0.852 | 0.770 |
| Pramana-0.8B | 0.744 | 0.822 | 0.838 | 0.784 |
| Decider-4B | 0.774 | 0.844 | 0.864 | 0.826 |
| Arbiter-4B | 0.742 | 0.848 | 0.872 | 0.794 |
| sysone-0.4B | 0.736 | 0.816 | 0.866 | 0.792 |
Pramana-2B's KL by workflow: agent traces 0.079, customer service 0.083, invoices 0.077, security incidents 0.063. Its accuracy is highest on invoices (0.852) and lowest on agent traces (0.750).
Selective accuracy
Accuracy on the most confident fraction of decisions, ranking by max(probabilities). This is the operating mode of a deployed decision model, which hands its least confident cases to a person. Selective AUC is the area under accuracy-versus-coverage.
| Model | 50% coverage | 70% | 80% | 90% | Selective AUC |
|---|---|---|---|---|---|
| Pramana-2B | 0.947 | 0.897 | 0.871 | 0.832 | 0.927 |
| Pramana-0.8B | 0.942 | 0.894 | 0.863 | 0.834 | 0.922 |
| Decider-4B | 0.949 | 0.908 | 0.874 | 0.853 | 0.932 |
| Arbiter-4B | 0.938 | 0.895 | 0.874 | 0.843 | 0.915 |
| sysone-0.4B | 0.932 | 0.889 | 0.866 | 0.837 | 0.922 |
Pramana-2B answers its most confident half of decisions at 94.7% accuracy and 80% of decisions at 87.1%; accuracy declines smoothly as coverage approaches 100%.
Conformal coverage
Split-conformal prediction sets at a 90% coverage target. Pooled coverage can look compliant while one question type falls short, so the per-type (Mondrian) minimum is reported alongside it.
| Model | Pooled coverage | Mean set size | Lowest per-type coverage |
|---|---|---|---|
| Pramana-2B | 0.920 | 0.443 | 0.893 (noul) |
| Pramana-0.8B | 0.929 | 0.456 | 0.883 (choice) |
| Decider-4B | 0.915 | 0.408 | 0.880 (noul) |
| Arbiter-4B | 0.916 | 0.429 | 0.867 (noul) |
| sysone-0.4B | 0.929 | 0.443 | 0.863 (choice) |
Every model meets the 90% target pooled, and every model misses it on at least one question type. Pramana-2B has the smallest per-type shortfall.
Ordinal error on score questions
| Model | Expected-score MAE | Level error rate | Mass ≥ 2 levels from gold |
|---|---|---|---|
| Pramana-2B | 0.191 | 0.235 | 0.030 |
| Pramana-0.8B | 0.198 | 0.249 | 0.034 |
| Decider-4B | 0.192 | 0.205 | 0.024 |
| Arbiter-4B | 0.187 | 0.214 | 0.025 |
| sysone-0.4B | 0.191 | 0.218 | 0.038 |
Expected-score error is level across models; Pramana picks the wrong level slightly more often, consistent with its lower score accuracy.
Murphy decomposition
Brier-style decomposition of top-label confidence into reliability (calibration error, lower is better) and resolution (how much confidence separates right from wrong answers, higher is better).
| Model | Reliability ↓ | Resolution ↑ |
|---|---|---|
| Pramana-2B | 0.0331 | 0.0289 |
| Pramana-0.8B | 0.0325 | 0.0293 |
| Decider-4B | 0.0292 | 0.0179 |
| Arbiter-4B | 0.0397 | 0.0204 |
| sysone-0.4B | 0.0339 | 0.0233 |
Both Pramana models have the highest resolution: their confidence separates correct from incorrect answers more sharply than the references', even though their raw top-probability calibration is ordinary. That is the property the decoupled confidence output exploits.
Cost-optimal deferral
If an error costs k times a human review, the best policy answers only decisions whose expected accuracy exceeds 1 − 1/k and defers the rest. Coverage and accuracy at that optimum:
| Model | 5:1 coverage | 5:1 accuracy | 20:1 coverage | 20:1 accuracy |
|---|---|---|---|---|
| Pramana-2B | 0.498 | 0.949 | 0.292 | 0.993 |
| Pramana-0.8B | 0.508 | 0.942 | 0.279 | 0.987 |
| Decider-4B | 0.642 | 0.928 | 0.296 | 0.987 |
| Arbiter-4B | 0.570 | 0.928 | 0.226 | 0.969 |
| sysone-0.4B | 0.548 | 0.927 | 0.236 | 0.996 |
At an error-to-review cost of 5:1, Pramana-2B answers about half of all decisions at 94.9% accuracy; at 20:1 it answers 29% at 99.3%.
A learned review flag is not an uncertainty signal
Two workflows ask the model directly whether a case needs human review. Ranking cases by that predicted probability instead of by confidence loses 9 to 12 points of accuracy at 80% coverage for every model (Pramana-2B: 0.871 by confidence, 0.754 by its review flag). The review question encodes consequence, not the model's own uncertainty.
Method
The metric conflict
The soft gold is the average of three teacher samples, so it spreads probability more widely than the teacher is actually wrong. KL rewards a model that matches that spread. Top-label ECE rewards a model whose top probability equals its chance of being right. Those want different temperatures.
The chart traces what happens when each model's output distribution is rescaled by a single temperature T, from sharp (T = 0.33) to flat (T = 3). Every curve has a low-KL end and a low-ECE end, and no point that is low on both.
Decoupled calibration
Instead of choosing a point on the curve, Pramana returns two calibrations of the same logits. Both temperatures are fitted per question type on 80 training cases held out from training, never on test, and are shipped with the weights.
| Model | KL | ECE of max(probabilities) | ECE of confidence |
|---|---|---|---|
| Pramana-2B | 0.0755 | 0.168 | 0.038 |
| Pramana-0.8B | 0.0819 | 0.165 | 0.025 |
Training without a calibration step
Pramana is trained with cross-entropy against the soft gold plus a Brier term (weight 1.0) and a small ordinal term (0.1), and is shipped without a post-hoc hard-label calibration. The fitted probability temperature for all three released adapters is 1.00 to 1.05; Pramana-0.8B's holdout KL is 0.0747 before and after fitting. On an earlier run, the library's default hard-label calibration sharpened the same model about 3× and raised KL from 0.076 to 0.50.
Two adapters on one base
The released Pramana-2B averages the probabilities of two LoRA adapters trained on the same base with learning rates 2e-4 and 1e-4. Alone they score 0.7970 and 0.7940; together, 0.8005 with lower KL and Brier than either. Inference costs two 2B passes. In fairness: averaging was decided after both adapters had been scored on test. Its accuracy gain over the better adapter is within noise; the KL and Brier gains are the consistent part.
A cascade between the two sizes
Training configuration
| Setting | Pramana-2B (each adapter) | Pramana-0.8B |
|---|---|---|
| Base | Qwen3.5-2B, 4-bit | Qwen3.5-0.8B, 4-bit |
| LoRA | rank 16, alpha 16 | rank 16, alpha 16 |
| Data | train split, 1,120 cases (80 held out) | same |
| Steps · batch | 400 · 8 (~2.9 epochs) | 400 · 8 |
| Learning rate | 2e-4 and 1e-4, cosine | 3e-4, cosine |
| Sequence length | 1,024 | 2,048 |
| Loss | soft-gold CE + Brier 1.0 + ordinal 0.1 | same |
| Hardware | one A100 80GB | one A100 80GB |
What did not work
Negative results are the most useful part of this report for anyone working on the same benchmark.
| Change | Test accuracy | Verdict |
|---|---|---|
| Brier weight 0.1, 250 steps (vs 1.0, 400 steps) | 0.784 vs 0.801 | second is better on every metric; two changes at once, so confounded |
| Ordinal weight 1.0 + label smoothing | 0.7965 | no accuracy gain |
| Sequence length 2,048 (2B) | 0.7950 | no gain on long-log workflows |
| LoRA rank 64, LR 3e-4 | 0.7855 | overfits: best holdout (0.8325), worst test |
| Learning rate 1e-4 alone | 0.7940 | worse holdout KL; rejected before test |
| Three seeds of one recipe | 0.797 ± 0.005 | KL 0.0762 ± 0.0007: very stable |
The accuracy gap to the leaders is concentrated in the long-log workflows (security incidents, agent traces), which suggested truncation; the 2,048-token run ruled that out. The two models above Pramana on accuracy report training on extra decision data. That, not more tuning, is the remaining lever.
Reproduce
# score the released predictions with the benchmark's metric definitions
uv run --with pyarrow python bench/score.py results/final/pramana2b/preds.jsonl
# refit both temperatures and the cascade threshold from the shipped logits
uv run --with torch --with pyarrow python bench/ensemble.py \
results/final/large/logits.pt results/final/lowlr/logits.pt \
--small results/final/small/logits.pt --out results/final/pramana2b
# retrain the release candidates (GPU machine, one A100 80GB)
bash jobs/final.sh
Limitations
- Pramana answers only the four workflows it was trained on.
- Scores measure agreement with a teacher model, not ground truth.
- Temperatures and the cascade threshold are fitted on 400 held-out decisions; validate on production traffic.
- Accuracy is not state of the art: several larger fine-tuned models score higher.
Acknowledgements
Base models Qwen3.5-2B and Qwen3.5-0.8B by the Qwen team. Training framework and decision head: Unsloth 2026.10.7. Benchmark, metric definitions and all comparison figures: Typed Decisions by LocalLLaMA.