Pramana.
Technical report · October 2026

Pramana: Open-Source Decision Models

Small, calibrated models for structured decisions. This report covers how the models were trained, the calibration method at their core, how they were evaluated, and what did not work.

Pramana-2B and Pramana-0.8B · open weights · Apache 2.0

KL from gold
0.0755
Pramana-2B. Lowest of any fine-tuned entry listed.
Brier score
0.0415
Pramana-2B. Lowest of any fine-tuned entry listed.
Confidence ECE
0.025
Pramana-0.8B's separate confidence output.
Accuracy
0.8005
Pramana-2B. Level with small models, behind the best 4B.

All Pramana numbers are computed from our published prediction files with the benchmark's metric code. Every other number is as listed on the benchmark leaderboard unless stated.

In short

Results

Typed Decisions has 400 test cases across four workflows (agent-trace review, customer service, invoice processing, security incidents), each with five typed questions: 600 yes/no (noul), 600 one-of-N (choice) and 800 ordered-rubric (score) decisions. The gold label for each decision is a soft distribution g, the mean of three teacher samples; the model outputs a distribution p over the same options.

MetricDefinition (per decision, averaged over 2,000)What it rewards
Accuracy ↑argmax p = argmax gpicking the teacher's most likely option
KL ↓Σ g·log(g / p)matching the full soft distribution, heavily punishing near-zero p where g has mass
Brier ↓Σ (p − g)²matching the distribution, with bounded penalties
ECE ↓10 equal-width bins of max p; Σ (n_b/N)·|acc_b − conf_b|top probability equal to the chance of being right

The benchmark keeps fine-tuned models and zero-shot models in separate tables, and so do we.

Like for like: models fine-tuned on the train split

ModelParamsAccuracy ↑KL ↓Brier ↓ECE ↓
Pramana-2B2B0.80050.07550.04150.168 · 0.038†
Pramana-0.8B0.8B0.79700.08190.04460.165 · 0.025†
Decider-4B4B0.82050.3010.1320.031
Arbiter-4B4B0.8170.0920.0480.184
Darwin-27B-ZTC-v527B0.812–––
ministral3-3b-reasoning3B0.8080.0770.0420.182
sysone-0.4B0.4B0.80250.0800.04380.174
OpenDecider-large-td80B-A3B0.8010.0810.044–
od1-typed-decisions4B0.7970.0820.0450.156
OpenDecider-nano0.4B0.7960.0790.043–

† ECE of probabilities · ECE of the separate confidence output. The benchmark computes everyone's ECE from probabilities, so the first number is the like-for-like one.

KL from gold, fine-tuned models
Lower is better. One entry with a much higher listed KL (0.301) is omitted from the scale.
The best entries sit within 0.005 of each other; Pramana-2B has the lowest listed value.

Zero-shot systems, for scale only

These general systems answer any question schema and have never seen these workflows. They are not comparable with fine-tuned models; Pramana is trained for its four workflows and transfers poorly outside them (an earlier adapter scored 0.572 on CLINC150 and 0.255 on Banking77).

ModelAccessAccuracyKLBrierECE
Quyet-1.0-Large (31B)open weights0.7910.2810.1140.093
meraGPT Decider 1API0.7680.0960.0520.180
Baseten Mammoth 1 (27B)API0.7680.1220.0630.153
Liquid AI d1API0.7420.4750.1550.124
TypeSafe Jev 1.13.0API0.7271.4420.1480.144
A ceiling worth knowing Gold labels are the mean of three samples from a roughly 4B teacher whose self-agreement is 0.735. Every fine-tuned model in the table, Pramana included, scores above that, so part of what accuracy measures here is agreement with the teacher's habits.

Detailed evaluation

The tables below break both Pramana models down by question type, workflow and operating point. Three fine-tuned references that publish per-decision predictions are shown for context, recomputed with the same code, so their figures come from public predictions rather than leaderboard rows. Best value per column in bold.

By question type

Modelnoul (600)choice (600)score (800)
AccKLBrierAccKLBrierAccKLBrier
Pramana-2B0.8580.03650.02870.7670.10370.05460.7820.08350.0413
Pramana-0.8B0.8650.03910.02970.7620.11150.05840.7720.09190.0454
Decider-4B0.8800.04100.03000.8000.10670.05480.8070.09000.0438
Arbiter-4B0.8650.10540.05720.7800.09970.05200.8010.07670.0379
sysone-0.4B0.8620.03760.02920.7520.11640.06070.7960.08440.0421

Pramana's KL and Brier advantage comes from yes/no questions, where its distributions are the closest to gold in the table (KL 0.0365). On choice and score questions its accuracy is lower, which accounts for its overall accuracy level.

By workflow (accuracy)

ModelAgent tracesCustomer serviceInvoicesSecurity incidents
Pramana-2B0.7500.8300.8520.770
Pramana-0.8B0.7440.8220.8380.784
Decider-4B0.7740.8440.8640.826
Arbiter-4B0.7420.8480.8720.794
sysone-0.4B0.7360.8160.8660.792

Pramana-2B's KL by workflow: agent traces 0.079, customer service 0.083, invoices 0.077, security incidents 0.063. Its accuracy is highest on invoices (0.852) and lowest on agent traces (0.750).

Selective accuracy

Accuracy on the most confident fraction of decisions, ranking by max(probabilities). This is the operating mode of a deployed decision model, which hands its least confident cases to a person. Selective AUC is the area under accuracy-versus-coverage.

Model50% coverage70%80%90%Selective AUC
Pramana-2B0.9470.8970.8710.8320.927
Pramana-0.8B0.9420.8940.8630.8340.922
Decider-4B0.9490.9080.8740.8530.932
Arbiter-4B0.9380.8950.8740.8430.915
sysone-0.4B0.9320.8890.8660.8370.922

Pramana-2B answers its most confident half of decisions at 94.7% accuracy and 80% of decisions at 87.1%; accuracy declines smoothly as coverage approaches 100%.

Conformal coverage

Split-conformal prediction sets at a 90% coverage target. Pooled coverage can look compliant while one question type falls short, so the per-type (Mondrian) minimum is reported alongside it.

ModelPooled coverageMean set sizeLowest per-type coverage
Pramana-2B0.9200.4430.893 (noul)
Pramana-0.8B0.9290.4560.883 (choice)
Decider-4B0.9150.4080.880 (noul)
Arbiter-4B0.9160.4290.867 (noul)
sysone-0.4B0.9290.4430.863 (choice)

Every model meets the 90% target pooled, and every model misses it on at least one question type. Pramana-2B has the smallest per-type shortfall.

Ordinal error on score questions

ModelExpected-score MAELevel error rateMass ≥ 2 levels from gold
Pramana-2B0.1910.2350.030
Pramana-0.8B0.1980.2490.034
Decider-4B0.1920.2050.024
Arbiter-4B0.1870.2140.025
sysone-0.4B0.1910.2180.038

Expected-score error is level across models; Pramana picks the wrong level slightly more often, consistent with its lower score accuracy.

Murphy decomposition

Brier-style decomposition of top-label confidence into reliability (calibration error, lower is better) and resolution (how much confidence separates right from wrong answers, higher is better).

ModelReliability ↓Resolution ↑
Pramana-2B0.03310.0289
Pramana-0.8B0.03250.0293
Decider-4B0.02920.0179
Arbiter-4B0.03970.0204
sysone-0.4B0.03390.0233

Both Pramana models have the highest resolution: their confidence separates correct from incorrect answers more sharply than the references', even though their raw top-probability calibration is ordinary. That is the property the decoupled confidence output exploits.

Cost-optimal deferral

If an error costs k times a human review, the best policy answers only decisions whose expected accuracy exceeds 1 − 1/k and defers the rest. Coverage and accuracy at that optimum:

Model5:1 coverage5:1 accuracy20:1 coverage20:1 accuracy
Pramana-2B0.4980.9490.2920.993
Pramana-0.8B0.5080.9420.2790.987
Decider-4B0.6420.9280.2960.987
Arbiter-4B0.5700.9280.2260.969
sysone-0.4B0.5480.9270.2360.996

At an error-to-review cost of 5:1, Pramana-2B answers about half of all decisions at 94.9% accuracy; at 20:1 it answers 29% at 99.3%.

A learned review flag is not an uncertainty signal

Two workflows ask the model directly whether a case needs human review. Ranking cases by that predicted probability instead of by confidence loses 9 to 12 points of accuracy at 80% coverage for every model (Pramana-2B: 0.871 by confidence, 0.754 by its review flag). The review question encodes consequence, not the model's own uncertainty.

Method

The metric conflict

The soft gold is the average of three teacher samples, so it spreads probability more widely than the teacher is actually wrong. KL rewards a model that matches that spread. Top-label ECE rewards a model whose top probability equals its chance of being right. Those want different temperatures.

The chart traces what happens when each model's output distribution is rescaled by a single temperature T, from sharp (T = 0.33) to flat (T = 3). Every curve has a low-KL end and a low-ECE end, and no point that is low on both.

KL versus ECE as one temperature varies
Each line is one model's outputs rescaled from T = 0.33 to T = 3. Lower-left is better on both.
Every curve has a low-KL end and a low-ECE end, and no point that is low on both. The star is Pramana-2B's decoupled output, which no single curve reaches.

Decoupled calibration

Instead of choosing a point on the curve, Pramana returns two calibrations of the same logits. Both temperatures are fitted per question type on 80 training cases held out from training, never on test, and are shipped with the weights.

Logits z one forward pass ÷ T_p (≈ 1.0) fitted to the soft gold ÷ T_c (≈ 0.3–0.4) fitted to the hard label probabilities scored · KL 0.0755 · Brier 0.0415 confidence routing · ECE 0.038 (vs 0.168)
ModelKLECE of max(probabilities)ECE of confidence
Pramana-2B0.07550.1680.038
Pramana-0.8B0.08190.1650.025

Training without a calibration step

Pramana is trained with cross-entropy against the soft gold plus a Brier term (weight 1.0) and a small ordinal term (0.1), and is shipped without a post-hoc hard-label calibration. The fitted probability temperature for all three released adapters is 1.00 to 1.05; Pramana-0.8B's holdout KL is 0.0747 before and after fitting. On an earlier run, the library's default hard-label calibration sharpened the same model about 3× and raised KL from 0.076 to 0.50.

Two adapters on one base

The released Pramana-2B averages the probabilities of two LoRA adapters trained on the same base with learning rates 2e-4 and 1e-4. Alone they score 0.7970 and 0.7940; together, 0.8005 with lower KL and Brier than either. Inference costs two 2B passes. In fairness: averaging was decided after both adapters had been scored on test. Its accuracy gain over the better adapter is within noise; the KL and Brier gains are the consistent part.

A cascade between the two sizes

Who answers each decision
Pramana-0.8B answers when its top probability is at least 0.68 (chosen on held-out training rows); otherwise Pramana-2B answers.
Cascade accuracy 0.8000 against 0.8005 for Pramana-2B alone; KL 0.0783, Brier 0.0425. Validate the threshold on your own traffic before relying on it.

Training configuration

SettingPramana-2B (each adapter)Pramana-0.8B
BaseQwen3.5-2B, 4-bitQwen3.5-0.8B, 4-bit
LoRArank 16, alpha 16rank 16, alpha 16
Datatrain split, 1,120 cases (80 held out)same
Steps · batch400 · 8 (~2.9 epochs)400 · 8
Learning rate2e-4 and 1e-4, cosine3e-4, cosine
Sequence length1,0242,048
Losssoft-gold CE + Brier 1.0 + ordinal 0.1same
Hardwareone A100 80GBone A100 80GB

What did not work

Negative results are the most useful part of this report for anyone working on the same benchmark.

ChangeTest accuracyVerdict
Brier weight 0.1, 250 steps (vs 1.0, 400 steps)0.784 vs 0.801second is better on every metric; two changes at once, so confounded
Ordinal weight 1.0 + label smoothing0.7965no accuracy gain
Sequence length 2,048 (2B)0.7950no gain on long-log workflows
LoRA rank 64, LR 3e-40.7855overfits: best holdout (0.8325), worst test
Learning rate 1e-4 alone0.7940worse holdout KL; rejected before test
Three seeds of one recipe0.797 ± 0.005KL 0.0762 ± 0.0007: very stable

The accuracy gap to the leaders is concentrated in the long-log workflows (security incidents, agent traces), which suggested truncation; the 2,048-token run ruled that out. The two models above Pramana on accuracy report training on extra decision data. That, not more tuning, is the remaining lever.

Reproduce

# score the released predictions with the benchmark's metric definitions
uv run --with pyarrow python bench/score.py results/final/pramana2b/preds.jsonl

# refit both temperatures and the cascade threshold from the shipped logits
uv run --with torch --with pyarrow python bench/ensemble.py \
    results/final/large/logits.pt results/final/lowlr/logits.pt \
    --small results/final/small/logits.pt --out results/final/pramana2b

# retrain the release candidates (GPU machine, one A100 80GB)
bash jobs/final.sh

Limitations

Acknowledgements

Base models Qwen3.5-2B and Qwen3.5-0.8B by the Qwen team. Training framework and decision head: Unsloth 2026.10.7. Benchmark, metric definitions and all comparison figures: Typed Decisions by LocalLLaMA.