Example Benchmark Snapshot
Use your own evaluation results. This table shows the kind of inputs the calculator expects.
| Scenario | Baseline Accuracy | Few-shot Accuracy | Stability | Few-shot Latency | Few-shot Tokens |
|---|---|---|---|---|---|
| Customer Support FAQ | 72.5% | 83.0% | 0.86 | 1210 ms | 860 |
| Medical Triage Drafting | 68.0% | 78.5% | 0.80 | 1560 ms | 1040 |
| Invoice Line Extraction | 84.2% | 89.4% | 0.92 | 980 ms | 740 |
Formula Used
This score blends multiple signals into a single 0–100 index. It rewards accuracy lift and stability, while penalizing latency and cost increases.
How to Use This Calculator
- Run your baseline prompt and record accuracy, latency, and token usage.
- Add few-shot examples, then re-run the same benchmark and record the new metrics.
- Enter stability (repeatability) and safety pass rate from your evaluation procedure.
- Adjust weights if your application values speed or cost more than accuracy.
- Click Calculate Score. Export CSV for tracking or PDF for sharing.
Why few-shot scoring matters
Few-shot prompting can improve task success without changing a model, but the gains are rarely free. A single score helps teams compare variants across datasets, identify regressions quickly, and communicate results to stakeholders. When quality, speed, and cost are tracked together, decisions become repeatable instead of subjective. This calculator converts core evaluation signals into a consistent 0–100 index. It also highlights which lever drives improvement, so teams can tune examples, instructions, and evaluation coverage with less guesswork.
Interpreting accuracy lift
Accuracy lift is the difference between few-shot and baseline accuracy, expressed in percentage points. Normalizing by remaining headroom to 100% prevents over-crediting improvements when the baseline is already strong. Diminishing returns further reduces the impact of tiny lifts that may be within noise. Use a stable benchmark and keep labels consistent so lift reflects prompt design, not dataset drift. Report both absolute lift and normalized lift to avoid misleading comparisons across tasks.
Balancing stability and safety
Stability captures repeatability across reruns, temperature settings, or slightly varied inputs. High stability reduces operational surprises, especially for customer-facing flows. Safety pass rate measures how often outputs meet policy or compliance checks. Weight these higher when the application is regulated, user-visible, or irreversible. Together they reward reliable behavior, not only peak accuracy on a single run. For chat systems, sample diverse user intents to measure stability realistically.
Latency and cost tradeoffs
Few-shot examples add tokens, which often increases latency and request cost. The calculator penalizes relative increases so your score drops when the prompt becomes too heavy for production. Penalty sensitivity lets you model stricter service-level needs. Track prompt tokens separately to spot context bloat early, and consider shorter demonstrations, caching, or selective routing when costs rise under peak load.
Using the score for iteration and monitoring
During iteration, run the same benchmark after each change and export results to build a simple prompt leaderboard. In deployment, re-evaluate periodically and compare scores across versions to detect drift from new data or model updates. If confidence adjustment is enabled, small test sets reduce the final score, encouraging larger evaluations before shipping. Use the grade and subscores to prioritize next improvements.
FAQs
What does the overall score represent?
It is a 0–100 index combining accuracy lift, stability, safety, latency, and cost using normalized weights, so you can compare prompt variants consistently.
Why normalize accuracy lift by headroom?
Headroom normalization avoids overstating gains when baseline accuracy is already high, making improvements more comparable across tasks with different starting performance.
How should I measure stability?
Run the same test set multiple times with fixed settings, then compute agreement or pass rate across runs. Use a representative mix of prompts and edge cases.
How are latency and cost penalized?
The calculator computes relative increases versus baseline and subtracts a scaled penalty from 100. Raising the sensitivity factor increases the penalty for production constraints.
What is the confidence adjustment?
It reduces the final score when the benchmark size is small, encouraging larger evaluations. Disable it if you already handle uncertainty with confidence intervals.
How do I use exports in my workflow?
Export CSV to track iterations in a sheet or dashboard, and export PDF for sharing. Keep scenario names consistent so comparisons remain clear over time.