Kaelum Bench v0.2 · Public Calibration
Methodology for public calibration.
A concise account of what the current calibration measures, how results are produced, and where interpretation must stop.
Current release
Results are provisional, not verified.
Configuration-specific comparisons describe the recorded model, provider route, and reasoning setup. They are not standardized cross-provider measurements.
- Suites
- 5 suites
- 4 suites with results
- Tasks
- 340 defined tasks
- 240 represented tasks
- Runs
- 33 completed calibration runs
- Single versioned runs
- Scoring modes
- 24 deterministic-only
- 9 hybrid
Task Design
Expert-designed tasks cover Accounting Knowledge, HGB Reporting, IFRS Reporting, Financial Analysis, and Accounting Capstone. The active evaluation material stays hidden; the public sample task is synthetic and excluded from scoring.
Each task fixes its input, required output shape, allowed tools, and scoring basis before a run begins.
Scoring
Deterministic checks are used where exact fields, calculations, and structured outputs can be evaluated directly. Accounting Knowledge uses hybrid scoring; deterministic checks and AI-judged rubric components contribute to its suite result.
No overall score or cross-suite aggregate is published.
Run Protocol
A public result represents one completed, versioned run under a recorded model and provider configuration. The current public runs use no tool access.
Comparisons remain configuration-specific. Single runs; no statistical ranking.
AI-Judge
Hybrid runs use an AI judge only for rubric-scored components that are not reduced to deterministic checks. Judge identity, usage, and retry aggregates are attached to the sanitized run evidence where the contract provides them.
AI-judge output does not change the provisional release status.
Contamination
Hidden evaluation material reduces direct task exposure but cannot prove that a model has never encountered related concepts. Public demonstration tasks are kept separate from the active hidden set and are not scored.
Limitations
Current results cover 4 suites and 240 represented tasks. Accounting Capstone is defined but has no public results. The release does not establish statistical rank certainty or standardized comparability.
It does not claim complete coverage of professional financial work or performance outside the recorded configuration.
The current scope does not test XLSX or workbook artifacts.
Release Status
Kaelum Bench v0.2 · Public Calibration includes 33 provisional results: 24 deterministic-only runs and 9 hybrid runs. No result is currently presented as verified.
The current release is authenticated by a production Ed25519 publication signature.
View the synthetic sample task →