Refua Bench
Test Refua model updates against saved baselines. Compare results with statistical checks and flag performance regressions before release.
What Is Refua Bench?
Refua Bench is a standalone benchmark and regression-gating project for Refua model workflows. It defines benchmark suites, runs evaluations through adapters, and checks model updates using statistical regression gates with minimum practical effect sizes and bootstrap confidence intervals.
Save named baselines and promote a new one after review. This makes changes in measured performance easier to spot.
Key Features
- YAML/JSON benchmark suite schema with tasks, metrics, and tolerances
- Adapter system: golden, file, command, and custom adapters
- Statistical regression gating with bootstrap CI
- Named baseline registry with safe promotion flow
- Automatic run provenance (git, runtime, model, deps)
- JSON + Markdown output artifacts
How It Works
Define Suite
Write a benchmark suite in YAML with tasks, metrics, expected values, and regression tolerances.
Run Benchmark
Execute the suite with a model adapter (file, command, golden, or custom) to produce a run artifact.
Compare & Gate
Compare the candidate run against a baseline with statistical gating and bootstrap confidence intervals.
Promote
If the candidate passes all gates, promote it as the new baseline in the registry.
Supported Metrics
MAE
Mean Absolute Error for continuous prediction tasks like affinity and binding energy estimation.
RMSE
Root Mean Square Error for regression tasks where large deviations are particularly penalized.
Accuracy
Classification accuracy for binary or multi-class prediction tasks.
Exact Match
Exact string or value match for deterministic outputs where precision matters.
F1 Score
Binary F1 score balancing precision and recall for imbalanced classification problems.
Get Started
Installation
pip install refua-benchRun + Gate in One Command
refua-bench gate \
--suite benchmarks/sample_suite.yaml \
--baseline benchmarks/sample_baseline_run.json \
--adapter file \
--adapter-config benchmarks/sample_file_adapter_config.yaml \
--min-effect-size 0.02 \
--bootstrap-resamples 1000Ship Models With Confidence
Use benchmark results and confidence intervals to decide whether a model update is ready to release.