I built Whetstone, an open source promotion gate for models, agents, prompts, and workflows.
The question it answers is:
Did the new version actually improve, or did the aggregate score hide something it broke?
Whetstone compares paired item-level outcomes, checks for hidden regressions, applies an exact significance test, and produces a reproducible PASS, HOLD, or BLOCK receipt.
In a recent local test, a baseline and routed candidate both scored 6/31. A normal scoreboard would have called them equal. Whetstone found three gains and three regressions, so it blocked the candidate.
You can inspect and test the public workbench here:
https://whetstone.cyberelf.link
The hosted version is stateless and intended for disposable or sanitized inputs. It does not contain a private exam bank. Real promotion tests run locally so that private items, model outputs, and evaluation data can remain inside your own environment.
I am looking for people who have something worth challenging, such as:
- a base model and fine-tune
- two checkpoints from the same training run
- a merged or routed model
- a quantized version that should preserve behavior
- an agent, prompt, or workflow update
- any candidate that scored better but may have traded away an existing ability
I am not looking for polished testimonials. I want people to attack the assumptions, try awkward cases, and show me where the gate gives the wrong answer or becomes impractical.
If you have a baseline and candidate pair you would be willing to test, reply here or message me. I can help structure the first local run while I improve the setup and documentation.
Source: