GSM8K run · maths word problems
First run: 1,350 maths problems, Llama models only, checked against known answers.
Refinement summary
Individual votes in, verified training data out, checked afterwards against official answers the nodes never saw.
Where every item stands
Each problem is answered by 3 different nodes. All agree → kept. Two agree → the judge decides. No agreement → thrown out.
Which model and prompt style is right most often
Each vote uses a random model and one of 5 prompt styles. Cells show how often that pair matched the official answer.
Technical details: nodes, judge, activity, settings
Nodes
Volunteer machines and the models each one runs. Numbers are for this run only. Credits come only from verified work: votes that matched a kept answer, plus hidden test problems passed.
| Node · votes per Llama | Status | Votes | Honeypots passed | Agreed | Too fast | Credits | Last seen |
|---|
Judge
A large Llama on Cloudflare settles 2-to-1 splits. It solves the problem itself; only if it lands on the majority answer is it kept.
Recent activity
Who worked on what, and how long it took. Answers are never shown, so nodes can't copy each other.
Rules of the network
The settings the coordinator is running with right now.
How this run worked
A coordinator on Cloudflare hands math problems to volunteer nodes on free Kaggle and Colab GPUs. For every problem, each node picks a random Llama model (1B, 3B or 8B) and a random instruction style, solves it, and signs the answer. The coordinator collects 3 independent answers, keeps the problem if they agree, and asks a 70B Llama judge when only two agree. Honeypots, timing checks and quarantine stop nodes from faking work. The kept problems, each with a full record of who answered what, become training data for a larger model. If small models agreeing can teach a bigger one, that's weak-to-strong generalisation (Burns et al., 2023).