Meta Global AI Hackathon · Indie track

Raw data in.
Refined data out.

Five small models check every item. Only what they agree on is kept.

Main result

–→–

wrong labels in the dataset, before and after refining

We hid wrong answers among labelled maths problems. The models never saw the given answers and solved each problem themselves. Live: updates as the test runs.

Input
–

labels, some secretly wrong

Errors corrected
–

corrupted labels identified and fixed

Errors remaining
–

wrong labels left in the kept data

Label error rate
–before
–after

View the full case study: input, method and every corrected label →
Search and refine the data you needDescribe it. An AI agent finds sources, collects a sample, sizes the job and quotes it.
Try it →

A little refined data, a better model

1,331refined headline labels
592refined maths answers
~4 minof training, 1 epoch

Llama 3.2 1B, trained only on refinery data, then tested on questions it never saw:

base modelfine-tuned on refined data
See the fine-tuned example →

Inputs and outputs, every run

RunItems inRefined outRejectedKept

Decision process

015 independent votes

No node sees another's answer.

024–5 agree → accepted

Strong consensus is kept.

033 agree → hint, re-vote

Muse Glimmer hints, never answers.

04≤2 agree → rejected

Unsure items never enter the data.

Architecture

1 · Assign2 · Vote3 · Count4 · Hint5 · Accept
Node 1Llama 3.2 3B Node 2Gemma 4 E4B Node 3Ministral 3 3B Node 4SmolLM3 3B Node 5Qwen 3.5 4B VOLUNTEER NODES Coordinator Cloudflare Worker waiting Vote ledger every vote, right or wrong JUDGE Muse Glimmer 30B only for 3-of-5 splits hint, never the answer OUTPUT Refined dataset answer + full vote record 0 items

The coordinator sends the same item to five independent nodes.

What happens to bad votesoutvoted, logged, never become data · click to expand
Never become dataA disagreeing answer is outvoted; only the consensus answer is written to the dataset.
Kept as evidenceEvery vote stays in the ledger and in each item's exported record, marked as disagreeing.
Count against the nodeOutvoted answers earn nothing. Repeat failures get a node blocked.
Useful laterWrong answers become "rejected" examples for preference training.

More agreement, more accuracy

500 maths problems, checked against official answers.

Maths word problems (GSM8K)

4 or 5 of 5 agree → right 99% of the time.

Before vs after refining

Share of answers that were correct, checked against official answers.

Research foundation

OpenAI · Burns et al., 2023 · arXiv:2312.09390
Weak-to-Strong Generalization
ShowsA strong model can learn from weaker models and beat them.
For usOur small models are the weak teachers; voting cleans their labels.
Google · Wang et al., 2022
Self-Consistency

Majority vote beats any single answer.
Google · Huang et al., 2022
LLMs Can Self-Improve

Training on high-agreement answers improves a model.
Tencent · Li et al., 2024
More Agents Is All You Need

More voters, higher accuracy.
MIT · Northcutt et al., 2021
Pervasive Label Errors

Even benchmarks contain wrong labels.
OpenAI · Cobbe et al., 2021
GSM8K

The maths benchmark with official answers we use to verify every result.
NYU · Rein et al., 2023
GPQA

Graduate-level questions written to resist search; our test of where small models stop.
Meta · Zhou et al., 2023
LIMA: Less Is More

1,000 good examples rival far larger datasets.
Chen et al., 2023
AlpaGasus

9k filtered examples beat the full 52k.
Microsoft · Gunasekar et al., 2023
Textbooks Are All You Need

Clean data lets small models beat big ones.
Hugging Face · Penedo et al., 2024
FineWeb

Quality filtering beats more raw text.
Meta · Prasad et al., 2024
Self-Consistency Preference Opt.

Majority vs minority answers as training pairs.
OpenAI · Lightman et al., 2023
Let's Verify Step by Step

Check every step, not just the answer.
Anti-cheating and integrity9 safeguards · honeypots, HMAC-SHA256 signatures, timing checks · click to expand

Every vote is untrusted until verified.

Honeypot questionshidden test items; nodes scoring below 30% are blocked
HMAC-SHA256 signaturesevery answer is signed with the node's secret key; forged or altered votes are rejected
Blind votingnodes only receive the question, never other answers, so they can't copy
Timing checksanswers faster than the model could compute are discarded as fake work
One vote per node per itemleases and duplicate checks stop a node from voting twice
Judges can't decidesigned hints only, with answers and letters scrubbed out automatically
Model diversityten model families, so no single model's blind spot can win
Credit for agreeing with consensusa vote earns credit when it matches what the network accepted. No answer key needed
Full audit trailevery signed vote is stored, so any result can be recounted
Infrastructure: demo today, production nextfree Cloudflare Workers + Supabase now · production-grade upgrade path · click to expand
Now (hackathon demo)Production upgrade
Coordinator

Cloudflare Workers, free tier

Paid Workers + Durable Objects, multiple regions

Database

Supabase Postgres, free tier

Managed Postgres with replicas, or a public vote ledger

Nodes

Free Colab and Kaggle GPUs

One-click client on volunteer PCs and partner GPUs

Judge

Muse Glimmer 30B on Kaggle

Dedicated judge pool, 30–70B models

Scale

~20,000 votes so far

Millions of votes per day across thousands of nodes

Free tiers keep the demo at zero cost. Same code scales up.

Future: decentralisation

Every idle GPU becomes part of the refinery.Home nodes, judge nodes, rewards, and a live cost calculator →

All experiments

01
Error detection main result
Labelled maths problems with deliberately wrong labels hidden among them.
–wrong labels found and fixed
→ 02
Maths, ten model families works
500 school maths problems with official answers.
–correct: one model → refinery
→ 03
Maths, Llama only first run
1,350 problems, three Llama sizes voting.
–correct: one model → refinery
→ 04
Graduate-level science paused early
Expert questions beyond what small models know.
–questions tested
→ 05
News sentiment no answer key
Indian small-cap headlines labelled positive, negative or neutral.
–headlines labelled with agreement
→ 06
Trading signals closed
Buy / sell / skip calls on Nifty 50 news, paper money.
–calls the models agreed on
→

Overall score

–→–

answers correct: one model alone → after the refinery

Items in
–

all tests

Kept
–

refined data out

Improvement
–

accuracy points gained

Votes
–

from all nodes, so far

Accuracy is checked on every test with known answers (maths and error detection). Science is excluded.