Covenant-72B: What the Biggest Decentralized AI Training Run Proved

Covenant-72B is a 72-billion-parameter language model pre-trained on about 1.1 trillion tokens by computers that joined freely over the internet, coordinated through Bittensor's subnet 3. Its March 2026 paper shows that permissionless training works at this scale: the base model lands close to Meta's 2023 Llama 2 70B on standard benchmarks. It does not show that decentralized training can reach today's frontier, that it costs less, or that a token captures the value.

Key takeaways

• At least 70 unique peers took part, no more than 20 per round, each expected to bring at least eight Nvidia B200 GPUs.
• Compressing updates more than 146-fold let peers sync over ordinary internet links while their GPUs stayed busy about 94.5% of the time.
• The base model beat Llama 2 70B on MMLU, 67.1 to 65.6, but trailed it on five of seven benchmarks.
• A month after the paper, the team announced it was leaving Bittensor. The result stands; the episode shows how much a subnet can depend on one team.

What was Covenant-72B?

Covenant-72B is a dense, Llama-3-style transformer with about 72.7 billion parameters, trained from scratch by Covenant AI with two co-authors from the Mila research institute. The team posted its technical report to arXiv on March 9, 2026, and released the base model, a chat version and intermediate checkpoints under the Apache 2.0 license.

The paper calls it "the largest collaborative globally distributed pre-training run" to date, by both compute and model scale. The two open runs it compares against, INTELLECT-1 (10 billion parameters) and Psyche Consilience (40 billion), admitted only whitelisted participants. Covenant's run was open to anyone who could do the work.

How did strangers train one model together?

Two pieces of software did the coordinating:

• Gauntlet, the incentive layer running on Bittensor, uses a validator to score each peer's update by how much it lowers the model's loss on small batches of data. It filters out copied or low-quality submissions and picks the updates that go into each round.
• SparseLoCo, the optimizer, lets each peer train on its own for 30 steps, then share a heavily compressed version of its changes instead of full gradients. The compression ratio is more than 146 to 1.

Peers exchanged those updates through Cloudflare's R2 cloud storage rather than directly. Each round gave them a 20-minute compute window, and syncing took about 70 seconds under a cap of 500 Mb/s down and 110 Mb/s up. That kept GPUs busy about 94.5% of the time, against about 82.1% for INTELLECT-1, by the paper's comparison.

Participation churned. An average of 24.4 peers were active per step and 16.9 had their updates selected, against a cap of 20. At least 70 unique peers contributed, a lower bound because Bittensor registrations can change hands.

The hardware was not hobbyist gear: each peer was expected to run at least eight Nvidia B200 data-center GPUs. The run was decentralized in who could join and how peers talked, not in what it took to join.

How good is the model?

Zero-shot results from the paper:

Against centralized models of similar size, it is competitive rather than better: ahead of Llama 2 70B on MMLU and ARC-Easy, behind on the other five tests. Against earlier decentralized runs, it is a large step up.

The paper adds two caveats of its own. The comparisons are "not fully controlled," with different data, tokenizers and training recipes. And much of the MMLU score arrived late: it rose from 62.5 to 67.1 during a final 14.2-billion-token phase on higher-quality data, about 27% of it instruction examples. A fine-tuned chat version scored 67.4 on MMLU and 63.9 on GSM8K, a grade-school math test.

What did the run prove, and what didn't it?

In our analysis, it settles one question and leaves three open.

What it proved: a 72-billion-parameter model can be pre-trained by peers nobody vetted in advance, over internet links, with an incentive system screening out bad work, and come out comparable to a centralized model of similar size. Our earlier look at decentralized AI training named communication overhead and verification as the main hurdles. This run cleared both at 72B scale.

What it didn't prove:

• The frontier. Its yardstick, Llama 2 70B, dates from 2023. Matching it is not the same as matching today's leading models.
• Cost. The paper reports no training cost, so there is no evidence yet that this approach is cheaper. Every peer still needed data-center GPUs.
• Value for a token. The weights are free to use under Apache 2.0. How much value reaches subnet token holders depends on emissions and pool prices, not benchmarks; our guide to Bittensor subnet tokens explains those mechanics.

What happened next?

The run became a market story. TAO rose about 90% in March 2026, and subnet 3's token, τemplar, gained 444% in 30 days, CoinDesk reported. The Block put its market cap at roughly $130 million on April 2.

On April 9, Covenant AI announced it was leaving Bittensor, saying the network's governance conflicted with its commitment to decentralization. Bittensor's co-founder denied the team's claims, Cointelegraph reported. TAO fell about 15% within two hours, according to The Block, which also reported that the team's decentralized training work would continue outside Bittensor.

Taostats now lists subnet 3 under a new name, Teutonic. The subnet's token was worth about $38 million on our tracker on October 16, 2026. The lesson for token holders is about dependence more than models: a subnet's value can rest on one team, and teams can leave. Our Bittensor explainer covers how subnets fit into the wider network.

What the AI Score shows

The AI Score does not grade models or benchmark tables. It rates tokens, including TAO and subnet tokens, from 0 to 100 on five questions: is anyone building it, can you get in and out, will unlocks dilute you, how badly could this hurt, and is the market agreeing. It is recalculated every 30 minutes from public market data (CoinGecko) and public code activity, with development counting most and momentum least. Model quality and who stands behind a subnet are yours to check: read the paper, see who ran the evaluations, and look up the subnet's status on taostats.

Check any token's AI Score on AiTokens.app before you decide.

Frequently Asked Questions

What is Covenant-72B? A 72-billion-parameter language model pre-trained on about 1.1 trillion tokens through Bittensor's subnet 3, with participation open to anyone who could contribute the compute. Its weights are public under the Apache 2.0 license.

Was Covenant-72B trained on home computers? No. Peers connected over ordinary internet links, but each was expected to bring at least eight Nvidia B200 GPUs, which are data-center hardware.

Is Covenant AI still on Bittensor? No. It announced in April 2026 that it was leaving over a disagreement about governance, and Bittensor's co-founder denied its claims. Taostats now lists subnet 3 as Teutonic.

Does a successful training run make a subnet token more valuable? Not automatically. The model is free to use, and a subnet token's price comes from its pool and emissions. Subnet 3's token rallied in March 2026, then traded at about $38 million in market value on October 16.

Sources

• arXiv, Lidin et al. (Covenant AI, Mila): Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet, March 10, 2026
• Hugging Face, Covenant AI: Covenant-72B model card, accessed October 16, 2026
• CoinDesk, Shaurya Malwa: Bittensor ecosystem tokens' value hits $1.5 billion as TAO rockets 90% in March, March 29, 2026
• The Block, Ivan Wu and Bryan Samsoedin: Bittensor breakout fuels AI token rally as distributed training gains credibility, April 2, 2026
• Cointelegraph, Zoltan Vardai: Covenant AI Leaves Bittensor Amid Decentralization Concerns, TAO Drops 18%, April 10, 2026
• taostats: Subnet 3, accessed October 16, 2026
• AiTokens.app: AI token tracker, market data from CoinGecko, snapshot of October 16, 2026, 20:44 UTC

This article is for educational purposes and is not financial advice.