Covenant-72B
Covenant-72B is the largest permissionless collaboratively trained language model trained entirely from scratch at the 72 billion parameter scale.
It was trained with 20+ globally distributed participants coordinated via decentralized infrastructure on the Bittensor blockchain.
Training Details
| Property | Value |
|---|---|
| Model size | 72B |
| Architecture | LLaMA-style |
| Target token budget | 1.1T |
| Compute participants | 20+ |
| Minimal compute per participant | 8×B200 or equivalent |
| Dataset | DCLM-baseline |
| Optimizer | SparseLoCo (communication-efficient optimizer) |
Performance on Benchmarks
All results are 0-shot acc-norm (%) unless noted.
| Model | Compute Environment / Permissions | Size | Tokens | ARC-C | ARC-E | PIQA | OpenBookQA | HellaSwag | Winogrande (acc) | MMLU (acc) |
|---|---|---|---|---|---|---|---|---|---|---|
| Intellect-1 | Internet / Whitelist | 10B | 1T | 44.80 | 71.76 | 77.37 | 43.80 | 70.26 | 63.30 | 32.69 |
| Psyche Consilience-7Y9 | Internet / Whitelist | 40B | 1.2T | 31.14 | 55.77 | 76.12 | 35.20 | 63.67 | 56.99 | 24.23 |
| Covenant-72B (Checkpoint-Two) | Internet / Permissionless | 72B | 420B | 53.84 | 77.74 | 80.58 | 44.60 | 77.08 | 71.43 | 47.49 |
| Covenant-72B (base) | Internet / Permissionless | 72B | 1.1T | 56.48 | 79.76 | 80.90 | 44.80 | 78.07 | 73.24 | 61.00 |
| LLM360 K2 ckpt_108 | Centralized Cluster | 65B | 420B | 45.73 | 70.54 | 80.90 | 43.20 | 78.23 | 71.90 | 50.01 |
| LLM360 K2 Stage 1 | Centralized Cluster | 65B | 1.4T | 53.75 | 75.97 | 82.54 | 48.00 | 82.86 | 76.40 | 65.51 |
| LLaMA-2-7B | Centralized Cluster | 7B | 2T | 45.05 | 73.82 | 78.73 | 44.20 | 76.18 | 69.38 | 41.73 |
| LLaMA-2-70B | Centralized Cluster | 70B | 2T | 57.42 | 79.55 | 82.59 | 49.40 | 84.34 | 80.43 | 65.63 |
- Downloads last month
- 81
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support