The Holy Grail of Crypto AI: Cutting-edge Exploration of Decentralization Training

In the entire value chain of AI, model training is the most resource-intensive and technically challenging stage, directly determining the upper limit of the model's capabilities and its practical application effects. Compared to the lightweight calls in the inference phase, the training process requires sustained large-scale computational power investment, complex data processing workflows, and high-intensity optimization algorithm support, making it the true "heavy industry" of AI system construction. From the perspective of architectural paradigms, training methods can be divided into four categories: centralized training, distributed training, federated learning, and decentralized training, which this paper focuses on.

Centralized training is the most common traditional method, completed by a single organization within a local high-performance cluster, coordinating all training processes from hardware, underlying software, cluster scheduling systems, to components of the training framework through a unified control system. This deeply collaborative architecture optimizes the efficiency of memory sharing, gradient synchronization, and fault tolerance mechanisms, making it very suitable for training large-scale models like GPT and Gemini, with advantages of high efficiency and controllable resources. However, it also faces issues such as data monopolization, resource barriers, energy consumption, and single point risks.

Distributed training is the mainstream method for training large models at present. Its core is to break down the model training tasks and distribute them to multiple machines for collaborative execution, in order to overcome the bottlenecks of single-machine computing and storage. Although it physically possesses "distributed" characteristics, it is still overall controlled, scheduled, and synchronized by centralized institutions, often operating in high-speed local area network environments. Through NVLink high-speed interconnect bus technology, the main node uniformly coordinates each sub-task. Mainstream methods include:

• Data parallelism: Each node trains on different data while sharing parameters, requiring matching model weights.
• Model parallelism: Deploy different parts of the model on different nodes to achieve strong scalability.
• Pipeline parallelism: Executing in stages serially to improve throughput.
• Tensor parallelism: fine-grained segmentation of matrix calculations to improve parallel granularity

Distributed training is a combination of "centralized control + distributed execution," analogous to the same boss remotely directing multiple "office" employees to collaborate on completing tasks. Currently, almost all mainstream large models are trained using this method.

Decentralization training represents a future path with greater openness and anti-censorship characteristics. Its core feature lies in: multiple untrusted nodes collaboratively completing training tasks without a central coordinator, usually driven by protocols for task distribution and collaboration, and leveraging cryptographic incentive mechanisms to ensure the honesty of contributions. The main challenges faced by this model include:

• Difficulty in heterogeneous devices and task segmentation: High coordination difficulty of heterogeneous devices, low efficiency in task segmentation.
• Communication efficiency bottleneck: Network communication is unstable, and the gradient synchronization bottleneck is significant.
• Lack of Trusted Execution: The absence of a trusted execution environment makes it difficult to verify whether nodes are genuinely participating in the computation.
• Lack of unified coordination: No central scheduler, task distribution and exception rollback mechanisms are complex.

Decentralization training can be understood as: a group of global volunteers contributing computing power to collaboratively train a model. However, "truly feasible large-scale decentralization training" still poses a systematic engineering challenge, involving various aspects such as system architecture, communication protocols, cryptographic security, economic mechanisms, and model verification. The question of whether it can achieve "collaborative effectiveness + incentive honesty + correct results" is still in the early prototype exploration stage.

Federated learning, as a transitional form between distributed and Decentralization, emphasizes local data retention and centralized aggregation of model parameters, making it suitable for scenarios that prioritize privacy compliance. Federated learning has the engineering structure of distributed training and local collaborative capabilities, while also possessing the advantage of data dispersion from Decentralization training, but it still relies on trusted coordinating parties and does not have fully open and censorship-resistant characteristics. It can be viewed as a "controlled Decentralization" solution in privacy-compliant scenarios, relatively moderate in terms of training tasks, trust structures, and communication mechanisms, making it more suitable as a transitional deployment architecture in the industry.

Decentralization Training: Boundaries, Opportunities, and Realistic Paths

From the perspective of training paradigms, decentralization training is not suitable for all types of tasks. In certain scenarios, due to the complexity of task structures, extremely high resource demands, or difficulties in collaboration, it is inherently unsuitable for efficient completion across heterogeneous, trustless nodes. For example, large model training often relies on high memory, low latency, and high bandwidth, making it difficult to effectively segment and synchronize in an open network; tasks with strong data privacy and sovereignty restrictions are limited by legal compliance and ethical constraints, making them unable to be openly shared; and tasks that lack a foundation for collaborative incentives lack external participation motivation. These boundaries together constitute the current realistic limitations of decentralization training.

However, this does not mean that decentralized training is a false proposition. In fact, decentralized training shows clear application prospects in task types that are lightweight in structure, easy to parallelize, and incentivizable. This includes but is not limited to: LoRA fine-tuning, behavior-aligned post-training tasks, data crowdsourcing training and labeling tasks, resource-controllable small foundational model training, and collaborative training scenarios involving edge devices. These tasks generally exhibit high parallelism, low coupling, and tolerance for heterogeneous computing power, making them very suitable for collaborative training through P2P networks, Swarm protocols, distributed optimizers, and other means.

Decentralization Training Classic Project Analysis

Currently, representative blockchain projects in the forefront of decentralized training and federated learning mainly include Prime Intellect, Pluralis.ai, Gensyn, Nous Research, and Flock.io. In terms of technological innovation and engineering implementation difficulty, Prime Intellect, Nous Research, and Pluralis.ai have proposed a number of original explorations in system architecture and algorithm design, representing the cutting-edge direction of current theoretical research; while the implementation paths of Gensyn and Flock.io are relatively clear, with preliminary engineering progress already visible. This article will sequentially analyze the core technologies and engineering architecture behind these five projects, and further explore their differences and complementary relationships in the decentralized AI training system.

Prime Intellect: Training trajectories verifiable reinforcement learning collaborative network pioneer

Prime Intellect is committed to building a trustless AI training network that allows anyone to participate in training and receive credible rewards for their computational contributions. Prime Intellect aims to develop a verifiable, open, and fully incentivized AI Decentralization training system through the three main modules: PRIME-RL, TOPLOC, and SHARDCAST.

01、Prime Intellect Protocol Stack Structure and Key Module Value

The protocol stack of Prime Intellect includes three core modules: PRIME-RL, TOPLOC, and SHARDCAST, which respectively address the issues of asynchronous training, trusted verification, and weight propagation.

02, Detailed Explanation of Prime Intellect Training Key Mechanisms

PRIME-RL: Decoupled Asynchronous Reinforcement Learning Task Architecture

PRIME-RL is a task modeling and execution framework customized by Prime Intellect for decentralized training scenarios, specifically designed for heterogeneous networks and asynchronous participation. It adopts reinforcement learning as a priority adaptation object, structurally decoupling the training, inference, and weight uploading processes, allowing each training node to independently complete the task loop locally and collaborate with validation and aggregation mechanisms through standardized interfaces. Compared to traditional supervised learning processes, PRIME-RL is more suitable for implementing elastic training in environments without central scheduling, reducing system complexity while laying the foundation for supporting multi-task parallelism and policy evolution.

TOPLOC: Lightweight Training Behavior Verification Mechanism

TOPLOC is a core mechanism for verifiable training proposed by Prime Intellect, used to determine whether a node has truly completed effective policy learning based on observational data. Unlike heavyweight solutions such as ZKML, TOPLOC does not rely on full model recomputation; instead, it achieves lightweight structural verification by analyzing the local consistency trajectory between "observation sequence ↔ policy update." It transforms the behavioral trajectory during the training process into a verifiable object for the first time, which is a key innovation for realizing trustless training reward allocation and provides a feasible path for constructing auditable and incentivized Decentralization collaborative training networks.

SHARDCAST: Asynchronous Weight Aggregation and Propagation Protocol

SHARDCAST is a weight propagation and aggregation protocol designed by Prime Intellect, optimized for real network environments that are asynchronous, bandwidth-constrained, and have variable node states. It combines a gossip propagation mechanism with a local synchronization strategy, allowing multiple nodes to continuously submit partial updates while in unsynchronized states, achieving progressive convergence of weights and multi-version evolution. Compared to centralized or synchronized AllReduce methods, SHARDCAST significantly enhances the scalability and fault tolerance of decentralized training, serving as the core foundation for establishing stable weight consensus and continuous training iterations.

OpenDiLoCo: Sparse Asynchronous Communication Framework

OpenDiLoCo is a communication optimization framework independently implemented and open-sourced by the Prime Intellect team based on the DiLoCo concept proposed by DeepMind. It is specifically designed to address challenges commonly encountered in decentralized training, such as bandwidth limitations, device heterogeneity, and unstable nodes. Its architecture is based on data parallelism, constructing sparse topologies like Ring, Expander, and Small-World to avoid the high communication overhead of global synchronization, relying only on local neighboring nodes to complete model collaborative training. By combining asynchronous updates and checkpoint fault tolerance mechanisms, OpenDiLoCo enables consumer-grade GPUs and edge devices to stably participate in training tasks, significantly enhancing the accessibility of global collaborative training and serving as one of the key communication infrastructures for building decentralized training networks.

PCCL: Collaborative Communication Library

PCCL is a lightweight communication library tailored for decentralized AI training environments by Prime Intellect, aimed at addressing the adaptation bottlenecks of traditional communication libraries in heterogeneous devices and low-bandwidth networks. PCCL supports sparse topology, gradient compression, low-precision synchronization, and checkpoint recovery, and can run on consumer-grade GPUs and unstable nodes. It is a fundamental component supporting the asynchronous communication capabilities of the OpenDiLoCo protocol. It significantly enhances the bandwidth tolerance and device compatibility of training networks, paving the way for building a truly open and trustless collaborative training network by bridging the "last mile" of communication.

03, Prime Intellect Incentive Network and Role Division

Prime Intellect has built a permissionless, verifiable, and economically incentivized training network that allows anyone to participate in tasks and earn rewards based on real contributions. The protocol operates based on three core roles:

• Task initiator: define training environment, initial model, reward function, and validation criteria
• Training Node: Execute local training, submit weight updates and observation trajectories.
• Validator nodes: Use the TOPLOC mechanism to verify the authenticity of training behaviors and participate in reward calculation and strategy aggregation.

The core process of the protocol includes task publishing, node training, trajectory verification, weight aggregation, and reward distribution, forming an incentive closed loop around "real training behavior".

04, INTELLECT-2: The release of the first verifiable Decentralization training model.

Prime Intellect released INTELLECT-2 in May 2025, which is the world's first large-scale reinforcement learning model trained by asynchronous, trustless decentralized node collaboration, with a parameter scale of 32B. The INTELLECT-2 model was collaboratively trained by over 100 GPU heterogeneous nodes across three continents, using a fully asynchronous architecture, with a training duration of over 400 hours, demonstrating the feasibility and stability of asynchronous collaborative networks. This model not only represents a breakthrough in performance but also marks the first systematic implementation of Prime Intellect's proposed "training as consensus" paradigm. INTELLECT-2 integrates core protocol modules such as PRIME-RL, TOPLOC, and SHARDCAST, signifying that the decentralized training network has achieved openness, verifiability, and an economic incentive closed loop in the training process for the first time.

In terms of performance