The Evolution of Decentralization Storage: From Ideals to Reality

Storage was once one of the hottest tracks in the blockchain industry. Filecoin, as the leading project of the last bull market, had a market capitalization that once exceeded ten billion dollars. Arweave, with its selling point of permanent storage, reached a maximum market capitalization of 3.5 billion dollars. However, as the limitations of cold data storage were revealed, the necessity of permanent storage has been questioned, and the prospects of Decentralization storage have also been cast into doubt.

The emergence of Walrus has brought new vitality to the long-silent storage track. Now, Aptos has teamed up with Jump Crypto to launch Shelby, aiming to elevate decentralized storage in the hot data domain to new heights. So, can decentralized storage make a comeback and provide solutions for a wide range of application scenarios? Or is it merely another round of hype? This article will analyze the narrative evolution of decentralized storage by examining the development paths of four projects: Filecoin, Arweave, Walrus, and Shelby, and explore the future prospects of the popularization of decentralized storage.

Filecoin: The Name of Storage, The Reality of Mining

Filecoin is one of the early rising cryptocurrency projects, and its development direction naturally revolves around Decentralization. This is a common characteristic of early crypto projects - seeking the significance of Decentralization in various traditional fields. Filecoin is no exception; it links storage with Decentralization, thereby naturally pointing out the drawbacks of centralized data storage: the trust assumption on centralized storage service providers. Therefore, the goal of Filecoin is to shift centralized storage to Decentralized storage. However, certain aspects sacrificed for the sake of Decentralization later became pain points that projects like Arweave or Walrus attempted to address. To understand why Filecoin is essentially just a mining coin, one needs to grasp the objective limitations of its underlying technology IPFS, which is not suitable for handling hot data.

IPFS: Decentralization architecture, but limited by transmission bottlenecks

IPFS( InterPlanetary File System ) was introduced around 2015, aiming to disrupt the traditional HTTP protocol through content addressing. The biggest drawback of IPFS is its extremely slow retrieval speed. In an era where traditional data service providers can achieve millisecond-level responses, retrieving a file with IPFS still takes over ten seconds, making it difficult to promote in practical applications, and explaining why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is mainly suitable for "cold data", which refers to static content that does not change often, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic web pages, online games, or artificial intelligence applications, the P2P protocol does not have a significant advantage over traditional CDNs.

Although IPFS itself is not a blockchain, its directed acyclic graph (DAG) design concept aligns closely with many public chains and Web3 protocols, making it inherently suitable as a foundational building framework for blockchains. Therefore, even if it has no practical value, it is sufficient as a foundational framework that carries the blockchain narrative. Early crypto projects only needed a functioning framework to realize their grand visions, but as Filecoin developed to a certain stage, the limitations brought by IPFS began to hinder its progress.

The logic of mining coins under the storage cloak

The original intention of IPFS was to allow users to store data while also being part of the storage network. However, without economic incentives, it is difficult for users to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS but will not contribute their own storage space or store others' files. It is in this context that Filecoin was born.

In the token economic model of Filecoin, there are three main roles: users are responsible for paying fees to store data; storage miners receive token incentives for storing user data; and retrieval miners provide data when users need it and receive incentives.

This model has a potential space for malicious behavior. Storage miners may fill up garbage data after providing storage space in order to receive rewards. Since this garbage data will not be retrieved, even if they are lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. Filecoin's proof of replication consensus can only ensure that user data has not been privately deleted, but cannot prevent miners from filling up garbage data.

The operation of Filecoin largely relies on miners' continuous investment in the token economy, rather than on the actual demand for distributed storage from end users. Although the project is still iterating, at this stage, the ecological construction of Filecoin aligns more with the "mining coin logic" rather than the definition of a "application-driven" storage project.

Arweave: Success through long-termism, failure through long-termism

If the design goal of Filecoin is to build an incentivized, provable Decentralization "data cloud" shell, then Arweave takes an extreme direction in storage: providing the ability for permanent data storage. Arweave does not attempt to build a distributed computing platform; its entire system revolves around one core assumption - important data should be stored once and remain on the network forever. This extreme long-termism makes Arweave vastly different from Filecoin in terms of mechanisms, incentive models, hardware requirements, and narrative perspectives.

Arweave uses Bitcoin as a learning object, attempting to continuously optimize its permanent storage network over long periods measured in years. Arweave does not care about marketing, nor does it care about competitors and market development trends. It is simply moving forward on the path of iterating its network architecture, indifferent to whether anyone pays attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave was warmly embraced during the last bull market; and because of long-termism, even if it falls to the bottom, Arweave may still withstand several rounds of bull and bear markets. The only question is whether there will be a place for Arweave in the future of Decentralization storage. The existence value of permanent storage can only be proven over time.

Since version 1.5, the Arweave mainnet has been committed to allowing a wider range of miners to participate in the network at minimal cost, and to incentivizing miners to maximize data storage, despite losing market attention until the recent version 2.9. Arweave has taken a conservative approach, fully aware that it does not align with market preferences, refraining from embracing the miner community, leading to a complete stagnation of the ecosystem. The mainnet has been upgraded at minimal cost, continuously lowering hardware thresholds without compromising network security.

A review of the upgrade path from 1.5 to 2.9

Version 1.5 of Arweave exposed a vulnerability where miners could rely on GPU stacking instead of real storage to optimize block creation chances. To curb this trend, version 1.7 introduced the RandomX algorithm, limiting the use of specialized computing power and requiring general-purpose CPUs to participate in mining, thereby weakening the centralization of computing power.

In version 2.0, Arweave adopts SPoA, transforming data proof into a concise path of Merkle tree structure, and introduces format 2 transactions to reduce synchronization burden. This architecture alleviates network bandwidth pressure, significantly enhancing the collaborative capability of nodes. However, some miners can still evade the responsibility of holding real data through centralized high-speed storage pool strategies.

To correct this bias, 2.4 introduced the SPoRA mechanism, which incorporates global indexing and slow hashing random access, requiring miners to genuinely hold data blocks to participate in effective block production, thus weakening the effects of hash power stacking from a mechanistic perspective. As a result, miners began to focus on storage access speed, promoting the application of SSDs and high-speed read/write devices. 2.6 introduced hash chain control to regulate the block production rhythm, balancing the marginal benefits of high-performance devices and providing a fair participation space for small and medium-sized miners.

Subsequent versions further enhance network collaboration capabilities and storage diversity: 2.7 introduces collaborative mining and pool mechanisms to improve the competitiveness of small miners; 2.8 launches a composite packaging mechanism, allowing large-capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in the replica_2_9 format, significantly improving efficiency and reducing computational dependence, completing the closed loop of data-driven mining models.

Overall, Arweave's upgrade path clearly presents its long-term strategy oriented towards storage: while continuously resisting the trend of computational power centralization, it consistently lowers the participation threshold to ensure the possibility of long-term operation of the protocol.

Walrus: Is embracing hot data hype or is there more to it?

From a design perspective, Walrus is completely different from Filecoin and Arweave. The starting point of Filecoin is to create a decentralized and verifiable storage system, at the cost of cold data storage; the starting point of Arweave is to create an on-chain library of Alexandria that can permanently store data, at the cost of having too few use cases; the starting point of Walrus is to optimize the storage costs of hot data storage protocols.

Magic Modified Error Correction Code: Cost Innovation or Old Wine in a New Bottle?

In terms of storage cost design, Walrus believes that the storage overhead of Filecoin and Arweave is unreasonable, as both adopt a fully replicated architecture. Their main advantage lies in the fact that each node holds a complete copy, providing strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still maintains data availability. However, this also means that the system requires multiple copies for redundancy to maintain robustness, thereby increasing storage costs. Especially in the design of Arweave, the consensus mechanism itself encourages node redundancy storage to enhance data security. In contrast, Filecoin offers more flexibility in cost control, but at the cost of potentially higher data loss risks for some low-cost storage options. Walrus attempts to find a balance between the two, controlling replication costs while enhancing availability through structured redundancy, thus establishing a new compromise path between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology for reducing node redundancy, which originates from Reed-Solomon(RS) coding. RS coding is a very traditional error correction code algorithm, and error correction codes are a technology that allows for doubling the dataset by adding redundant fragments(erasure code) to reconstruct the original data. From CD-ROMs to satellite communications to QR codes, it is frequently used in daily life.

Erasure codes allow users to take a block, for example, 1MB in size, and then "expand" it to 2MB, where the additional 1MB is special data known as erasure codes. If any byte in the block is lost, users can easily recover those bytes using the codes. Even if up to 1MB of the block is lost, you can still recover the entire block. The same technology allows computers to read all the data on a CD-ROM, even if it has been damaged.

The most commonly used is RS coding. The implementation method starts with k information blocks, constructs the related polynomial, and evaluates it at different x coordinates to obtain the coded blocks. Using RS erasure codes, the probability of randomly sampling large chunks of data loss is very small.

For example: Divide a file into 6 data blocks and 4 parity blocks, totaling 10 parts. As long as any 6 parts are retained, the original data can be completely restored.

Advantages: Strong fault tolerance, widely used in CD/DVD, fault-tolerant hard disk arrays ( RAID ), as well as cloud storage systems ( such as Azure Storage and Facebook F4).

Disadvantages: decoding calculations are complex and the overhead is high; not suitable for scenarios with frequent data changes. Therefore, it is typically used for data recovery and scheduling in off-chain centralized environments.

Under the Decentralization architecture, Storj and Sia have adjusted traditional RS coding to meet the actual needs of distributed networks. Walrus has also proposed its own variant - the RedStuff coding algorithm, to achieve lower costs and more flexible redundancy storage mechanisms.

What is the biggest feature of Redstuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed and stored in a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This is made possible while maintaining a replication factor of only 4 to 5 times.

Therefore, define Walrus as a circle