The Innovative Path of Sharding Technology: Shardeum and Dynamic State Sharding

On September 15, 2022, Ethereum completed the Merge(. This is a historic moment that Ethereum has been preparing for over 5 years, having postponed it 6 times. Due to long-term development and debugging, coupled with the highly publicized halo effect, many people mistakenly believed that the merge would naturally bring higher scalability, security, and sustainability, but this is not the case. The transition from PoW) to PoS( merely changes the tracks and wheels; it does not directly lead to faster speeds, larger capacities, or lower fees. What can truly achieve these goals is a complete set of solutions: a mainnet with Sharding capabilities paired with Layer2 solutions that enhance scalability.

As pointed out by Ethereum founder Vitalik Buterin, Sharding is a scalability solution to the blockchain trilemma, which involves dividing the nodes in the network into smaller groups to handle different sets of transactions and achieve parallel processing. By sharing the burden of processing the large amount of data required to aggregate across the entire network, it can intuitively reduce waiting times and improve checkout efficiency, just like when we shop at Walmart and check out through multiple cash register lanes.

This is the logic of Sharding, direct and simple; however, the devil is in the details - the principles and direction are correct, but there are always many problems encountered in implementation. This article aims to clarify the direction and dilemmas on the path of "Sharding," creating a map for the Sharding explorer that looks up at the stars while keeping their feet on the ground. At the same time, by comparing existing Sharding solutions, it seeks to identify some common problems and propose a feasible exploration direction: Shardeum and dynamic Sharding.

1. About "Sharding"

In simple terms, considering the constraints of the impossible triangle, starting from Ethereum as the origin of the coordinate system ), 0(, we can categorize the current scalability methods of blockchain into two main types based on "vertical" and "horizontal" approaches:

Vertical Scaling ): Achieved by improving the performance of existing hardware in the system. Establish a decentralized network where each node in the network has supercomputing capabilities, meaning each node requires "better" hardware - this method is simple and effective, allowing for preliminary improvements in throughput, especially suitable for high-frequency trading, gaming, and other latency-sensitive application scenarios. However, this scaling method limits the level of decentralization of the network, as the costs of running validator nodes or full nodes increase. Maintaining the level of decentralization is constrained by the approximate growth rate of computing hardware performance (, which is known as "Moore's Law": the number of transistors on a chip doubles every two years, and the cost of computation is halved ).

Horizontal Scaling (: Horizontal scaling generally has several approaches. One approach is to distribute the transaction computation of a certain ecosystem across multiple independent blockchains within the context of blockchain, where each chain has its own block producers and execution capabilities. This method allows for full customization of the execution layer of each chain, such as node hardware requirements, privacy features, gas fees, virtual machines, and permission settings, etc. Another horizontal scaling solution is modular blockchain, which divides the blockchain infrastructure into execution layer, data availability layer ) DA (, and consensus layer. The most mainstream modular mechanism in blockchain is rollup. Another approach is to split a blockchain into many shards that execute in parallel. Each shard can be seen as a blockchain, meaning many blockchains can execute in parallel. Additionally, there is usually a main chain whose sole task is to keep all shards synchronized.

It should be noted that the above scalability approaches do not exist in isolation; each solution finds a trade-off point within the impossible triangle, combined with the incentive mechanisms created by economic forces in the system, to achieve an effective balance on both macro and micro levels.

To discuss "Sharding", we need to start from scratch.

Still assuming such a scenario, at Walmart checkout, in order to improve checkout efficiency and reduce customer wait times, we expand from a single checkout lane to 10 checkout windows. To avoid bookkeeping errors, we need to establish unified rules at this time.

First, if we have 10 cashiers, how should we assign them to work at which window?

Secondly, if we have 1000 customers queued up, how do we decide which window each customer goes to for checkout?

Third, how should the 10 separate ledgers corresponding to these 10 windows be summarized?

Fourth, how can we prevent cashiers from making mistakes in order to avoid discrepancies in accounts?

These questions actually correspond to several key issues in Sharding, which are:

How to determine which Sharding the nodes/validators of the entire network belong to? That is: how to perform network Sharding )NetworkSharding(;

How to determine which shard each transaction is assigned to? That is: how to perform transaction sharding )Transaction Sharding(;

How is blockchain data stored in different Shardings? That is: how to perform state sharding )State Sharding(;

Complexity means risk. Based on all of the above, how can we avoid the fragmentation of the entire system's security?

) 01 Network Sharding (

If we simply understand blockchain as a decentralized ledger, both PoS and PoW consensus mechanisms are designed to allow nodes to compete for the right to record transactions according to certain established rules, ensuring the accuracy of the ledger in the process. Sharding refers to the need for another established rule to partition the blockchain network, allowing each shard to handle on-chain transactions and compete for the right to record transactions while minimizing communication between them - that is, the grouping rules of the nodes.

The problem encountered in this process is that as the internal nodes of the blockchain are divided into different Shardings, the difficulty and cost for attackers will decrease sharply. We can infer that if the rules and results of this grouping process are fixed and predictable, then an attacker wanting to control the entire blockchain network only needs to focus on controlling one Sharding and can buy off some nodes within that Sharding.

Alexander Skidanov, the founder of Near, describes the problem this way: If a single chain with X validators decides to hard fork into a Sharding chain and divides the X validators into 10 Shards, each Shard now only has X/10 validators, and compromising one Shard only requires compromising 5.1% of the total number of validators, which is ) 51% / 10(. This leads to the second point: who selects validators for each Shard? It is only harmful to control 5.1% of the validators when all of these 5.1% are in the same Shard. If validators cannot choose which Shard to validate in, it is extremely unlikely that participants controlling 5.1% of the validators will place all validators in the same Shard, thus greatly reducing their ability to compromise the system.

The sharding system must develop a mechanism to trust that the network will not reverse these transactions from external shards. So far, the best answer may be to ensure that the number of validators within a shard is above a certain minimum threshold, thus making it very unlikely for dishonest validators to overpower a single shard. The most common approach is to build a certain degree of unbiased randomness, relying on mathematical methods to minimize the attacker's probability of success. For example, Ethereum's solution is to randomly select validators for a shard from all validators, and change the validators every 6.4 minutes at a length of ) one epoch.

In simple terms, it means randomly grouping nodes and then assigning work for each group of nodes to verify independently.

However, it should be noted that randomness in blockchain is a very challenging topic. Logically speaking, the process of generating this random number should not rely on the computation of any specific Sharding. For this computation, many existing design ideas involve developing a separate blockchain to maintain the entire network. Such chains are referred to as Beacon chains in Ethereum and Near, Relay chains in PolkaDot, and Cosmos Hub in Cosmos.

![Detailed Explanation of the New Public Chain Shardeum: Another Possibility of Sharding]###https://img-cdn.gateio.im/webp-social/moments-69c7de2bfe4ae7b233bec1f706fad9ad.webp(

) 02 Transaction Sharding (

Transaction sharding refers to the rules governing "which transactions are assigned to which shards," which can achieve the goal of parallel processing while avoiding the occurrence of double-spending problems. The different ledger models of blockchain can impact the development of transaction sharding.

Currently, there are two types of accounting methods in blockchain networks, namely the UTXO) Unspent Transaction Outputs model and the account/balance model. The former is typically represented by BTC, while the latter is represented by ETH.

UTXO Model: In BTC transactions, each transaction will have one or more outputs. UTXO refers to the outputs of blockchain transactions that have not been spent and can be used as inputs for new transactions, while spent transaction outputs cannot be spent again. This is similar to the payment and change scenario with paper money, where a customer pays the store with one or more bills, and the store gives change back to the customer with one or more bills. Under the UTXO model, transaction sharding requires cross-shard communication. A transaction may include multiple inputs and multiple outputs, with no concept of accounts and no record of balances. One possible method is to process it through a hash function based on a certain input value of the transaction to generate a discrete hash value to determine which shard the data should go to. As follows:

To ensure that entries are placed in the correct Sharding in a consistent manner, the values entered into the hash function should all come from the same column. This column is called the Shard Key. Subsequently, transactions resulting in a value of 1 are assigned to Shard 1, while transactions resulting in a value of 2 are assigned to Shard 2. The downside of this method is that communication between Shardings is necessary to avoid double spending attacks. If cross-Sharding transactions are restricted, it will limit the platform's availability, while allowing cross-Sharding transactions requires weighing the costs of cross-Sharding communication against the performance benefits it brings.

Account/Balances Model: The system records the balance of each account. When a transaction is made, the system checks whether the account has sufficient balance for payment, similar to how banks record the balance of each account during bank transfers. A transaction can only proceed if the account balance is greater than the required transfer amount. Under the account/balances model, since a transaction has only one input, as long as the transaction is sharded according to the sender's address, it can ensure that multiple transactions from the same account are processed within the same shard, effectively preventing double spending. Therefore, most blockchains that adopt sharding technology are account-based ledger systems like Ethereum.

( 03 状态Sharding)State Sharding###

State sharding refers to how blockchain data is allocated and stored across different shards.

Continuing with our Walmart queue example, each window has its own account; how do they record their ledgers? If a customer queues at a particular line, that account is recorded. For example, if customer A goes to window A, but the next day that customer goes to another checkout window, such as window B, and window B does not have that customer’s past account information (, such as related to stored value cards and other payment methods ), what should be done? Should window A's account information for that customer be called?

State sharding is the biggest challenge in sharding, more difficult than the previously mentioned network sharding and transaction sharding. Because under the sharding mechanism, transactions are assigned to different shards based on their addresses, meaning that the state will only be stored in the shard where its address is located. One problem that arises at this point is that transactions do not occur only within a single shard, often involving cross-sharding (Cross-Sharding).

Consider a transfer scenario where Account A transfers 10U to Account B. Account A's address is allocated in Sharding 1, and the transaction record will also be stored in Sharding 1. Account B's address is allocated in Sharding 2, and the transaction record will be stored in Sharding 2.

Once A wants to transfer funds to B, a cross-Sharding transaction will occur, and Sharding 2 will call past transaction records from Sharding 1 to verify the validity of the transaction. If A frequently sends coins to B, Sharding 2 must constantly interact with Sharding 1, which will reduce the efficiency of transaction processing. However, if participants do not download and verify the entire history of a specific Sharding, they cannot be sure that the state of their interactions is the result of some valid block sequences and that such block sequences are indeed the canonical chain within the Sharding.

Therefore, compared to a single chain without sharding, the sharding system faces new challenges for users.