Development of Web3 Data Access: Introduction to Indexers and Related Projects

Data is key to blockchain technology and is the foundation for developing decentralized applications ( dApp ). Although most discussions currently focus on data availability ( DA ), data accessibility is equally important yet often overlooked.

In the modular blockchain era, DA solutions have become indispensable. They ensure that all participants can use transaction data for real-time validation, maintaining the integrity of the network. However, the DA layer resembles a billboard rather than a database, as data is not permanently stored.

In contrast, data accessibility focuses on the ability to retrieve historical data, which is crucial for developing dApps and conducting blockchain analysis. Although there is less discussion about it, it is equally important as data availability. Both play different but complementary roles in the blockchain ecosystem, and a comprehensive data management approach must address both issues simultaneously.

Since its inception, blockchain has driven the creation of dApps in areas such as gaming, finance, and social networking. However, building these dApps requires access to a large amount of blockchain data, which is both difficult and costly.

For dApp developers, one option is to host and run their own archival RPC nodes. This allows for full access to data, but the maintenance costs are high and the querying capabilities are limited. Another approach is to use commercial RPC node providers. These providers are responsible for the costs and management of the nodes, providing data through RPC endpoints. However, even simple data retrieval requires a significant amount of communication, and the efficiency is low for complex queries.

Blockchain indexers play a key role in organizing on-chain data and sending it to databases for easier querying. They index blockchain data and make it available through a SQL-like query language. By providing a unified query interface, indexers allow developers to quickly and accurately retrieve the information they need, greatly simplifying the process.

Different types of indexers optimize data retrieval:

1. Complete Node Indexer: Extracts data directly from complete blockchain nodes, ensuring completeness and accuracy, but requires substantial storage and processing power.

2. Lightweight Indexer: Retrieves specific data from full nodes as needed, reducing storage requirements but potentially increasing query time.

3. Dedicated Indexer: Optimized for specific types of data or blockchain, such as NFT data or DeFi transactions.

4. Aggregated Indexer: Extracts data from multiple blockchains and sources, including off-chain information, providing a unified query interface.

Indexers allow for complex queries, easy data filtering, and post-analysis. Some indexers can also aggregate data from multiple sources, avoiding the need to deploy multiple APIs in multi-chain dApps. By being distributed across multiple nodes, indexers provide enhanced security and performance.

Overall, compared to RPC node providers, indexers improve the efficiency and reliability of data retrieval while reducing deployment costs. This makes blockchain indexer protocols the preferred choice for dApp developers.

Indexers have important applications in various fields:

DeFi protocols require different information to provide users with specific prices, rates, and fees. Automated market maker (AMM) needs price and liquidity information to calculate swap rates, while lending protocols need utilization rates to determine lending rates and the debt ratio for liquidation.

GameFi needs to quickly index and access data to ensure a smooth gaming experience for users. Only through lightning-fast data retrieval and execution can Web3 games compete with Web2 games in terms of performance. Indexers can ensure stable data flow and uptime.

NFT markets and lending platforms need to index various information, such as NFT metadata, ownership and transfer data, royalty information, etc. Quickly indexing such data can avoid browsing each NFT one by one.

The indexer also provides opportunities for more specific data analysis. For example, perpetual trading protocols can analyze token trading volume and fees, while DEX developers can gain insights into liquidity pool yields and liquidity. Public dashboards can also be created, allowing developers to flexibly query various types of data.

Currently, there are multiple blockchain indexers available in the market:

The Graph is the first indexing protocol launched on Ethereum. It uses subgraphs to define and filter subsets of blockchain data. Indexers stake GRT tokens to provide services, and delegators can choose to stake tokens. Its infrastructure brings the average cost of one million queries to $40, which is much lower than self-hosted nodes.

Subsquid is a decentralized data lake that efficiently aggregates on-chain and off-chain data. It supports real-time indexing and custom data format storage. Although still in the testing phase, it already has over 80,000 testnet users and 20,000 verified developers.

SubQuery provides RPC and indexing data services, supporting over 200 chains. It introduces a consumer role to submit purchase orders, ensuring income for indexers. Users can choose to pay approximately 1 SQT token per 1000 requests or set a custom fee.

Covalent is a decentralized indexing network that creates copies of blockchain data through batch exports. Developers can easily extract data through a unified API without having to write complex queries.

When choosing an indexer, the following factors need to be considered:

1. Data Customizability: Some indexers only provide standard datasets, while others allow for more customization.

2. Security: Index data must be secure and reliable. Different indexers employ different security measures, such as token staking, proof mechanisms, etc.

3. Speed and Scalability: As the blockchain grows, indexing efficiency becomes increasingly important. Different indexers adopt various methods to improve performance.

4. Supported Networks: With the rise of different blockchains, indexers that support diverse networks may have an advantage.

With the popularization of AI in Web3, the potential of indexers will further expand. AI applications require access to high-quality data to train models and develop agents. Some indexers have begun to explore integration with AI, such as building AI agents and providing dynamic pricing for query services.

The integration of AI and indexers is expected to enhance data accessibility and usability within the blockchain ecosystem. By leveraging AI technology, indexers can provide more efficient and accurate data retrieval, enabling developers to build more complex dApps and analytical tools. As these two technologies evolve together, the future of data indexing and its role in shaping the decentralized digital landscape is promising.