Evolution History of Web3 Decentralized Storage

With the development of blockchain applications in NFT, games, and Metaverse, the data format has become diversified and the data scale has also increased significantly. It was also during this period that more people began to pay attention to and value decentralized storage . In the first quarter of 2021, the concept of NFT exploded completely, and all currencies in the storage sector achieved a 10-fold increase during the same period. Among them, FIL rose from $21 to $238, and AR rose from $2.6 to $91. The storage sector not only outperformed the DeFi sector, but also outperformed BTC.

In fact, decentralized storage is one of the earliest and most concerned infrastructures. The earliest decentralized storage solution Storj and the IPFS protocol were launched in 2017. After 5 years of development, the effective storage capacity of Filecoin's entire network is 59.6 PB, and the total storage capacity of Arweave has reached 76.3 TB. According to Web3 Index statistics, Arweave's storage fee in the past 90 days was $185,000, and Storj's was $55,000. It can be seen that regardless of storage scale or performance, decentralized storage is still in its infancy and is far from centralized storage.

In March 2021, the original file of the NFT album sold by musician 3LAU on NiftyGateway for $11 million was lost, and this is not an isolated case. The loss of the original NFT file made people realize that the decentralization of NFT asset certificates does not mean the security of the content itself. NFTs lose their meaning if the file itself is lost. Decentralized storage is an indispensable infrastructure for Web3. In the next cycle, with the penetration of blockchain in various applications, the demand for decentralized storage will undoubtedly usher in a substantial increase.

Looking back at the development process of decentralized storage, different problems have been solved at different stages. Storj and Sia use smart contracts to achieve P2P supply and demand matching, which is suitable for large-scale data storage, but its essence is the docking of storage resources. It lacks effective content addressing methods and is not conducive to file sharing (such as movies, audio, etc. IPFS realizes File addressing is a major innovation to the underlying protocols of the Internet. Projects such as Filecoin and Crust have built IPFS storage networks through incentive layers to provide stable decentralized storage services. Arweave achieves permanent storage of files through innovative consensus mechanisms. Last year The online Ceramic database storage protocol can meet high-frequency data needs. Stratos, which has been launched on the test network, has launched a Web3 decentralized infrastructure that integrates blockchain, storage, and computing.

The Evolution Process of Decentralized Storage

In this report, we selected some representative storage projects: Storj, Filecoin, Arweave, Stratos, Ceramic, summarized and compared their performance, cost, market positioning, market value and other information, and compared the information of each project The technical principles and ecological progress are summarized.

1. Horizontal comparison of storage items

Storj is a P2P cloud storage marketplace for individual and business-level customers, with tokens used to pay for it. IPFS and Ceramic are storage protocols, so these two projects have no tokens, and the incentive layer built on the basis of the protocol has corresponding tokens. At present, the highest circulating market value is Filecoin, which is 1.2 billion US dollars, and the lowest is Stratos, which is currently in the testnet stage, which is 3.9 million US dollars.

As can be seen from the storage costs, the storage costs of Storj and Filecoin are lower than those of AWS cloud storage. Arweave focuses on permanent storage (at least 200 years), and all three projects are converted into storage for 200 years. Storj is $9.36, and Filecoin is $0.00024.

Every project except Storj has a protocol layer. The application side directly adopts the storage protocol, and needs to deploy the storage server by itself, and adjust it as the business and bandwidth requirements change. The function of the incentive layer is to build a decentralized storage network through the consensus mechanism, and the application party can use the decentralized storage service by paying the fee. One of the more special ones is Stratos. In its architecture, storage and computing use the same blockchain as the incentive settlement layer, which means that Stratos can provide applications with storage and computing infrastructure services.

Different applications have different storage requirements. Text, pictures, audio, and video are static data storage, while applications such as identities and social graphs that require frequent data recording, reading, and modification require database storage. Currently, projects that can provide database storage include Ceramic and Stratos.

2. Main storage ecological status Storj, Filecoin, Arweave, Stratos, Ceramic

Storj: A distributed cloud storage protocol based on Ethereum

Storj is an Ethereum-based distributed cloud storage protocol developed by the for-profit company Stroj Labs. Users can use its platform token $STORJ to purchase storage services on the Storj platform, and its model is similar to Airbnb and Uber. Users use idle storage space to provide storage services and get $STORJ rewards. Compared with centralized cloud storage services, Storj's distributed cloud storage can provide users with higher security and privacy, and provide cheaper prices by utilizing idle storage resources.

Storj focuses on enterprise-level storage services, benchmarking Amazon's S3 service, and has established a partnership with Microsoft Azure to deploy some of their development tools. So far, Storj has more than 13,000 nodes and rents out the network to thousands of users. Recently Storj has greatly improved the performance of video storage and management.

Storj's technical architecture revolves around four operations:
1) Store data: When a user stores data, the client encrypts the data and breaks it into multiple fragments. These shards are distributed to nodes through the network. At the same time, the client generates metadata that contains information about where to find the data again.
2) Retrieving data: When a user retrieves data, the client will refer to metadata to identify the location of previously stored chunks, then retrieve these fragments, and reassemble the original data on the client's local machine.
3) Maintain data: When the redundancy of data is below a certain threshold, the necessary data for the missing part will be regenerated and replaced.
4) Payment: Users pay for the services provided by the network.

There are three main types of roles in the Storj ecosystem: client, node, and satellite. A Storj satellite is a server cluster that connects clients and nodes. When the client needs to upload a file, the satellite will help the client find the node with the fastest upload speed, and record the expenditure and income of the client and the node at the same time. The data uploaded at the same time is encrypted, and only the owner has the key, so others cannot decrypt the data. The existence of satellites makes Storj relatively decentralized in the network structure.

Storj focuses on providing high SLA service standards and high stability performance to meet the high requirements of enterprise-level storage, and reduces high costs to meet the discerning practical needs of the long-tail enterprise market. Storj also has more advantages than other decentralized storage projects in terms of commercial implementation.

Filecoin: Let IPFS become a new generation of Internet underlying protocol

Launched in 2015, the IPFS protocol is a decentralized network underlying protocol that aims to replace the current Internet Hypertext Transfer Protocol (HTTP). IPFS is based on the storage mode of content addressing, and the same files will not be stored repeatedly, which greatly saves storage space. In addition, IPFS is based on a P2P network, which can download data from multiple nodes concurrently, thus saving bandwidth costs by nearly 60%. The historical version backtracking function of IPFS makes it easy to view the historical version of the file, and the data cannot be deleted and can be stored permanently. IPFS built on a decentralized distributed network is difficult to be centrally managed and restricted. The Internet will be more open and can greatly reduce the dependence on the central backbone network. IPFS is a brand-new data storage and transmission method that can make web pages faster, safer, more robust, and more durable. It also allows people to have one more choice besides the HTTP protocol.

Just like anyone can build a website using the HTTP protocol, people can build a website using IPFS. To build a website using the IPFS protocol, the project party needs to run several seed storage nodes by itself. However, if the amount of data is too large, the number of nodes is insufficient, and the number of visits is low, the redundancy of data storage will not be guaranteed, resulting in poor website experience. In addition, it is not easy to get people to abandon HTTP and turn to IPFS. Peter Thiel mentioned in "From 0 to 1" that in order to occupy the market, technologically innovative companies need to make breakthroughs to solve existing problems. He set the "breakthrough" standard to be 10 times better. Therefore, the purpose of Filecoin is to motivate a large number of nodes around the world to provide users with storage and retrieval services, and to promote the widespread use of the IPFS file storage and transfer protocol. Filecoin defines an incentive mechanism for storage services and the corresponding payment behavior, which solves the problem of how to reward storage behavior or retrieval behavior. At the same time, Filecoin also solves the trust problem in the decentralized network, that is, how to ensure that nodes store content correctly.

Filecoin has been the darling of the market since the beginning. In 2017, Filecoin completed the high-profile super ICO, raising 257 million US dollars within 30 minutes. YC Incubator, Sequoia Capital and other institutions participated in the early investment. In October 2020, the Filecoin mainnet was officially launched, and the market value rose to more than 59 billion US dollars. From the perspective of storage scale, Filecoin is a well-deserved leader. According to the data collected in the Messari report, as of the second quarter of 2022, network storage capacity will exceed 16 EB, an increase of 7% from the previous quarter, and active transaction storage will exceed 115 PB, an increase of 128% from the previous quarter. In the first quarter of 2022, Filecoin's revenue exceeded $9 million, the most among projects included in the Web3 Index.

As of the second quarter of 2022, the Filecoin network storage capacity exceeds 16 EB

In the fourth quarter of 2021, Filecoin has transitioned from Filecoin Regular (unverified data transactions) to primarily Filecoin Plus (verified data transactions). Filecoin Plus further enables two major services: NFT.Storage and Web3.Storage, which aim to provide essentially free storage on Filecoin.

NFT.Storage is a service that provides a simple user interface dedicated to storing NFT content and metadata on Filecoin. It brings NFT permanent storage to platform users ranging from individuals to OpenSea, MakersPlace, MagicEden, Holaplex, Jigstack, and Project Galaxy.

Similar to NFT.Storage, Web3.Storage simplifies the process of storing and retrieving Web3 data on Filecoin for developers and end users. Its focus is to allow developers to easily integrate Filecoin storage into their applications.

By participating in the Filecoin Plus incentive program, both Web3.Storage and NFT.Storage are able to provide storage on the Filecoin network at no additional cost to end users. Overall, both services experienced significant growth in Q2 2022, with Web3.Storage up 25% and NFT.Storage up 39% quarter-on-quarter.

Arweave - Web 3 persistent storage network

Arweave is not based on the incentive layer of the existing distributed storage scheme, but integrates data storage and incentives, focusing on the permanent storage of data. Users can use the storage service provided by Arweave for at least 200 years through a one-time payment .

Arweave's main network was launched in June 2018, and it has won the support of some public chain parties before 2021, using it as the main solution for main chain data storage. Solana and Nervos have adopted Arweave as their data storage solution in 2020. With the explosion of the NFT track in 2021 and the prosperity of the Web3 creator economy, Arweave's storage business has also exploded. According to Viewblock data, the current storage capacity of Arweave reaches 74.5T, which has increased by more than ten times in the past year. Data storage on the Arweave network grew 977% year-over-year in Q1 2022, thanks in large part to Bundlr, which accounted for more than 70% of Arweave transactions.

According to Viewblock data, Arweave currently has a storage capacity of 74.5T

Arweave issued AR tokens as incentives. Users need to pay fees through AR for using Arweave services. Miners get rewards by competing for bookkeeping rights, and encapsulate users' requests for storing data into new blocks. The block structure of Arweave is called Blockweave, which is different from the single-chain structure of other projects. Each block of Arweave needs to connect the previous block and recall block (Recall Block). Blockweave is scalable and sharded. Recall Block refers to the past historical block, which determines the new block generated next.

Arweave does not require miners to save all block records, but encourages miners to save as many blocks as possible, especially to save a small number of blocks, through Succinct Proof of Random Access (SPoRA). The network will select a Recall Block, and the miners who save this Recall Block will fight for the bookkeeping right by finding random numbers through PoW. Since the selection of the Recall Block is random, any historical block may become the target. If the number of participants in the final PoW competition is small, the user has a greater probability of being rewarded. Therefore, in order to increase the probability of block generation, miners need to save as many blocks as possible, or focus on selecting and storing less blocks.

Each block of Arweave should be connected to the previous block and recall block (Recall Block)

The advantage of this proof is that once data is stored in the blockchain, it is permanently stored and cannot be modified. At the same time, the SPoRA mechanism reduces the weight of PoW in the previous probability of miners producing blocks, and adds the dimension of data access speed. In addition, SPoRA allows miners to focus more on maintaining their own local hardware and nodes, preventing miners from storing data in the same low-rate data center, which facilitates the diversification of node locations and improves the decentralization of the entire network. degree of transformation.

Arweave ecology: far more than simple storage

1. Storage service platform and application

In addition to being a permanent storage tool, Arweave's greater potential lies in the fact that it has gradually built a set of storage public chain applications based on Permaweb's protocol ecology and SmartWeave smart contract ecology around its underlying storage services. Arweave has provided developers with underlying architectures such as Smartweave, Redstone, Bundles, etc., and the ecology is also growing. According to the data of ArweaveApps, the number of Arweave applications that have been included has reached 400+.

Arweave ecological application

permaweb provides a web2-like user experience. On top of Arweave's core data storage layer, Permaweb is a collection of interconnected files and applications that can be used to store a variety of information: from Wikipedia, to PDFs, to videos, and various dApps, such as Uniswap. Permaweb is similar to Web2's network, but the key difference is that all content on Permaweb is permanent. At the same time, Arweave's decentralized applications are running on the Permaweb layer, on which third-party project developers can create their own Dapps for users to use, thus forming an application platform.

Permaweb runs decentralized applications on top of Arweave's core data storage layer

Bundlr : Bundlr is Permaweb's second-tier solution, providing simpler and faster multi-chain permanent storage services, compatible with Ethereum, Solana, Polygon and other chains. Bundlr currently accounts for over 60% of data uploaded to Arweave, and as Bundlr scales, the network will be able to handle multiple orders of magnitude more data. Since the beginning of this year, Bundlr Network has successively integrated with Polygon, Arbitrum, zkSync, Avalanche and other networks. Through the Bundlr Network protocol, these networks can use any token to pay for storage fees.

Kyve Network : KYVE network can be regarded as the middle layer of Arweave's cross-chain storage. Through KYVE, the data of other blockchains can be permanently stored on Arweave as a backup. Different blockchains can also exchange data through the KYVE network. In addition, KYVE guarantees the accuracy of the uploaded data through the storage pool. The uploader is responsible for storing the latest data, and the verifier ensures the accuracy of the uploaded data. Both parties pledge KYVE tokens to the storage pool to ensure accuracy and fairness.

ArDrive : ArDrive can be regarded as a "Baidu Netdisk" stored on Arweave. It is a set of applications suitable for PC and mobile terminals. It can upload various forms of data such as text, pictures, and videos to Arweave for permanent storage. At present, ArDrive is a one-time payment and can be used permanently. The storage price is about 20 US dollars for 1G capacity, which is slightly higher than the current traditional storage price, but its permanent storage and data security value far exceed traditional storage methods.

2. Create profit-sharing tokens for smart contracts

Arweave also has its own ERC20 token format. This new token standard is called Profit Sharing Token (PST).

Developers can set the corresponding PST token when deploying smart contracts, and agree that PST holders will share proportionally the gas fees generated by the contract in the future. Therefore, the more a smart contract deployed on Arweave is used, the higher the intrinsic value of its PST token.

At present, many native Arweave ecological projects have issued their own PST tokens, such as VRT issued by Verto, a trading platform built on Permaweb, and ARDRIVE issued by ArDrive, an Arweave-based network disk application, all of which belong to the PST model. .

3. NFT issuance

NFT trading platform can be built directly on Arweave. Pianity is a music NFT casting and trading platform, which can help creators to NFT their music works and sell them to generate continuous income.

Pianity : As an NFT casting and trading platform, Pianity uses a new NFT standard based on Arweave, which fundamentally solves the problem of off-chain storage of NFT metadata. The process of minting NFT is to send a transaction to the NFT contract on Arweave, and save the audio file of the song to the chain while minting a new NFT. While enjoying low-cost on-chain storage, it also further reduces the user's trust cost in the process of trading NFT. Users no longer need to understand complex concepts such as metadata and various storage methods, making the transaction process realize what you see is what you get, and truly achieve trustless transactions.

Arweave will become an important infrastructure of Web 3

The permanent storage solution provided by Arweave is unique in the decentralized storage track. The flourishing of public chains in the Web3 era, the demand for permanent storage of NFT projects, and the prosperity of the creator economy will all promote the further development of Arweave. Arweave has gradually built a set of protocol ecology based on its Permaweb and SmartWeave smart contract ecology around its underlying storage service. It is developing towards the Web3 full-stack protocol and will become an important infrastructure of Web3. Compared with Filecoin, Arweave's mechanism is simpler in concept and technology, and its development and implementation are more friendly to users and miners.

Of course, Arweave also currently has insufficient data privacy. The solution for storing data on the chain determines that data cannot be changed once uploaded, resulting in high iteration costs of the program and insufficient mobility. As a storage solution with a public chain nature, Arweave needs to be continuously verified for its security, interoperability, and storage scale.

Stratos - Next Generation Decentralized Data Network

In addition to providing decentralized storage solutions, Stratos also provides database and computing services based on a powerful decentralized platform and native blockchain with high throughput. Stratos is therefore a decentralized network infrastructure. In terms of storage, what Stratos aims to achieve is that users can not only store data, but also use data. Stratos has received investment from Fundmental Labs, Kenetic, Fenbushi Capital, Assembly Partners and other institutions, and opened the storage test network in February this year, and the storage layer main network is expected to go online this year.

Stratos' network consists of four modules and three layers, providing developers with the lowest-level infrastructure. The four modules are blockchain, decentralized storage, decentralized database and decentralized computing. The three layers are value layer, resource layer and metadata routing layer, each layer is associated with a different consensus method. The value layer uses Proof-of-Stake (PoS) consensus, the resource layer uses Proof-of-Traffic (PoT) consensus, and the metadata routing layer uses Proof-of-Authority (PoA) consensus.

Stratos' network consists of four modules and three layers

The specific operation process of the three levels is: the metadata routing layer is responsible for matching the user's service request with the nodes of the resource layer empowered by the algorithm; the resource layer provides storage or computing services after receiving the task, and finally, the value layer serves as the settlement layer, through a specific mechanism to calculate the workload of the metadata layer and the resource layer, and then distribute the rewards to the resource layer. At the same time, the PoA consensus adopted by the metadata routing layer is the closest to the centralized consensus algorithm, which can ensure that the metadata routing layer is still efficient when it is decentralized. PoS can guarantee the security and environmental protection of the value layer, and PoT can promote the scalability of the resource layer.

The design of Stratos' three levels and three consensus uses clever methods to solve the decentralized impossible triangle problem to the greatest extent

The design concept of complete decentralization cannot coexist with safety and efficiency. The design of Stratos' three levels and three consensuses solves the decentralized impossible triangle problem in a clever way.

PoT - Incentives for Stratos Innovation

PoT (Proof of Traffic) is a unique incentive method created by Stratos, which intends to motivate all Stratos network participants through traffic standards. On the one hand, traffic and the number of users are the standards for measuring Internet companies today, and traffic also has huge commercial value. Traffic as a value measurement standard is in line with the law of value. On the other hand, the overall performance of a node is determined by computing power, storage, and network bandwidth. A simple Proof of Storage or PoW mechanism will make miners only focus on storage or workload. Over time, the overall performance of the network will will get worse.

The PoT mechanism pays attention to the need for files stored on Stratos to have fast enough response capabilities. Therefore, this also encourages miners to pay more attention to the overall performance of the network and increase the network bandwidth of each machine so that the Stratos network can still be expanded at a high speed. Ensure the health and efficiency of the network. Conversely, if the node always responds slowly to customers and the user service level drops, the tasks assigned to the node will decrease, and the income will also decrease.

Stratos Ecological Application

1. Accelerate network (decentralized CDN)

Stratos can cooperate with Arweave and IPFS for network acceleration, and can also provide CDN services for centralized Internet platforms. Stratos decentralized storage not only allows users to upload and download data efficiently, but also allows users to play audio or video directly across the network without prior download. According to the current operation of the Stratos test network, online video playback can be realized on Stratos.

2. Evolved NFT

The NFT platform is developed by combining blockchain and decentralized storage. It not only provides users with a digital asset copyright trading platform, but also provides digital asset usage rights trading.

For example, normally a musician would retain the copyright and sell the music digitally to fans, but this is not possible in the current NFT market, but it can be easily achieved in the Stratos NFT market, the musician only needs to sell the music rights once, or simply by selling usage rights to make his music heard by more fans. When Stratos launches the NFT platform for usage rights trading, it will transform the NFT platform from a luxury auction platform to a digital trading market for all. Products such as music, videos, games, artwork, legal contract templates, etc. will be easily traded on the platform after digitization.

3. Enhanced oracle (Oracle)

Stratos' decentralized database can help build entirely new oracle platforms. The Stratos decentralized database can support a richer data structure, and can better collect data of different dimensions from different sources for cross calculation to ensure the credibility and richness of the final data output.

4. Decentralized social media

The key to the decentralization of social media is the equal rights between accounts and the ownership of each account data. Stratos' decentralized database can fully guarantee equal rights and data ownership, and no longer allow the centralized voice of social media to cover and drown out the voice of any personal account.

5. Edge Computing and IoT Computing

Stratos not only encourages small data centers to access the network as a whole, but also welcomes individual nodes to access the network to provide decentralized computing services. Small data centers can form the backbone of the Stratos network, the information highway, while individual nodes are the branches of the highway. Although the computing power of personal nodes is limited, personal nodes can take advantage of the low latency of the network to provide users around the nodes with edge computing services required by smart appliances, smart home networking, and industrial IoT devices.

Ceramic - a decentralized open source platform to create, host and share data

Many applications need to use dynamic database to store files, such as:
User's identity information: profile, social graph, reputation score, linked social accounts, etc. User-generated content: posts, blog posts, interactive messages, social media, etc.
Dapp's various dynamic application data, user tables, etc.

Ceramic can meet such storage needs. Ceramic was formerly known as 3Box, a user identity data protocol based on Ethereum, which can provide calculation, state transition and consensus synchronization for all types of data structures stored on the decentralized network. Ceramic's data stream (steam) process enables developers to build secure, trustless, censorship-resistant Dapps based on dynamic information without a trusted database server.

1. Dynamic storage

Many current Web3 protocols have achieved success in decentralized file storage. IPFS (including IPLD and Libp2p), as a universal file system for the decentralized web, provides an extremely flexible content naming and routing system. A persistent network (like Filecoin, Arweave, and Sia) acts as a storage disk, ensuring that the content represented in an IPFS file is persisted and remains available. Although these Web3 protocol stacks perform well in storing static files, they all lack the computing and state management capabilities to implement more advanced database-like functions (such as mutability, version control, access control and programmable logic).

IPFS performs well in storing static files, but it lacks computing and state management capabilities, and is not good at storing files that need to be updated and upgraded at any time. More advanced database-like functionality cannot be achieved. Cermaic can solve these problems to a certain extent.

2. Composability

Ceramic enables data composability across applications primarily through a novel abstraction called the data model, which unifies how similar applications store and retrieve the state of each user on the network. For example, you can imagine that every decentralized Twitter implementation would run on some shared data model: one for each user's tweets, one for their social graph, one for their DMs, etc. wait. By adopting the same underlying data model, applications can natively interoperate on the same data.

In a way, you can compare Ceramic's use of data model standards to its use of token standards for distributed ledgers of assets. For example, on Ethereum, the introduction of ERC-20 homogeneous tokens and ERC-721 non-homogeneous token standards has given birth to the entire ecosystem of tokens and financial applications, making them interoperable naturally. And Ceramic brings this concept to data.

This composability also leads to a better developer experience. Building an app on Ceramic looks like browsing a marketplace of data models, plugging them into your app, and automatically gaining access to all the data stored in those models on the web. With Ceramic, developers won't need to worry about bootstrapping their applications with their own siled users and data. The speed of composite innovation across developers will be greatly accelerated.

3. Efficient version control

In Ceramic, each piece of stored information is represented as a superimposed log (a log file that records the running status of the program in the computer), called a Stream. Stream is conceptually similar to Git. Git is an open source distributed version control system, which can effectively and quickly handle version management of large and small projects. It is currently the most popular version control software, which can be used to store code, track revision history, merge code changes, and restore to Older code versions and so on.

Git processes data like a "snapshot", which is a bit similar to sharing documents with google docs and viewing historical versions. Whenever you submit an update or save the data state, it will create a snapshot of all the files at that time and save the index of this snapshot. If the file has not been modified, Git no longer re-stores the file, but only keeps a link pointing to the previously stored file, which greatly improves efficiency.

Git processes data like a "snapshot", which is a bit similar to sharing documents with google docs and viewing historical versions. Whenever you submit an update or save the data state, it will create a snapshot of all the files at that time and save the index of this snapshot. If the file has not been modified, Git no longer re-stores the file, but only keeps a link pointing to the previously stored file, which greatly improves efficiency.

Summarize

Web3 is the natural evolution of the contemporary Internet into the next era. The next-generation Internet must meet the needs of machine intelligence and the interconnection of all things. Decentralized storage is bound to become an important infrastructure of Web3. This article lists decentralized storage projects represented by Storj, IPFS, Arweave, Stratos, and Ceramic, which provide decentralized storage solutions from different levels. Compared with centralized storage, decentralized storage is still in its infancy. To achieve the unity of security, efficiency, and decentralization, the decentralized storage track faces different challenges such as technical iteration, incentive mechanism, and ecological development. We also look forward to the continuous breakthrough of the decentralized storage project and release greater energy in the next round of bull market.