P2P (Peer-to-Peer) networks, also known as point-to-point networks, are internet systems that rely on user groups (Peers) to exchange information without a central server (Figure 2-1). They are a type of distributed network. Generally speaking, the status of each node in this network is equal. Unlike centralized networks with a central server, such as Client-Server (C/S) systems (Figure 2-1), each node in a peer-to-peer network acts as both a client and a server. Nodes communicate information through their connections, sharing resources they possess (such as disk storage space, network bandwidth, processor usage, etc.) to provide services and content. Therefore, when new nodes join the network, the overall capacity of the system increases accordingly.
Compared to C/S network models, P2P networks are particularly suitable for file sharing: in a C/S structure, resources are stored on a central server, and under fixed bandwidth, the more users request downloads, the slower the average data transfer becomes for each user. In contrast, in a P2P network, many nodes store copies of the same file, allowing users to download it simultaneously from multiple nodes. Additionally, files that have already been downloaded can be uploaded to other nodes that are downloading, resulting in faster speeds as the network grows. P2P networks make full use of the bandwidth of other peer nodes in the network, rather than just relying on the bandwidth of the file source node.
The success of file sharing has made P2P networks very popular; however, since most shared files are popular music and movies, issues of copyright infringement have led to significant criticism of P2P networks. In a typical P2P network, data can be copied freely, and copies can be stored arbitrarily. However, assets cannot be copied freely or exist in multiple copies. The Bitcoin project created by Satoshi Nakamoto retains the "distributed" characteristic of P2P networks while addressing the issue of asset transfer within P2P networks: assets flow between different addresses rather than being simply "copied"; miners verify the destination of assets during the transaction process. The following will provide a detailed explanation of the P2P network in Bitcoin, which may offer insights into using blockchain technology to solve issues such as copyright protection.
P2P Network in Bitcoin#
In the Bitcoin network, each node can randomly connect to other nodes, and nodes can join or leave the network at any time. The total number of nodes in the entire network is uncertain. When nodes update information, they are not real-time consistent but only need to reach consensus within a certain time frame. In other words, the exit or crash of some nodes does not lead to the paralysis of the entire network; the joining and leaving of users do not significantly impact the overall network.
Information exchange between nodes is mainly reflected in transaction broadcasting. Transaction broadcasting is achieved through flooding. Specifically, when a node initiates a transaction, it informs all neighboring nodes of the transaction information. The neighboring nodes will verify whether the transaction can proceed based on the historical transaction information they have stored. If the verification passes, the transaction information will continue to be relayed to the next batch of neighboring nodes. The transaction information spreads like ripples among the nodes. When the transaction information received by a node overlaps with the information in that node's transaction pool, it indicates that this information has been propagated once, and the broadcast will then stop. Since each transaction has a unique hash value, it is very convenient to check for duplicate transaction information using the hash value.
It should be noted that in a P2P network, due to bandwidth and other reasons, the transmission of information is often delayed. Therefore, the contents of transaction pools in different nodes may vary slightly. If someone initiates a double-spending attack and both payments reach the same node, the honest node will only retain one transaction, and the other will not be broadcasted. Eventually, different nodes in the network may have discrepancies in recording different transactions for a short period, but this is not a problem. Over time, the consensus mechanism ensures that only one transaction will be recorded, making double-spending attacks impossible.
In summary, the Bitcoin project establishes a cash payment system within a P2P network, primarily relying on the following factors: the equal status of nodes; transaction information is propagated among nodes through flooding; nodes check whether transactions are valid; and the consensus mechanism determines the legitimacy of transaction information.
Limitations and Trade-offs of P2P Networks#
The advantages of P2P networks, such as fault tolerance, scalable transmission speed, and data security, come at the cost of low transaction processing capacity in blockchain projects. Currently, in the fierce competition among public chains, many projects are showcasing their transaction processing capabilities (for example, claiming to handle over ten thousand transactions per second), which also indicates that this is a problem that existing blockchain technology still needs to address. In fact, as more nodes are added to the network, the transmission delay of information between nodes gradually accumulates, and the time required for information to propagate throughout the entire network increases. Therefore, P2P network projects must balance between low transaction throughput and centralization. When a small number of "super nodes" are set up to verify transaction information, the efficiency of processing transaction information can be improved, but this also makes the network more centralized. In a network where all nodes have equal status, the verification of transactions by all nodes will result in a certain degree of redundant labor and resource waste.
The excitement surrounding blockchain technology is closely related to its decentralized characteristics, which are largely based on P2P networks. P2P networks are a very balanced concept, but they also require a certain amount of resources as a cost. A trade-off must be made between a balanced network and improved work efficiency. Establishing a highly efficient peer-to-peer network will require continued advancements in communication technology.
Blockchain technology proposes a distributed ledger architecture that eliminates third-party institutions from the system, allowing people to conduct transactions directly with each other. The solution offered by blockchain is to allow all users to have a ledger, with all users participating in the accounting process. However, this also raises a question: how can it be ensured that all users have the same ledger? How can the consistency of ledger information be guaranteed?
In blockchain, transaction information is broadcasted to the entire network, and each node can receive transaction information. Thus, the consistency issue of ledger information essentially becomes a "uniqueness" issue. As long as a rule is designed to ensure that only one unique transaction information can be retained through filtering, it can guarantee that each user records the same information.
The "block" and "chain" are the data structures that achieve this uniqueness. A block stores transaction information over a period of time and is essentially a packaging of transaction information; in Bitcoin, a block can store about 3,000 transaction records. Once this block is confirmed, all 3,000 transactions are confirmed together. If transaction information is not packaged, confirming each transaction would require frequent confirmation operations, which would be inefficient.
Each block contains address information pointing to the previous block, thus forming a "chain" that links the latest block to the genesis block. Different consensus mechanisms provide different solutions for generating new blocks; sometimes, it is possible to produce two (or more) blocks with different contents within the same time frame, which is known as a "fork." After different blocks are created, each will generate new blocks, extending each chain. However, generally speaking, the speed of chain extension varies. The vast majority of blockchain projects follow the rule of "selecting the longest chain as the main chain," which ensures that even if a fork occurs, there will always be a recognized "main chain" after a certain period.
Since the longest chain is unique, all users will record the same chain in their local databases, which guarantees the uniqueness of the ledger and thus resolves the consistency issue of the ledger.
Moreover, the chain structure brings another benefit. All blocks are connected through the "chain," forming a cohesive whole. If a hacker wants to tamper with the content of a historical transaction, they need to alter the block containing that transaction; hackers cannot create a new block and directly replace it but must replace the entire chain from that block to the latest block (in Bitcoin, this means starting to mine from the block that needs modification, continuously mining all subsequent blocks, and ultimately surpassing all other miners in progress to create a new longest chain), which is costly. This helps prevent tampering with transactions and other attacks.
However, this data structure still has issues: low data processing capacity. In blockchain, to avoid conflicting transaction information (no continuous forks allowed) and to ensure the consistency of the ledger (a unique chain must be selected), the blockchain adopts the longest single-chain structure. Since only one block can be added at a time, the propagation and confirmation of block information in a P2P network take time, and the limited capacity of blocks means that there is an upper limit to the transaction information that can be recorded over a period. It is evident that "low data processing capacity" is actually the cost of meeting consistency requirements. Currently, the Bitcoin blockchain can only process about 7 transactions per second on average, which is significantly lower than centralized electronic payment systems.
To solve the problem of low data processing capacity (small transaction throughput), an important idea is to allow multiple transactions to be processed in parallel. Sidechain technology connects different blockchains by locking and unlocking funds on the main chain, expanding the space for transaction processing. The sharding idea divides users into different shards, allowing transactions within each shard to be independently verified and processed in parallel, while transactions across shards require additional processing. In sidechain and sharding technologies, the main chain still exists, and both limit the flexibility of transactions (such as freezing funds, restricting transaction parties, etc.) to ensure consistency of the ledger while maintaining security.
On the other hand, DAG (Directed Acyclic Graph) is an exploration of another form of data structure. In general blockchain projects, all nodes store the same information; however, projects using DAG technology allow each node to store different information. In DAG, blocks can be generated at any time, and a block connects to multiple parent blocks. This way, everyone can record transactions at any time, significantly increasing the speed of recording transaction information.
However, since multiple blocks can be generated simultaneously and all are valid, DAG cannot guarantee consistency using a "unique longest chain." In this regard, some projects ensure the consistency of the ledger on DAG by using "temporal" methods. Specifically, in DAG, a new block randomly selects two recent blocks to connect, while verifying the transaction information of all blocks connected to it. Blocks that have undergone multiple verifications have a low probability of conflicting transaction content and can be considered confirmed transaction information. In this scheme, the verification of consistency relies on the extension and growth of the block network.
Other projects ensure ledger consistency through "full connectivity," meaning that each new block connects to all previous blocks and verifies all prior transaction information. Some projects use "ordering" to ensure consistency, confirming new blocks through recursive voting among blocks.
DAG improves throughput; however, "consistency" remains a complex issue that needs to be resolved. Currently, solving this problem requires some trade-offs: it may delay the exact verification time of transaction information; or it may require extensive network communication between nodes, making the actual transaction speed still uncertain.
Ultimately, the "consistency" issue of distributed ledgers is a balancing problem. It can be said that "consistency" is a system goal, and achieving this goal requires corresponding resources. Therefore, sacrificing transaction speed, limiting transaction flexibility, delaying confirmation times, or increasing transmission requirements across the entire network are all adaptive choices made under different system conditions. It is believed that the various technologies mentioned above will be further explored and validated in different application scenarios.
In a P2P network, all nodes have equal status, and information is transmitted between nodes; in a distributed ledger, it is necessary to ensure that the content stored among nodes is the same. The "consensus mechanism" is a solution that ensures the content among nodes is the same. In a P2P network, there may be situations where node performance declines, network congestion occurs, etc., leading to the propagation of incorrect information within the system. Therefore, when designing a consensus mechanism, it is essential to assume that unreliable nodes exist in the system. From an algorithmic perspective, the design of these mechanisms is essentially based on economic interest games, ensuring that the rewards for honest accounting exceed the rewards for malicious destruction, thereby guaranteeing cooperation from the vast majority.
The Boundaries of Technology#
When designing a new product, one must understand the current boundaries of technology: which technologies can be fully utilized and which still need to wait for a while. For technologies that need to wait, people have to consider them later. Of course, science and technology are somewhat different; scientific research can provide theoretical limits, while engineering design focuses more on how to do the best within the approximate boundaries that are likely to occur. This is similar to an optimization problem, where one needs to know the given constraints to solve it correctly.
Regarding consensus mechanisms, previous research has provided two important boundaries:
● Fischer-Lynch-Paterson Theorem: It proves that in a multi-process asynchronous system, as long as there is one unreliable process, there cannot be a protocol that guarantees all processes reach consensus within a finite time.
● CAP Principle: A distributed computing system cannot simultaneously ensure consistency, availability, and partition tolerance. Designs often need to weaken the guarantee of one of these properties.
Among them, consistency refers to the agreement among service nodes in the system regarding processing results; availability refers to the ability of any non-failed node to respond to requests within a finite time; partition tolerance refers to the possibility of network partitioning, which prevents communication between nodes.
Achieving complete consistency in a distributed scenario is impossible, but solving many engineering problems lies in how to make reasonable trade-offs. One can sacrifice some costs to achieve consistency in a distributed scenario. Currently, the differences in various consensus mechanisms designed based on blockchain mainly stem from the following two aspects:
First, different algorithmic assumptions. For example, algorithms like Paxos and Raft assume that nodes will not intentionally send erroneous messages, which is a relatively strong condition. The PoW consensus mechanism used by Bitcoin does not pre-know how many accounting nodes are in the system, while protocols like PBFT commonly used in consortium chains assume that nodes need permission.
Second, sacrificing some costs to achieve a certain degree of consistency. For example, according to the CAP principle, weakening availability by refusing service during system failures, algorithms like Paxos and Raft weaken availability to ensure result consistency. Similarly, Bitcoin sacrifices some fault tolerance (potential forks may occur) but guarantees consistency in the entire blockchain system after a certain time limit.
Algorithms are certainly not omnipotent; their boundaries determine that some other incentives and constraints must be introduced to ensure the normal operation of the entire system.
In blockchain projects based on PoS (Proof of Stake), creating new blocks does not require consuming computational power, and there are no penalties for malicious behavior by nodes; for a node, the choice that maximizes benefits is to mine on multiple chains simultaneously, which can lead to severe forking phenomena. Generally, additional rules need to be introduced to address these situations, such as incorporating penalty protocols.
Common Consensus Mechanisms in Public Chains#
The design of consensus mechanisms in public chains mainly revolves around decentralization and enhanced incentives. Many new blockchain systems currently support pluggable consensus mechanism modules, allowing different consensus mechanisms to be switched based on application scenarios and needs.
Maintaining the "uniqueness" of the main chain is crucial for public chains, as this is key to solving the "double spending" problem: to avoid double spending, one must be aware of all historical transaction information to ensure that the transaction does not conflict with previous history. How to ensure that transactions can proceed smoothly in an environment of asymmetric and uncertain information is the "Byzantine Generals Problem."
The PoW (Proof of Work) mechanism of Bitcoin solves the Byzantine Generals Problem through the following methods:
● Maintaining periodic cycles to ensure nodes are synchronized: Adjusting difficulty to ensure that the network consistently takes 10 minutes to find a solution to a mathematical problem and generate a new block. During this 10 minutes, participants in the network send transaction information and complete transactions, and only then is the block information broadcasted, thus eliminating the state of nodes sending commands without limit or regularity.
● Ensuring that new blocks are generated by a single node through a computational competition: Bitcoin ensures that only one (or a few, in the case of forks) node can mine a new block within a certain time period through timestamps and electronic signatures.
● Using a common ledger through the blockchain: Each node in the Bitcoin network is synchronized in information during each cycle.
In fact, regardless of the method used, as long as time is unified, steps are synchronized, single-point broadcasting is achieved, and a single chain is maintained, the Byzantine Generals Problem of distributed systems in cryptocurrency can be solved.
As another consensus mechanism, PoS allows miners to have a probability of generating new blocks equal to the proportion of cryptocurrency they hold. This can lead to wealthier accounts having greater power, potentially controlling accounting rights, and may also result in increasing centralization of rights. However, PoS does significantly reduce the energy costs of mining. In the long run, more cryptocurrencies may develop in the direction of PoS.
In addition to the two commonly used mainstream consensus mechanisms mentioned above, the innovation in current public chain consensus mechanisms lies in the hybridization between the two, thereby improving data processing efficiency while retaining decentralized characteristics.
For example, the PoW/PoS hybrid consensus represented by Decred:
- The mining process is similar to Bitcoin, requiring a certain amount of proof of work, but there are differences in the consensus phase. Unlike Bitcoin, which requires all network nodes to verify blocks, the hybrid mechanism introduces PoS voting to determine whether the newly mined block is valid, greatly increasing verification speed. Additionally, there is also the PoW/PoS hybrid consensus represented by Hcash, which combines a dual-layer chain structure. It divides PoW difficulty into two levels, published on two chains, allowing both PoW miners and PoS miners to participate in system consensus and play a role.
Common Consensus Mechanisms in Consortium Chains#
Consortium chains place more emphasis on privacy, security, and regulation, thus incorporating more control elements and adopting consensus mechanisms similar to traditional Byzantine family consensus mechanisms.
Compared to public chains, consortium chains weaken the emphasis on decentralization. At the same time, due to the node admission system, certain trust has already been granted to the nodes.
In the DPoS (Delegated Proof-of-Stake) mechanism, those with stock rights are elected and replaced, rather than generated based on the number of coins like PoS. It periodically selects a small group of nodes through different strategies to create, verify, sign, and mutually supervise new blocks, significantly reducing the time and computational costs required for block creation and confirmation. DPoS does not require much trust; the selected delegates cannot change transaction details, and if nodes attempt to act maliciously, provide unstable computational power, or experience computer crashes, the public community can quickly vote to expel them.
If PoW and PoS primarily solve consensus issues based on economic models, then PBFT (Practical Byzantine Fault Tolerance) solves consensus through algorithmic models. It does not have a token distribution mechanism and has very low energy consumption. The process can be summarized as everyone voting to elect a leader, who records the transactions, and others vote to approve. In the PBFT algorithm, it can be proven that as long as the number of faulty Byzantine nodes is less than one-third of the total number of nodes in the system, the entire system can operate normally. Current improvement directions for the algorithm include using P2P networks, dynamically adjusting the number of nodes, and reducing the number of messages used in the protocol.
Innovations in consortium chain consensus mechanism algorithms also include hybridizing DPoS and PBFT, applying the authorization mechanism of DPoS to PBFT for dynamic authorization. Research has shown that such algorithms can achieve a TPS of 10,000-12,000 with a best block time interval of 20 seconds, with latency controlled between 100-200ms. It is precisely because consortium chains retain some degree of "centralization" that they achieve faster transaction speeds and significantly lower transaction costs.
Economic Costs of Consensus#
Consensus comes at a cost. Public chains like PoW incur significant computational costs, hardware wear, and natural resource consumption to perform calculations and solve a mathematically meaningless problem to compete for accounting rights. In consortium chains, achieving consensus is akin to democratic voting, requiring rounds of negotiation and opinion exchange to reach agreement. How to reduce the cost of democracy, achieving consensus with the fewest negotiation rounds and the least communication cost, is the goal pursued by algorithms and is also a crucial factor determining whether the blockchain machine runs fast enough.
Ultimately, we should focus on the balance between the cost and effect of consensus mechanisms. After all, blockchain technology must eventually be implemented. For enterprises, they should carefully consider their input-output ratio to determine whether to use blockchain technology or if there are lower-cost alternative solutions.
For example, using distributed databases to solve information asymmetry between enterprises, setting viewing permissions and encryption levels for data to achieve immutability, combined with a series of management measures, plus the fact that in most scenarios, leading enterprises may have little motivation to manipulate data and have sufficient motivation to maintain the database, in such cases, even the most complex consensus mechanism design may not be as effective as a good business model.
Characteristics of Hash Functions#
A hash function refers to a class of mathematical operations that accept input values of any size and produce a fixed-length output value after computation. This output value can serve as a digital fingerprint of the input value. Just as twins have unique fingerprints, the design of hash functions ensures they have the same characteristic: even very small differences in input values can result in significant differences in the output of the hash function. Additionally, hash functions have no heuristic algorithms; the relationship between input and output appears completely random. For example, given a specific output result, determining what the corresponding input value should be, or finding an input value that results in an output less than a certain value, has no tricks or methods to follow and can only be achieved through continuous trial and error. The more attempts made, the more likely an answer will be found.
People can utilize these characteristics of hash functions to achieve many functions. For example, data protection: sending both the content of the data and the hash value of the data, allowing the receiver to perform a hash operation on the received data and compare it to determine whether the data has been tampered with. Additionally, when users log into a website, they can store the hash value of the user's password in the database and compare it with the hash value of the password entered by the user to verify identity. The benefit is that if the database is leaked, hackers cannot reverse-engineer the user's password from these hash values, making it relatively secure.
It is worth noting that the input set of hash functions is infinite, while the output length is fixed, meaning the set of all possible outputs is finite. According to the pigeonhole principle: if there are n+1 elements placed into n sets, at least one set must contain at least two elements. Therefore, it is theoretically certain that two different input values will have the same hash value, but the probability of this happening is very small, and hash functions are continually being improved. The SHA1 function was once found to have effective attack methods by cryptanalysts, and subsequently, systems including Bitcoin adopted the more advanced SHA2 series algorithms. The long-term good operation of Bitcoin indicates that the SHA256 algorithm has stood the test of time. Furthermore, using hash functions multiple times is also a more secure choice.
Hash functions play a crucial role in Bitcoin in several ways.
- Purpose One: Compression and Verification of Transaction Information
Since the blockchain has to handle a large amount of transaction information, directly storing all data in each block in a sequential manner would be very inefficient and time-consuming. However, using hash functions allows for compression and verification of information. In Merkle trees (a binary tree structure that can be understood as a topology for storing data), combined with hash function technology, it is possible to quickly verify whether a transaction belongs to a certain block. For all transactions packed into a block, they are first divided into parts such as transaction information 1, transaction information 2, etc., and their corresponding hash values are calculated. Then, pairs are combined for hashing, ultimately obtaining the root hash value of this Merkle tree. If any transaction information recorded changes, the final calculated Merkle root hash value will also differ.
So why use such an algorithm instead of simply stringing all transaction information into a large block and calculating its hash value? The reason is that this binary tree structure allows for only a small amount of data to be verified, and if the transaction data is incorrect, it can quickly pinpoint the source of the error.
- Purpose Two: Proof of Work
What makes blockchain immutable? First, consider a simple hash chain: each time a block is packaged, it includes the hash value of the previous block and the relevant information of this block. If the information of a certain block is tampered with, all subsequent blocks' hash values will change, and others will notice this change. However, the problem with this design is that anyone can modify the information on a certain block, recalculate all the remaining chain's information, and claim that this is the correct chain.
The brilliance of Bitcoin's design lies in the fact that achieving this process requires a costly investment. It adopts a proof-of-work consensus mechanism, where everyone competes to prove they have completed a certain amount of work, with the first to complete it gaining accounting rights. The work required is to find a random number such that when added to a given string, the resulting hash value is less than a certain value. In Bitcoin, this given string includes version number, hash value of the previous block, transaction information stored as the Merkle root hash value, timestamp, and difficulty value. When miners find a random number that meets the requirements, they both "legally" declare their accounting rights and encode transaction information through the hash function, storing it in an immutable manner. If someone attempts to change transaction information, they must be exceptionally lucky to quickly and successfully find the correct random number for each subsequent block in the chain, making the altered chain the current longest chain. While this scenario is theoretically possible, the probability is quite low under limited computational power.
- Purpose Three: Bitcoin Wallet Address
In Bitcoin transactions, visible information includes the transaction number in the upper left corner, and the string of letters and numbers connected by arrows represents the Bitcoin address, indicating that Bitcoin has been transferred between two addresses. The generation of this address is derived from the wallet's public key through a hash function. The public key is formed by asymmetrically encrypting a randomly generated private key. During transactions, both the public key and Bitcoin address need to be publicly released to allow the blockchain system to verify the validity of payment transactions.
Here, the role of the hash function is quite clever: quantum computers can easily deduce the private key from the public key, but they find it difficult to identify two different input values that have the same hash value when facing hash algorithms. Satoshi Nakamoto's design cleverly utilizes the characteristics of hash functions, potentially allowing Bitcoin to resist threats from quantum computers: for example, each Bitcoin address is used only once, and each payment transfer is made to another person's address and one's own change address.
Satoshi Nakamoto effectively leveraged the characteristics of hash functions through clever design, ultimately forming a well-functioning system that involves multiple interdisciplinary fields and reminds people to abstract the essence of things and pay attention to integration with other fields during technological innovation. With technological advancements, new hash functions are continually being designed and tested.
Zero-Knowledge Proof#
Zero-knowledge proof is a probabilistic verification method that includes "factual statements" and "statements about personal knowledge." The verifier poses questions to the prover based on certain randomness, and if the prover can provide correct answers, it indicates that the prover likely possesses the claimed "knowledge." Zerocoin applies zero-knowledge verification in the process of minting and redeeming zerocoins to hide the sender and receiver information corresponding to a transaction. Zerocash adopts the more novel zkSNARKs technology, converting the transaction content that needs verification into proof that two polynomial products are equal, combining techniques like homomorphic encryption to protect hidden transaction amounts while verifying transactions.
The downside is that if the network is attacked and excessive zerocoins are generated, people may not be able to detect this situation or take measures; both Zerocoin and Zerocash require a pre-established "trusted setup," failing to achieve true "trustlessness." New technologies like Intel SGX and zkSTARKs may potentially address these issues, but they still require practical validation.
Zero-knowledge proof is a cryptographic scheme initially proposed in the 1980s by MIT researchers in a paper: "A zero-knowledge protocol is a method by which one party (the prover) can prove to another party (the verifier) that something is true without revealing any additional information beyond the fact that this specific statement is true." For example, in logging into a website, the server stores the hash value of the customer's password. To verify that the customer actually knows the password, most websites currently use a method where the server hashes the password entered by the customer and compares it with the stored result. However, the drawback of this method is that the server can know the customer's original password during the calculation, and if the server is attacked, the user's password will be leaked. If zero-knowledge proof could be implemented, it would allow verification of customer login without knowing the customer's password, ensuring that even if the server is attacked, the customer's plaintext password is not stored, keeping the user's account secure.
The basic zero-knowledge proof protocol is interactive, requiring the verifier to continuously ask the prover a series of questions about the "knowledge" they possess. If the prover can provide correct answers, it is highly probable that they indeed know the claimed "knowledge."
For example, if someone claims to know the answer to a Sudoku puzzle, a zero-knowledge proof method could involve the verifier randomly specifying whether to check by row, column, or grid. Each time, the verifier does not need to see the specific positions of the numbers, only whether the numbers 1-9 are included. As long as the verification occurs enough times, it can be reasonably believed that the prover knows the solution to the Sudoku puzzle. However, this simple method does not ensure that neither the prover nor the verifier is cheating; in the Sudoku case, both could have colluded beforehand, allowing the prover to pass verification without knowing the answer.
If they want to convince a third party, the verifier must also prove that their detection methods are random and that they have not colluded with the prover.
Due to the difficulty for third-party observers to verify the results of interactive zero-knowledge proofs, when proving something to multiple people, additional effort and costs are required. Non-interactive zero-knowledge proofs, as the name suggests, do not require an interactive process, avoiding the possibility of collusion, but may require additional machines and programs to determine the sequence of experiments: for example, in the Sudoku case, a program could determine which checks are made by row or column, but this sequence of experiments must remain confidential; otherwise, if the verifier knows the sequence in advance, they could prepare accordingly and pass verification without knowing the actual "knowledge."
The content of zero-knowledge proofs can be summarized into two categories: "factual" statements:
-
For example, proving "a specific graph can be three-colored," or "a number N is composite";
-
Statements about personal knowledge: for example, "I know the coloring scheme for this specific graph" or "I know the factorization of N."
However, not all problems have zero-knowledge proof cryptographic schemes. Goldreich, Micali, and Wigderson provided the theoretical range of problems that can have zero-knowledge proof solutions. They found that for decision problems of polynomial complexity (where the answer is simply yes/no), there are known zero-knowledge proof schemes. One only needs to find the statement to be proved within such NP problems and convert it into an instance of the three-coloring problem to utilize existing protocols to achieve zero-knowledge proof. Since the three-coloring problem is an NPC problem, any other NP problem can be transformed into an instance of this problem.
In blockchain transactions, such as those in Bitcoin and Ethereum networks, in addition to using addresses to replace the real identities of the transaction parties, the sending and receiving addresses and amounts are known. Others may potentially correlate Bitcoin addresses with real identities through various information on the network and interactions occurring in the real world, thus exposing privacy risks. Zerocoin designs a completely new approach that prevents the real identity of users from being obtained through transaction history analysis. In Zerocoin, a certain value of the currency to be traded is consumed to generate a zerocoin with a unique serial number. Zero-knowledge proof can verify that you indeed spent this money without revealing which specific currency was used. To transfer this money to others, it logically requires that this zerocoin cannot be spent again by others. The method for zerocoins is to maintain a list of all spent zerocoins' serial numbers. When miners verify this spending transaction, they use zero-knowledge proof methods to check whether the zerocoin's serial number is on the list of spent coins without needing to know which specific zerocoin was spent. Since the spending transaction does not contain the input address and signature information, the miners do not know the source of this zerocoin during the entire transaction process, making it difficult to analyze transaction history to obtain user identities.
In Zerocoin, the transaction amount can be known, while in Zerocash, which uses zkSNARKs technology, even the transaction amount can be hidden, with the only publicly recorded content on the ledger being the existence of the transaction. It can be proven that zkSNARKs exist for all problems in NP. It introduces multiple innovative technologies that allow them to be used in blockchain. Most importantly, zkSNARKs reduce the size of proofs and the computational effort required to verify them. Its process can be summarized as follows:
(1) Decomposing the program to be verified into logical verification steps, breaking these logical steps down into arithmetic circuits composed of addition, subtraction, multiplication, and division.
(2) Through a series of transformations, converting the program to be verified into a verification of whether the product of polynomials is equal, such as proving t(x)h(x)=w(x)v(x).
(3) To make the proof more concise, the verifier randomly selects several checkpoints s to check whether the equations hold at these points.
(4) Using homomorphic encoding/encryption, the verifier can compute the equations without knowing the actual input values but still perform verification.
(5) If a non-zero secret value k can be multiplied on both sides of the equation, then when verifying t(s)h(s)k equals w(s)v(s)k, it becomes impossible to know the specific values of t(s), h(s), w(s), and v(s), thus protecting information security.
Unlike Zerocoin's cryptographic primitive RSA accumulator, zkSNARKs technology is newer and has not been widely validated, posing risks. Additionally, due to stronger anonymity, vulnerabilities in Zerocash are harder to detect. Compared to Zerocoin, Zerocash's transaction amount information is also unknown, making it impossible to detect if an attacker issues an unlimited number of zerocoins.
Furthermore, both Zerocoin and Zerocash require pre-built parameters, and users must trust that these parameters have not been leaked when using these networks. However, once these parameters are leaked, the entire network faces catastrophic consequences. The complex trust setup makes Zerocash controversial, even though they have designed a "ceremony" (such as recording the process of destroying the computer storing the keys) to prove their security.
Possible solutions include utilizing modern "trusted execution environments" like Intel SGX and ARM TrustZone. In the case of Intel's SGX technology, even if applications, operating systems, BIOS, or VMM are compromised, the private keys remain secure. Additionally, the newly proposed zkSTARKs technology does not require a trust setup.
According to the zkSTARKs white paper, zkSTARKs is the first to achieve blockchain verification without relying on any trust setup while providing a system that accelerates computation exponentially as the amount of computational data increases. It does not rely on public key cryptosystems, and its simpler assumptions make it theoretically more secure, as its only cryptographic assumption is that hash functions (such as SHA2) are unpredictable (this assumption is also the basis for the stability of Bitcoin mining), thus making it resistant to quantum threats. As a novel technology, like zkSTARKs, it also requires the test of time.
If a problem can be solved by an algorithm in polynomial time, it is called a P-class problem. NP problems are those for which a solution can be verified in polynomial time. If a problem is an NP problem and all NP problems can be reduced to it, it is called an NPC problem.
Hash Time Lock Protocol#
Hash Time Lock Agreements (HTLAs) is a technology that enables token transactions and exchanges between different blockchain projects. In traditional exchanges, traders often need to pledge tokens to the exchange in advance, which brings certain trading risks and requires high transaction fees. In the hash time lock protocol, only the sender, connector, and receiver are needed to complete the token transaction, without any exchange platform, and if the transaction fails, the tokens have not actually been transferred, incurring no additional transaction fees. Compared to exchanges, the hash time lock protocol provides a "flea market" where no third-party custodians are needed, and the role of exchanges is decentralized to individuals within the community, allowing safe token transactions between people.
The idea of hash time lock protocol technology originated from a discussion on the BitcoinTalk forum in 2013, and its practical implementation is related to Bitcoin's Lightning Network. In the Lightning Network, to establish a small payment channel between two users, users must lock a portion of their funds in advance, and transactions involving that portion of funds occur off-chain. After a period, the final distribution of funds is determined, and this distribution plan is uploaded to the main chain (Figure 7-1). This allows for a large number of small transactions to occur off-chain, increasing the transaction throughput of the Bitcoin network.
The hash lock contract (HTLC) technology used to lock user funds in the Lightning Network inspired later developers. Token transactions require conversion through intermediaries, and the key lies in obtaining trust from all parties. The process of locking tokens is precisely a pledge process that can generate trust.
Taking a hypothetical transaction as an example. Suppose 1 Bitcoin is equivalent to 10 Ether, and the sender (Sender) has 1.1 Bitcoins and wishes to purchase services worth 10 Ethers from the receiver (Receiver). The sender can contact a connector (Connector) who has both a Bitcoin address and an Ethereum address and negotiate a token conversion fee of 0.1 Bitcoins. The transaction process is illustrated in the figure.
In this process, the step of transferring 1.1 Bitcoins carries a higher risk: the connector may exit the transaction immediately after receiving the 1.1 Bitcoins, causing losses to the sender. A reasonable solution is to delay the delivery of the Bitcoins. After the Ether is delivered, the Bitcoin delivery can occur, thus transferring the transaction risk to the connector (Plan Two).
To simultaneously protect the interests of the connector, the issue to be resolved is to ensure that when the receiver receives 10 Ethers, the sender's 1.1 Bitcoins are also sent to the connector, and both events need to happen simultaneously. This is essentially a manifestation of the transaction's "atomicity": either the funds are fully transferred, or they are not transferred at all, with no intermediate state. This issue is resolved by the Pre-Shared Key (PSK) technology.
If we consider 1.1 Bitcoins as one transaction package and 10 Ethers as another transaction package, in PSK technology, both transaction packages are initiated by the same key, thus achieving "simultaneous occurrence."
The sender pre-generates a key through encryption algorithms, sends the key to the receiver, and sends relevant information to the connector. At the same time, the sender locks their 1.1 Bitcoins in transaction package 1, which requires the key to transfer the funds.
The connector uses the information provided by the sender to create a transaction package 2 containing 10 Ethers and sends it to the receiver. When the receiver uses the key to unlock transaction package 2, they receive 10 Ethers, and the key is also sent to the connector, who can use that key to obtain the 1.1 Bitcoins in transaction package 1. This achieves the exchange of tokens.
To prevent the funds from being locked for too long, both transaction packages 1 and 2 need to agree on a time limit; after the time limit, the funds will be unlocked and returned to the original address. This is the time lock (Timelock) function. The previously mentioned pre-shared key is also hashed, which is why this technology solution is called the hash time lock protocol (Hashed-Timelock Agreements).
Limitations#
This technology still has certain limitations:
(1) The connector must bear some risk. In PSK technology, the connector must inject the key into transaction package 1 to obtain the Bitcoins, meaning that the delivery of the Bitcoins and Ethers does not occur simultaneously. Since both currencies have agreed-upon time limits, if the time limit for transaction package 2 is greater than that for transaction package 1, the receiver may obtain 10 Ethers but the connector may not be able to reclaim the 1.1 Bitcoins, resulting in a loss. This risk can be avoided by setting the time limit for transaction package 1 to be greater than that for transaction package 2.
(2) For blockchain projects that do not support hash time lock technology, the above process can only be conducted through another ledger platform. The additional ledger platform can store transaction records of token transfers. However, since the ledger platform itself does not facilitate the transfer of tokens, it essentially records information about credit and debt, requiring sufficient trust between the two parties for the transaction to proceed. By using the ledger platform, as long as both parties have a foundation of trust, non-blockchain assets can also be exchanged through this accounting method.
In essence, transactions and flows between different assets only require a foundation of trust (to establish a connection) and ensure the atomicity of transactions (asset delivery) to proceed. In the hash time lock protocol, the locking of tokens establishes a pledge of assets, providing a foundation of trust for the transaction. The transfer of keys ensures the atomicity of the transaction. The introduction of time locks prevents disputes or accidents caused by prolonged transaction times. Beyond blockchain projects, this model can be applied to the flow of different asset categories.
The hash time lock protocol technology has been basically implemented by the Ripple Interledger project, and in blockchain projects capable of running smart contracts, token exchange technology may gradually become common. This technology provides a possibility for the ecosystem of blockchain projects: mainstream blockchain projects like BTC serve as the main settlement system, while other application projects specifically address different user needs. When users enjoy services and settle payments, they use token exchange technology for payment. This way, mainstream projects convey value; application projects cater to segmented needs while sharing service pressure with mainstream projects; users get what they need. Ultimately, a globally trusted settlement network will be built, running all decentralized applications on this foundation, truly realizing a rich blockchain application ecosystem.
References:
https://bitcointalk.org/index.php?topic=193281.msg2224949#msg2224
Sharding Technology#
The traditional concept of sharding technology involves dividing a database into multiple fragments and placing them on different servers. In modern cloud services, data is often hosted at different sites and partitioned. This practice aims to balance the load between multiple computers, thereby improving scalability; it also enhances availability by storing data across multiple sites.
Blockchain sharding technology is a scaling technology based on the concept of database sharding.
Whether in the blockchain field or the database field, the first step in sharding is to extract key feature values of the data and partition these key feature values into different fragments for processing based on certain rules. The selection of key feature values is crucial, as it relates to ensuring the uniqueness of the data and the effectiveness of sharding. A concise standard for selecting feature values is: "Use what you consider the basic data pattern as the standard." Therefore, in blockchain projects, it is common to see sharding based on user private keys/account addresses, as these values are unique and do not change over time, making the logic of sharding relatively clear.
Traditional Database Sharding Methods#
Traditional database sharding mainly has three methods:
(1) Hashing and Modulus: For example, if the entire network is divided into 3 shards, data is hashed and then taken modulus 3 to determine which specific fragment it belongs to. This strategy aims to reduce the occurrence of uneven shard loads, as the results of hash function calculations are completely random, breaking the correlation between certain key feature values and load amounts, thus making data more likely to be evenly distributed among shards. A counterexample would be if the key feature value is registration time order; newly registered data is more active, which could lead to them all being placed in one shard. However, this method's drawback is that if new shards are added, it becomes challenging to rebalance shards; its advantage is that it does not require additional state information maintenance.
(2) Consistent Hashing: The consistent hashing method without virtual nodes maps data according to feature values onto a circular hash ring, while also mapping nodes according to certain rules. The first node found in a clockwise direction is the one where the data is stored. The consistent hashing method with virtual nodes is similar but maps virtual nodes onto the hash ring, allowing a physical node to occupy multiple ranges on the hash ring. This method requires maintaining state information, meaning it is necessary to know which data is assigned to which node, but its advantage is that if the number of shards needs to increase, rebalancing shards becomes easier. However, maintaining shard state information requires considering consistency issues, making it more complex.
(3) Manual Partitioning: Dividing according to key feature values into different intervals, with each node corresponding to one or more intervals, similar to consistent hashing, also requires maintaining state information.
- Consistency Challenges in Sharding
In blockchain technology, there needs to be a mechanism to know which node implements which shard. In traditional database systems, shard information (i.e., metadata, indicating which data is divided into which fragment) is generally stored on a dedicated server. Sometimes, to alleviate the pressure on the metadata server, distributed systems cache metadata on other nodes. The approach in blockchain is generally similar, requiring consistency of cached metadata among nodes or introducing a similar main server to ensure performance, but these solutions face challenges of data consistency.
The consistency and availability of multiple replicas fall within the scope of CAP theory, with two main available solutions.
The first is master-slave synchronization, where a master server is selected, and only the master server provides external services. The master server logs the update information of metadata into a shared storage space, and slave servers read the logs from the shared storage space and apply them to achieve a consistent state with the master server. If the master server is detected to be faulty, a new master server will be re-elected.
In the case of network partitioning, it is possible for everyone to believe that the original master server has crashed and elect a new master server, but the original master server continues to provide services, leading to a "dual-master" phenomenon. To solve this problem, it is necessary to isolate the old master server so that it cannot provide services normally. To ensure strong consistency of metadata, when preparing for a switch, the new master server must confirm that metadata is fully synchronized before it can continue to provide external services. One way to achieve this is to immediately notify all cached servers when metadata changes and lock the data. If the system task requires multiple shards to update the state simultaneously, access will be denied until the updates are complete. Another method frequently implemented in highly scalable NoSQL databases to maintain high consistency between replicated data is to use read-write arbitration and version control. This method avoids locking data, but at the cost of added complexity during data reading and writing.
The second method is to achieve consistency among multiple replicas through distributed consistency protocols, such as Paxos and Raft protocols. These protocols can ensure that all backups provide external services while guaranteeing strong consistency.
Sharding Methods in Blockchain Technology#
In blockchain networks, depending on the objects, technology can be divided into state sharding, transaction sharding, and network sharding. Among them, network sharding employs various technical solutions.
State sharding in blockchain refers to each node only storing a portion of the blockchain state information, and the consistency issues to be resolved are similar to those mentioned above.
The implementation of transaction sharding is simpler. In account-based blockchain systems, each transaction will have a sender's address, and the system can allocate a shard based on the sender's address. This ensures that two double-spending transactions will be verified within the same shard, allowing the system to easily detect double-spending transactions without requiring any cross-shard communication. If the nodes are determined, there are almost no issues arising from the updates of the metadata discussed above. However, if transaction verification involves communication across shards, the cost is usually high, affecting network throughput and economic efficiency.
Network sharding in blockchain refers to dividing miners into several groups to verify transactions simultaneously, enhancing the system's ability to process transactions in parallel. Typically, this can be achieved by periodically generating random numbers to determine which nodes are selected to reach consensus, mapping them to pre-numbered shards. However, if a node crashes, reassigning nodes requires forming a consensus on shard consistency.
It is important to note that when using network sharding technology in blockchain, which divides miners into several sub-networks responsible for verifying transactions on that shard, it is necessary to ensure that the number of malicious nodes is sufficiently small. Therefore, attention must be paid to ensuring randomness in the rules for assigning miners.
The key to sharding technology lies in the fact that since the data within each shard is updated separately, when designing application logic, it is essential to ensure successful updates of information while balancing efficiency and leaving some robustness to address potential issues that may arise during the final consensus process. In applying sharding technology in blockchain, it is also necessary to consider defenses against various attacks, such as Sybil attacks, DDoS attacks, and double-spending attacks, ensuring that the total number of nodes within each shard is sufficiently large and that honest nodes constitute the majority. Sharding technology has extremely high security requirements, and the number of nodes in a blockchain system may exceed that in traditional databases, facing bandwidth limitations. Therefore, it is crucial to fully consider the performance and security issues caused by inconsistencies due to latency, which is why there are very few relevant projects that have been implemented. Long-term testing and validation in large-scale networks, combined with rigorous theoretical solutions, are required to gain credibility.
Space Proof Consensus Algorithm#
Since the first Bitcoin was mined in 2009, the blockchain industry has gradually expanded into a massive global market. Besides BTC, various blockchain projects like LTC, ETH, and EOS have emerged. Currently, there are over 110,000 ERC20 token projects on Ethereum alone, and the number of companies publishing project white papers is countless.
Bitcoin realized a peer-to-peer electronic payment system, and the birth of this distributed system relies on the PoW (Proof of Work) consensus algorithm it adopts. Currently, the vast majority of blockchain projects with main chains still use PoW or modified PoW consensus algorithms, with only a few projects adopting PoS (Proof of Stake) or DPoS (Delegated Proof of Stake) algorithms. PoW provides a clear and effective consensus generation mechanism for distributed ledgers; however, it also generates some problems: a significant amount of energy is wasted in the process of calculating hash functions. Due to the emergence of ASIC and other chips, Bitcoin also faces increasing centralization challenges. The current state of Bitcoin is far from Satoshi Nakamoto's original design.
Both PoS and DPoS mechanisms also have centralization issues, and the voting process is often cumbersome, making neither an optimal solution. It is worth mentioning that some blockchain projects have emerged that adopt schemes like "transaction mining," "staking mining," "insurance mining," and "mining mining." However, fundamentally, these projects only issue ERC20 tokens on Ethereum. Because they do not have a main chain, these projects do not require a consensus mechanism. The so-called mining schemes essentially belong to airdrop schemes, serving as incentive measures without a necessary connection to the core technology of blockchain.
To truly solve the problems of energy waste and centralization arising from PoW, diverse mining schemes must address a series of technical issues:
(1) What can be used as proof of the user's cost without consuming work?
(2) How is this proof verified?
(3) How is the winner of the mining competition determined?
(4) How to avoid main chain forks, etc.?
Principles of Space Proof#
In the process of technological advancement, the PoSpace scheme has made significant explorations. PoSpace, or Proof of Space, aims to replace the PoW mechanism in Bitcoin and become a new consensus mechanism solution. This scheme has already been implemented in some blockchain projects. It uses the hard drive space paid by users as proof of cost, occupying hard drive space by downloading files; the larger the occupied space, the greater the user's contribution.
PoSpace can bring the following benefits: significantly reducing resource waste; users only need to pay for hard drive space once, and subsequent mining does not require additional contributions, etc. According to some teams' calculations, user behavior in PoSpace can be viewed as an expansive game model, with more and more users joining over time.
To address the issue of hard drive space fraud, PoSpace divides nodes into two roles: provers and verifiers. Provers are ordinary nodes that need to store large amounts of information data (e.g., 100GB), while verifiers store a small portion of the prover's storage information along with the database for verification.
When a user/prover first joins the network, they need to store a portion of data with a specific sequence based on the size of the chosen storage space (the stored data is determined by the user's public key, so each user's data is different). This data is stored in a Directed Acyclic Graph (DAG) structure, and the relationships between each data block are sent to the verifier in the form of a Merkle tree. Thus, the verifier can know which data the prover has stored based on the public key and the structure of the data as indicated by the Merkle tree.
In the verification phase, the verifier sends a "challenge" to the prover. This challenge is a random combination of the data blocks stored by the prover. The prover must generate the hash value of the corresponding combination of data based on the challenge information and return it to the verifier, who verifies whether the hash value is correct.
Since the challenge is a random combination of data, even slight differences in data will result in completely different hash values. Therefore, the prover must indeed store the data blocks indicated by the "challenge" to generate the correct hash value. The verifier, having stored the complete database, can also verify the hash value returned by the prover.
The prover may only store a small portion of data and still generate correct feedback through the verifier's challenge (the small portion of data stored by the prover happens to include the data combination in the challenge). However, as the "challenge" process is repeated multiple times, the probability of the prover generating correct feedback by storing a small amount of data significantly decreases. Thus, multiple verifications can be used to prevent cheating by the prover. This is the space confirmation process in PoSpace.
With a method to verify the storage space of users established, it is still necessary to determine the winner of the mining competition. A reasonable approach is that the miner with larger storage space is more likely to win the mining competition. PoSpace achieves this goal by designing a "quality function."
Quality Function#
The "quality function" needs to maintain a certain degree of randomness while distinguishing the probability of winning for each miner based on the size of their contributed space. A simplified approach is to use the hash value returned by the miner in response to the verifier's challenge (a string of numbers) directly as a random quantity and adjust this number based on the miner's occupied space. For example, if the total size of the space stored by the miner is N, then taking the hash value to the N-th power yields the quality function. This way, the larger the miner's storage space, the smaller the quality function value. We can stipulate that the miner with the smallest quality function in a single mining competition wins.
However, a problem still exists: since miners do not need to incur further costs after paying for hard drive space once, there is almost no cost to participate in the mining competition, making it easy to create main chain forks. To prevent miners from arbitrarily forking and causing double spending and other chaotic situations, a rule is still needed to determine which chain is the unique chain, and all users must record this unique chain, thus achieving true consensus.
Since each block is mined by the miner with the smallest "quality function," a natural idea arises: the unique main chain can be determined by the quality function. By setting a number i, the total quality function of the last i blocks can be summed up to obtain the total quality function of the chain. The chain with the smallest total quality function can be determined as the main chain. To emphasize that earlier blocks carry more weight, a discount function can be added to reduce the importance of earlier blocks.
Thus, when a main chain fork occurs, the total quality function of the two (or more) forked chains can be calculated to determine the unique chain, ensuring that only one main chain exists, thereby establishing a distributed and unified ledger system among users.
PoSpace uses physical hard drive space as proof of cost, solving the problem of PoW's continuous waste of resources in Bitcoin while establishing an electronic payment system that functions similarly to Bitcoin. PoSpace can be considered a significant advancement in consensus mechanisms based on PoW. However, PoSpace still has some issues: introducing the verifier role increases system risk; how to design and arrange verifiers remains a problem; using hard drive space as proof carries centralization risks, as a small number of individuals can purchase large amounts of hard drive space through significant financial resources, continuously monopolizing mining and causing a "51% attack." Satoshi Nakamoto's vision of "one CPU chip representing one individual, with each individual having an equal opportunity to mine" remains difficult to achieve.
Nevertheless, the PoSpace concept provides many insights, such as verifying the cost incurred by users through random methods; designing a block quality function to determine the winner of the mining competition; and designing a chain quality function to avoid main chain forks. Following this line of thought, it is entirely possible to develop consensus mechanisms suitable for different usage scenarios, such as "proof of attention" and "proof of time." Additionally, if the hard drive space stored in PoSpace is changed from meaningless bytes to meaningful content (such as videos and other materials), PoSpace may naturally be suitable for establishing a network resource-sharing community. It is believed that in the near future, consensus mechanisms will see further development and application.
Mimble-Wimble Technology#
In Bitcoin, full nodes need to download approximately 232GB of historical transaction data to verify new transactions. They must check all transactions submitted to the blockchain (which increase by about 300,000 daily) to obtain nearly 66 million UTXOs (Unspent Transaction Outputs). The requirement to check all historical transaction data significantly raises the threshold for becoming a Bitcoin full node, and lowering this threshold is crucial for ensuring the decentralization of the blockchain. Moreover, the historical ledger will continue to grow over time, making it increasingly difficult for ordinary personal computers to support the operation of Bitcoin full nodes in the near future.
Is there a way to allow full nodes to verify transactions without downloading all historical ledger data while still ensuring the security of the blockchain system?
At the same time, the existing privacy and anonymity of Bitcoin are not as good as imagined. Since transaction addresses and amounts are public, it is possible to analyze the identities of the transaction parties based on these transaction histories using big data analysis techniques, thereby threatening user privacy.
In fact, there are indeed solutions to the aforementioned problems in Bitcoin, which can not only reduce the historical ledger that originally required downloading 232GB to 50GB, significantly decreasing storage space and bandwidth usage, but also provide stronger privacy. This powerful technology is Mimble-Wimble.
Its original white paper was published in 2016, and the technology name and author are filled with magical colors; Mimble-Wimble is the "disappearing curse" from "Harry Potter." Like Satoshi Nakamoto, the author used the pseudonym Tom Elvis Jedusor (the French name for Voldemort in "Harry Potter") and disappeared after throwing down the white paper.
Implementation Principles#
In Mimble-Wimble, the reasons for ensuring privacy and scalability come from the following three points:
(1) There are no addresses in the blockchain; each time a transfer occurs, the receiver must construct a new transaction witness.
(2) The transaction amount is also hidden.
(3) Intermediate state transactions can be merged. Merging means that if, among all transactions to be packed into a block, money is first transferred from A to B, and then B transfers a certain amount to C, there is no need to record both transactions; it is sufficient to record how much A transferred to C, along with B's signature, ensuring the transaction's validity while significantly reducing the storage space required for the block.
Mimble-Wimble relies on ECC (Elliptic Curve Cryptography). In ECC, a very large number k is typically chosen as the private key. If H is a point on the elliptic curve, then k×H serves as the corresponding public key. The properties of the elliptic curve ensure that it is difficult to derive the private key k from the public key, as division of curve points is very challenging. Based on this property, it is possible to hide the actual transaction amount in transactions. The specific method is as follows:
Assuming the transaction amount is v, the node verifies that the output equals the input, which is equivalent to verifying v1 + v2 = v3. This equation is equivalent to multiplying both sides of the equation by the point H on the elliptic curve, thus needing to verify:
v1×H + v2×H = v3×H
While this makes it difficult to deduce the actual transaction amount, since the set of possible attempts is limited, an attacker may still deduce the value of v1. Therefore, a second elliptic curve point G and private key r are introduced, representing any input or output value in the transaction as r×G + v×H. Due to the properties of the elliptic curve, both r and v cannot be deduced, and the equation that needs to be verified becomes:
r1G + v1H + r2G + v2H = r3G + v3H
And it requires r1 + r2 = r3. This way, the actual transaction amount is well hidden. In actual transactions, only the parties know the transaction amount, while the nodes in the blockchain see the encrypted numbers, and the private key r is known only to the sender. To verify that the inputs equal the outputs while protecting the sender's private key from being cracked by the receiver, the sender must choose an additional value to add to their private key, so that from the receiver's perspective, only the sum of the two is visible, while only the sender knows the actual private key value. The method to verify that the transaction output equals the input and that the trader knows the additional value is to use it to construct an ECDSA signature. Thus, the additional value acts as a private key for a transaction, preventing double spending.
In merging transactions, the verification process is similar because the content that needs to be verified for a single transaction is that the output equals the input. Thus, the essence of the merged (including mixed and reduced intermediate states) transaction is also to verify that the final output equals the input, and the intermediate state transactions only need to verify their signatures. If double spending occurs, just like in Bitcoin, nodes can easily verify that the total transaction output does not equal the input. In Mimble-Wimble, nodes can compare the total sum of all funds generated through mining with the total amount held to check the correctness of the total currency supply. Range proofs can be used to ensure that no currency is issued excessively in transactions.
In summary, the main advantages of Mimble-Wimble are that it provides strong privacy while requiring minimal storage space, and it has high scalability. Since it does not require storing the entire historical transaction blockchain, by merging intermediate state transactions, only the source and current state of a coin need to be stored, saving a significant amount of space compared to other blockchains and making the amount of information that new nodes need to synchronize and transmit very small. However, it removes Bitcoin's scripting capabilities, and the encryption process requires some time, resulting in a block time of about 1 minute. Additionally, during transaction verification, both parties need to exchange some information, which may limit certain transaction functions.
As a newer and promising technology, its security needs to be tested over time, and a large number of developers must participate in testing and validation. Bitcoin technology has undergone years of development and should indeed be significantly optimized. Mimble-Wimble retains the advantages of PoW while optimizing the UTXO set and theoretically achieving significant storage space optimization, providing a new approach to improving scalability while maintaining decentralization, pointing to new prospects.
Since the launch of Bitcoin in 2008, the application of blockchain technology has mainly been divided into three stages:
(1) The blockchain 1.0 model, characterized by a programmable digital currency system, represented by Bitcoin, primarily solves the decentralization of currency payment methods.
(2) The blockchain 2.0 model, characterized by a programmable financial system, introduces the concept of "smart contracts," attempting to digitize any asset for trading, expanding blockchain from the initial currency system to the financial fields of securities trading and equity transfer.
(3) The blockchain 3.0 model, characterized by a programmable society, extends applications beyond currency and finance to all aspects of social life.
From an industry perspective, fields such as finance, copyright, healthcare, notarization, anti-counterfeiting, and supply chain have begun to recognize the importance of blockchain and are attempting to connect technology with real-world society. Blockchain technology can become a new tool, helping society reduce platform costs, rendering intermediary institutions obsolete, and potentially shifting the focus of existing business models, accelerating company development. At the same time, the data on the blockchain is expected to promote the transformation of traditional data recording, dissemination, and storage management methods. From a societal perspective, blockchain technology is expected to integrate law and economics, fundamentally overturning the original regulatory model of society, leading to changes in organizational forms. Blockchain may ultimately guide people toward a society of distributed autonomy.
● The products or services exchanged by both parties can be digitized;
● Services/products are standardized, and the evaluation system is clear and traceable;
● Services are provided by individuals, and services are consumed by individuals;
● As individuals gradually join, the value of the network increases.
A simple example is the shared mobility industry, where the service provider is primarily the driver (car owner), and the service consumer is the passenger. The driver takes the passenger from point A to point B, and the passenger pays the driver a certain fee, completing the transaction. The driver can calculate the driving distance using GPS navigation, record the driving time with timestamps, and automatically generate the fare according to a publicly announced calculation method. After delivering the passenger to their destination, the driver confirms, and the passenger's account automatically transfers the money to the driver's account. As more passengers and drivers join the blockchain network, drivers are more likely to receive more suitable orders, and passengers find it easier to get rides.
Many other industries, such as P2P insurance, credit, gambling, prediction, and gaming, can achieve decentralized transactions to solve trust issues through value "blockchainization."
Based on the underlying technology of blockchain, it is evident that the application of blockchain must face the "infeasible triangle" problem.
● Decentralization: Everyone has access to the entire ledger data and can participate in verification and recording;
● Consistency: Ensuring real-time consistency of transaction records among all participating nodes;
● Scalability and performance: Increasing transaction throughput and scale while reducing transaction latency.
Existing blockchain applications, such as Bitcoin blockchain, ensure decentralization and consistency, but have weak scalability, processing about 7 transactions per second; for example, POS/DPoS verification sacrifices a certain degree of decentralization, as not everyone needs to participate in verification, thus improving scalability performance by several orders of magnitude while maintaining ledger consistency.
When thinking of blockchain applications, the first area that comes to mind is digital currency. However, with the explosion of ICOs, the speculative nature of digital currencies has been wildly amplified, with some even becoming tools for fraud. Blockchain applications are still in their infancy and have not yet found real, concrete landing scenarios. Currently, applications are more common in finance, copyright protection, supply chains, etc. However, the technical rules are still not very compatible with existing commercial structures. For example, the current blockchain faces shortcomings in transforming financial infrastructure, including issues like irretrievability, inability to settle net positions, lack of securities lending, insufficient matching of transaction tokens with physical assets, and the potential for self-reinforcing feedback loops in smart contract execution, leading to financial instability.
Moreover, the application of blockchain must control the accuracy of input data while ensuring that output data interacts accurately with physical devices. For instance, the price assessment of assets on the blockchain may still require consensus among multiple parties to ensure the authenticity, accuracy, and completeness of the original data. In application scenarios where blockchain and the Internet of Things are combined, the chaotic nature of the real physical world must also be considered, as there may be many uncertain human or accidental interference factors. For example, the Juicero problem: a computer ecosystem can be established and linked to bagged fruits to squeeze the fruit bags, but not everything that happens with the fruit bags can be controlled by the computer; a person can squeeze the bag with their hand. How to ensure that the data input into the blockchain accurately maps the real world and that the blockchain's output is correctly implemented and operated is a critical issue for the successful application of blockchain. Thus, the slow pace of industry landing is normal, caused by multiple factors, including the need for further maturity of underlying technology, lack of smart contract public chain platforms, insufficient compatibility of various Token ecosystems, and unclear regulatory policies.
Finance creates value through the operation of funds. Robert Merton, a renowned finance professor at Harvard University, believes that the financial system has six basic functions:
(1) Clearing and payment functions: The financial system provides clearing and payment methods for transactions of goods and services. One of the highlights of the rise of internet finance is the significant change in payment methods.
(2) Financing and equity refinement functions: The financial system can legally integrate funds, allowing funds to flow from one party to another, achieving fund integration during the flow of funds, which is crucial for the construction of large projects and enterprises with development potential. Equity financing divides large investment projects into small shares for participation by small and medium investors, achieving equity refinement.
(3) Resource allocation function: Individual investors often find it challenging to make reasonable judgments about market investment environments and company investment expectations. Financial intermediaries help investors make investment decisions, dispersing investment risks and optimizing the allocation of social capital investments.
(4) Risk management function: Due to transaction costs and information asymmetry in financial markets, financial investments carry varying degrees of risk. The financial system can price, trade, disperse, and transfer risks, ensuring that financial market risks are reasonably allocated.
(5) Incentive function: This incentive mainly refers to equity issues. For enterprises, if their employees hold a certain amount of the company's stock or stock options, it can incentivize employees.
(6) Information function: In financial markets, fundraisers can obtain cost information for various financing methods, while investors can access information about investment product prices and influencing factors.
The essence of finance is credit, as credit enables the circulation and allocation of funds between different times, places, and participants. Therefore, there are many intermediary institutions in the financial industry, including banks, third-party payments, and asset management institutions. However, this centralized model has many problems. Centralization means high intercommunication costs between centers, time-consuming and labor-intensive communication, low operational efficiency, and centralized nodes are vulnerable to attacks and tampering, making data insecure.
The financial system has developed to form a stable scale and structure, but there are still some issues that need improvement in practical operations.
● Cross-border payment cycles are long and costs are high: The financial industry undertakes the payment function. When both parties engage in transactions of goods and services, there may be various payment methods depending on the situation. Based on the transaction parties, payments can be categorized into direct and indirect types. Direct payments occur simultaneously without the need for third-party guarantees or turnover. Indirect payments require both parties to rely on a third-party payment company, which must account for the transaction through the third-party company's bookkeeping before completing the payment. Due to geographical and trust issues in cross-border payments, third-party companies are needed for turnover. The actual cross-border payment arrival cycle typically takes about a week and incurs high costs. For example, PayPal charges a standard cross-border transaction fee of 4.4% + $0.3.
● Financing cycles are long and costs are high: For small and medium enterprises, third-party credit institutions need to spend a lot of time reviewing various certificates and accounts, resulting in high financing costs.
● The bill market has significant centralization risks: The bill market relies on central commercial banks, and if the central server fails, the bill market will collapse. Additionally, issues such as double-selling bills and asynchronous endorsements of electronic bills often arise.
● The authenticity of underlying assets is difficult to discern: Funds, asset securitization, and various wealth management products are all packaging of underlying assets for direct purchase by investors. The sources of underlying assets are diverse, ranging from standardized bonds, stocks, public funds to non-standardized accounts receivable and rights to income. Transforming underlying assets into products directly available for investor purchase requires the participation of multiple parties, and the transaction process faces information asymmetry, making it difficult for various trading institutions to grasp the authenticity and accuracy of underlying assets.
Finance is a form of credit transaction, and credit is the foundation of finance. Therefore, there are many intermediary institutions, including banks, governments, and others. To some extent, these institutions have value in existence, as they can reduce the credit cost of transactions and ensure that transactions proceed normally, but we must pay some costs for this, commonly seen as various fees. Blockchain, as a form of distributed ledger technology, with its publicly transparent and immutable ledger information, can serve as a means of credit and lending, reducing trust costs, transforming trust from centralized credit institutions to the data trust of blockchain ledgers.
Furthermore, blockchain technology can provide value circulation functions for currencies, better integrating into the financial industry system. Therefore, blockchain technology has a strong adaptability to the broad financial industry.