Why Consensus Is the Hardest Problem in Blockchain
Consensus allows a decentralised network to agree on the sequence of events and the state of records without a leader. For blockchains, this means the database can operate across the globe even if some users are malicious, offline, or attempting to game the system.
This guide defines consensus from the ground up. Then it covers how Bitcoin's proof-of-work (PoW) consensus solves the Byzantine Generals Problem. The sections that follow discuss mining, difficulty adjustments, forks, mining pools, currencies, network participants (nodes and wallets), and why newer blockchains are trying different methods like proof of stake.
The Byzantine Generals Problem
The Byzantine Generals Problem is a thought experiment in computer science. Generals are camped outside of a city and must collectively decide whether to attack or retreat.
- Messages might be late or lost.
- Some generals might not get the message.
- Some may deliberately send the wrong orders.
How can everyone agree if some participants can fail or be malicious? Blockchains have the same problem. When there is no leader like on a public blockchain, nodes need to agree on:
- Which transactions are real
- Which block comes next
- Which chain is the correct record
Why Consensus Is Hard in Open Networks
Centralized systems rely on some authority figure to fix things. A bank clears transactions, a website edits pages, and your employer updates records. Blockchains eliminate the central leader, so the community as a whole must police the rules.
A working consensus system should deal with:
- Bad behavior such as spam, fraud, and spending money twice
- Delays because messages aren't instant
- Network splits where parts of the network might be cut off
- Size across thousands of nodes in many countries
Consensus creates value for a ledger. Consensus doesn't just store information; it validates it.
Bitcoin's Solution, Proof of Work
Bitcoin reaches consensus through proof-of-work, known as the Nakamoto consensus.
The Core Idea
To create a block, miners must solve a challenge. The puzzle is hard to solve, but easy to check. If it was hard to check blocks then every node would struggle. If it was easy to create blocks, attackers could rewrite blockchain history.
How Proof of Work Functions
Miners compute block header information and modify a number called a nonce. Miners attempt to find a hash that satisfies a particular condition, usually beginning with a certain number of zeroes.
A real block has:
- A link to the last block (joining the chain)
- A summary of transactions (a Merkle root)
- A timestamp
- The difficulty
- The nonce that makes the hash meet the target
When a miner solves the puzzle, three things happen in sequence:
- The block is broadcast to the network.
- Other nodes check the proof.
- The network builds on that block.
The Longest Chain Rule
Bitcoin reaches consensus on the correct chain using proof-of-work. It is sometimes referred to as the longest chain, but what actually matters is work.
Occasionally two miners will mine a block at nearly the same time. The network forks temporarily as different nodes see different blocks first, resulting in multiple tips. One chain will receive another block, which the network then accepts. The other block is called a stale block. Stale blocks were mined correctly, but the network no longer uses them.
Forks, Temporary and Permanent
Not all forks are the same.
Temporary Forks
Forks happen when blocks are propagated at different times. The network eventually resolves these when one fork becomes longer than the other.
Hard Forks
Hard forks happen when a community decides to adopt incompatible rules. Two blocks will be created that are incompatible with each other, creating two networks that continue on independently. This is how Bitcoin and Bitcoin Cash are two separate networks with their own currencies and communities.
Consensus is both technical and social. The code may enforce rules, but the community chooses if a change to the rules makes a new version or fork.
Difficulty Adjustment
Bitcoin attempts to time one block every ten minutes. But mining power will vary as hardware is added to the network, miners drop off, or mining moves globally.
Bitcoin addresses this by adjusting difficulty every two weeks. If blocks are generated too quickly, the puzzle will be more difficult. If blocks are slow, it will be easier.
Mining has become competitive because:
- Hashing power rose rapidly
- Hardware evolved from CPUs to GPUs to ASICs (hashing machines)
Proof-of-work competitions are the reason why PoW systems are often accused of wasting resources. Some networks have chosen to look elsewhere.
Mining Pools
As mining a block is a matter of luck, solo mining is essentially buying lottery tickets and hoping you win once every few years. Mining pools allow miners to group together and claim rewards.
A pool operator does four things:
- Coordinates work
- Shares results
- Makes blocks
- Pays rewards
Pools take a cut. They centralize mining into mining collectives.
Currency and Incentives
Bitcoin's consensus system incentivizes participants with money. Miners collect coins and transaction fees when they successfully mine blocks. Coin rewards are affected by live crypto prices which is why tracking them allows miners and investors to react swiftly.
This system serves two main purposes:
- Security. It makes miners protect the chain.
- Issuance. It releases new coins over time.
Bitcoin's maximum supply is 21 million coins. Newly created coins become available at a steadily decreasing rate through "halving." This process continues until roughly 2140. Other networks use similar mechanisms. Ethereum's block rewards, known as "gas," and issuance policy have shifted with upgrades.
The Network Layer
Consensus is not only about miners. It relies on how the network is structured as well: who stores data, who validates transactions, and how information flows through the network.
Key roles:
- Full nodes. Store the blockchain and check transactions and blocks.
- Pruning nodes. Check like full nodes but discard old data to save room.
- Lightweight nodes (SPV). Store only block headers and rely on full nodes for verification.
- Miners. Perform proof of work, often via pools.
- Wallets. Manage keys and make transactions. See the full guide to crypto wallets for more.
- Mempool. A "waiting room" of real transactions spread across the network.
A blockchain must propagate messages in order to reach consensus. When partitions happen, multiple histories can exist for some time. Monetary incentives can ensure re-joining.
Beyond Proof of Work
Proof of work isn't the only consensus mechanism. Alternatives try to use resources differently, or reach consensus by selecting validators in new ways.
Alternative approaches include:
- Proof-of-stake (PoS). Validators are selected based on their stake and the rules.
- Selection mechanisms. The protocol chooses who proposes or validates blocks.
- Hybrid designs. Combine work, stake, voting, or checks.
- Resource mechanisms. Designs based on capacity or storage.
These approaches balance between:
- Decentralization
- Performance
- Security
- Complexity
The core question for any design: who mines the next block, and why should others believe it?
Why Consensus Matters
Consensus mechanisms are what make blockchains safe. They enable the network to agree on a ledger state despite delays and malicious actors. Bitcoin's proof-of-work consensus works because it makes blocks difficult to produce but easy to validate. It incentivizes miners and allows the longest chain to win.
Blockchains aren't just lines of code. They're about currency and networking too. Incentives. Hardware. Transmission of messages. Sociology of cooperation. All of these factors determine the agreed-upon reality of a network.
