Why Would a Public Blockchain Stop? Understanding Blockchain Consensus from Cosmos's 25-Hour Downtime

Author: imToken

On the evening of September 22, a user initiated an ATOM transfer.

After a night passed, the transaction remained in "waiting for confirmation."

The private key was not lost, and there were no signature anomalies in the wallet. Upon checking again the next day, multiple public RPCs showed that the Cosmos Hub was stuck at block height 33,086,740.

No new blocks were produced, and naturally, there was no place to package this transaction.

It wasn't until about a day later that the Cosmos Hub resumed block production, and the ATOM transfer that had been in waiting status finally succeeded.

For ordinary users, this may be the most intuitive lesson in understanding blockchain consensus.

We often say, "No central authority can shut down a public blockchain," but the reality is clearly much more complex. A sufficiently decentralized blockchain indeed usually does not have a "shutdown button" in a server room, yet it can still stop.

This time, the pause of the Cosmos Hub has fully exposed this underlying mechanism to ordinary users.

1. Why Did Cosmos Suddenly "Stop Producing Blocks"?

First, it is necessary to clarify a commonly confused issue: the Cosmos Hub was not directly attacked.

The incident first occurred in Neutron.

On September 22, a governance proposal called "AIATO: AI Agent Takeover" was passed in Neutron, and the attacker exploited a loophole in the chain-level governance authority, using privileged commands provided by the wasmd framework to change the contract administrators of applications like Astroport and Drop to addresses controlled by the attacker.

This is not what we typically understand as a "code vulnerability" or "protocol flaw."

It can be simply understood that the application itself has its own "lock," but the chain-level governance of Neutron holds a higher-level "master key," and when the attacker controls the governance outcome, it is equivalent to obtaining this key, allowing them to reassign administrators, migrate contracts, and further transfer assets within.

What truly brought the Cosmos Hub into the fray was the subsequent cross-chain fund transfer.

Cosmos Labs' review showed that before Neutron stopped running, the attacker had transferred some assets to multiple networks, including approximately 1.7 million ATOM transferred into the Cosmos Hub, which began to be exchanged through cross-chain liquidity.

In other words, the Cosmos Hub itself was not directly attacked, and ordinary Hub users' funds were not directly stolen due to the Neutron vulnerability.

However, the ATOM obtained from the attack had already entered the Hub, and to prevent the remaining ATOM from continuing to flow out, some Cosmos Hub validators began to stop their nodes from running.

By around 19:18 (SGT) on September 22, the validators that had stopped running represented more than one-third of the total voting power, causing the Cosmos Hub to be unable to continue forming new blocks, ultimately stopping at 33,086,740.

This step is crucial.

It means that there is no "Pause" button that a company can directly click on the Cosmos Hub, nor was there a prior on-chain governance vote. What truly caused the network to stop was that enough validators ceased to participate in forming consensus.

But what is even more noteworthy is the subsequent recovery process.

About 4 hours after the chain stopped, the validators received a complete recovery plan: to execute a one-time state modification at the height of the chain stop, transferring the remaining ATOM from the attacker's address to a multi-signature address jointly managed by community validators.

Subsequently, Cosmos Labs created a patch Gaia v28.3.0 based on the plan agreed upon by the validators, tested it, and distributed it to the validators.

This version of Gaia would execute a one-time state change at the specified recovery height, transferring 1,227,121 ATOM from the attacker's address to a 4-of-6 multi-signature address composed of Nansen, Keplr, Enigma, Silknodes, Kiln, and Polkachu.

By the early morning of September 23, the validators that confirmed the installation of v28.3.0 had exceeded 67% of the total voting power, so at 12:00 UTC that day, the Cosmos Hub coordinated a restart. About 6 minutes later, this one-time state modification was executed at block height 33,086,741, and the network resumed normal block production.

In essence, the entire process from the Cosmos stopping block production to recovery involved validators first causing the network to lose liveness, and then more than two-thirds of the voting power accepting a new set of state transition rules, ultimately allowing this set of rules to become the canonical state after recovery.

At this point, a seemingly simple question arises: Since it is a decentralized public chain, why can more than one-third of the validation power stop it, while recovering the network requires enough validators to collectively accept and run the same software?

The answer is actually hidden in the word "consensus."

2. Consensus Is Not About "Never Stopping"

One of the most easily misunderstood aspects of blockchain is equating "decentralization" with "never going down."

In reality, the consensus mechanism truly addresses the problem of how many nodes can agree on the transaction order and ledger state without a central bookkeeper.

However, different public chains implement this in different ways.

For example, Bitcoin's classic method is PoW, or proof of work—miners compete to produce blocks based on computational power. When two legitimate branches appear in the network for a short time, nodes choose one to continue building based on accumulated work.

Thus, Bitcoin does not have a clear moment of "67% voting after this block is forever finalized"; it is closer to a probabilistic finality, the more subsequent blocks there are, the higher the computational cost required to reorganize previous transactions.

This is why it has been said in the past that a Bitcoin transaction is best waited for 6 block confirmations; after all, no matter how high the computational power, it cannot simply bypass the consensus rules being executed by the nodes.

Of course, this does not mean that Bitcoin's state is "absolutely unmodifiable" under any circumstances. Theoretically, if the entire ecosystem accepts a new client and new consensus rules, a hard fork can also make state changes that were invalid under past rules become valid.

But the problem lies here: who has the ability to get enough miners, full nodes, trading platforms, wallets, and users to accept such a new set of rules?

Almost no one.

The development team cannot decide the consensus rules for the entire Bitcoin network, and miners and trading platforms find it difficult because the consensus threshold that needs to be crossed is very high. When Binance was hacked for 7,000 BTC, someone suggested CZ contact large miners to operate, but it ultimately came to nothing.

Ethereum provides another classic example.

After transitioning to PoS, Ethereum now uses a consensus called Gasper, which combines Casper FFG and LMD-GHOST. Simply put, part of the mechanism is responsible for determining "which chain should be followed currently," while another part is responsible for ensuring that blocks achieve true finality.

When validators representing at least two-thirds of staked ETH reach consensus on the corresponding checkpoint, the block can further move towards final confirmation; conversely, if more than one-third of the stake does not participate in correct voting for a long time, the network may temporarily be unable to form finality. However, Ethereum has also designed an inactivity leak, which gradually reduces the effective weight of offline validators when finalizing is not possible for a long time, giving the network a chance to restore finality.

To truly change this outcome, it also requires changing protocol rules and clients.

Just like in the 2016 DAO incident, the Ethereum community ultimately executed a special state modification directly referred to as an irregular state change by the Ethereum Foundation at block 1,920,000, transferring the relevant ETH into a recovery contract.

However, some miners and community members who refused to upgrade and continued to maintain the original state ultimately formed Ethereum Classic (ETC), leading to the well-known fork between ETH and ETC, indicating that not everyone accepted this set of rules.

Cosmos Hub is different; it uses CometBFT, which is closer to a typical BFT consensus.

It can be understood as a more typical BFT consensus, meaning that for a block to be truly submitted, it needs to obtain a commit from more than two-thirds of the voting power.

The advantage is that finality is very clear; once a block is submitted through sufficient voting power, there is no need to continue waiting for more blocks like in PoW, trading probability for security.

But its other side is very direct: if one-third or more of the voting power no longer provides the votes needed to form a commit, the remaining validators cannot reach more than two-thirds, no matter how hard they try.

At this point, the safest choice for the network, just like this time's "pause in block production," is to stop. Therefore, from the perspective of distributed systems, this short pause of the Cosmos Hub is not mysterious at all.

In summary, when a group of validators with sufficient voting power stops participating, the consensus protocol, according to its own rules, would rather lose availability than continue to confirm new blocks in the absence of sufficient consensus.

This actually corresponds to two concepts in distributed systems that are often confused by ordinary users:

• Safety: Preventing different nodes from simultaneously confirming two conflicting final states;
• Liveness: Whether the network can continue to run and process new transactions;

For BFT systems, when the number of nodes participating in consensus is insufficient, pausing is sometimes precisely the cost paid to maintain safety. To put it bluntly, this decentralized ledger would rather stop there than allow the remaining participants to each keep their own records.

Looking back from this perspective, we can see that many seemingly different incidents in the history of public blockchains actually revolve around the same issue:

What should the network do when distributed nodes cannot reach a consensus on the "correct state"?

III. From Bitcoin to Solana, What Are the Real Risk Boundaries of Public Blockchains?

This is not the first time Cosmos has brought this issue to the forefront.

As early as 2013, Bitcoin experienced a very classic chain fork incident.

At that time, Bitcoin 0.8 switched its underlying database from Berkeley DB to LevelDB, and subsequently, a block containing a large number of transaction inputs appeared. The new version of the node could process it normally, but some old version nodes, due to the limit on the number of Berkeley DB locks, deemed this block invalid.

An awkward scene unfolded: everyone was running Bitcoin, but the new and old clients began to produce different answers regarding whether "this block is valid."

As a result, the network split into two chains, and at one point, the new version 0.8 side had about 60% of the hash power, making it impossible to rely on normal hash power competition to quickly converge.

Ultimately, large mining pools coordinated to revert to the old version, regaining more hash power on the old rules side, allowing the network to converge again. Bitcoin later specifically reviewed this incident with BIP 50.

In 2016, the Ethereum DAO incident pushed the issue a step further.

As mentioned above, the Ethereum community ultimately executed a special state modification explicitly referred to by the Ethereum Foundation as an irregular state change through a Hard Fork at block 1,920,000, transferring the relevant ETH to a recovery contract.

However, not everyone agreed with this handling. Some miners and community members who refused to accept the state modification continued to maintain the original rules, leading to the long-standing existence of Ethereum Classic (ETC).

This DAO Fork is also a classic event, essentially telling everyone that when extreme events occur, there exists a social consensus beyond code consensus. If sufficient agreement cannot be formed, a chain can indeed split into two.

In 2021, Solana showcased a completely different failure path.

In September of that year, a large number of bot transactions flooded the network, causing the validator nodes' memory to run out, leading to the crash of many nodes. Ultimately, the entire network could not reach a consensus on the current state, halting the confirmation of new blocks for about 17 hours, after which validators coordinated to restore the network.

When we put these incidents together, we find that they are not the same:

• The issue with Bitcoin in 2013 was that different clients began to execute different validity rules;
• The issue with Solana in 2021 was that a large number of validator nodes could not continue to participate in consensus, causing the network to lose liveness;
• The Ethereum DAO faced a question closer to whether the community should actively modify the state through new protocol rules;
• And this time, Cosmos Hub had another layer of specificity, where the network proactively lost liveness through validator coordination to prevent the attack assets from continuing to move; afterward, a sufficiently high proportion of validators collectively accepted new software and recovery states, allowing the network to re-establish consensus;

Therefore, rather than simply attributing these events to "blockchains can also shut down" or "decentralization is a myth," it is better to acknowledge a more realistic fact:

The consensus mechanism has never been a machine that cannot break down; what it truly provides is a set of decentralized rules, such as who decides the correct chain when disagreements occur; how many participants are needed for a state to achieve finality; whether the network chooses to continue running or stop in the event of a failure; and in extreme cases, what kind of collective action can change the subsequent operational rules.

This also leaves the Cosmos incident with a question more worthy of ordinary users' consideration than "should the chain stop?"

Final Thoughts

We often say, not your keys, not your coins.

This statement still holds true; it emphasizes asset control— as long as the private key is in your hands, wallets, trading platforms, or other third parties cannot sign a transaction on your behalf.

The premise is that the blockchain you are on must have the capability to process this signature at any time.

On the day Cosmos Hub stopped producing blocks, users still held their private keys, and assets did not disappear into thin air; it was just that even if you correctly signed a transaction, there were no new blocks to accept it.

The recovery process further illustrates that if enough consensus participants accept a new set of state rules, the on-chain state of specific accounts may change even without the original address's private key signature.

This does not invalidate "Not your keys, not your coins," but reminds us that private key autonomy and underlying consensus authority are never the same thing.

The same applies to wallets.

A wallet can ensure that the private key and signing authority are in the user's hands, can quickly identify chain-level anomalies, accurately display transaction statuses, establish RPC and node redundancy, and reconfirm the final results of transactions after the network recovers.

However, a wallet cannot restore consensus for a public chain, cannot guarantee that the underlying network will never be interrupted, and cannot ensure that the rules and states on the chain will never undergo consensus-level changes.

Therefore, what a mature decentralized system truly needs to pursue may never be about "nothing should ever change"; rather, it should clarify these imperfect boundaries as much as possible: who can pause consensus? What weight is needed? Under what circumstances is emergency intervention allowed?

Because true decentralization cannot prevent the system from encountering incidents forever; the key is that even when incidents do occur, we can still know who, based on what rules, and with what level of consensus, decides how to record this account going forward.

This content is provided for general informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online promotions, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets. Crypto assets are highly volatile and may result in loss. The availability of WEEX services, products, and related events may vary by region. You are responsible for ensuring that your participation is in accordance with applicable local laws and regulations.