Testnet Retro - Launch Day

On May 1st, 2025, Aztec Public Testnet went live. 

Within the first 24 hours, over 20k users visited the Aztec Playground and started to send transactions on testnet. Additionally, 10 apps launched live on the testnet, including wallets, block explorers, and private DeFi and NFT marketplaces. Launching a decentralized testnet poses significant challenges, and we’re proud that the network has continued to run despite high levels of congestion that led to slow block production for a period of time.  

What happened leading up to the network slowdown 

Around 6 hours after announcing the network launch, more than 150 sequencers had joined the validator set to sequence transactions and propose blocks for the network. 500+ additional full nodes were spun up by node operators participating in our Discord community. These sequencers were flooded with over 5k transactions before block production slowed. Let’s dive into why block production slowed down. 

On Aztec, an epoch is a group of 32 blocks that are rolled up for settlement on Ethereum. Leading up to the slowdown of block production, there were entire epochs with full blocks (8 transactions, or 0.2TPS) in every slot. The sequencers were building blocks and absorbing the demand for blockspace from users of the Aztec playground, and there was a build up of 100s of pending transactions in sequencer mempools. 

Why did block production slow? 

Issues arose when these transactions started to exceed the mempool size, which was configured to hold only 100mb or about 700 transactions. 

As many new validators were brought through the funnel and started to come online, the mempools of existing validators (already full at 700 transactions) and new ones (at 0 transactions) diverged significantly. When earlier validators proposed blocks, newer validators didn't have the transactions and could not attest to blocks because the request/response protocol wasn't aggressive enough. When newer validators made proposals, earlier validators didn't have transactions (their mempools were full), so they could not attest to blocks. 

New validators then started to build up pending transactions. When validators with full mempools requested missing transactions from peers, they would evict existing transactions from their mempools (mempool is at max memory) based on priority fee. All transactions had default fee settings, so validators were randomly ejecting transactions and were not doing so in lockstep (different validators ejected different transactions). For a little over an hour, the mempools diverged significantly from each other, and block production slowed down to about 20% of the expected rate.

What happens next?

In order to stop the mempool from ejecting transactions, the p2p mempool size was increased. By increasing the mempool size, the likelihood of needing to evict transactions that might soon appear in proposals is reduced. This increases the chances that sequencers already have the necessary transactions locally when they receive a block proposal. As a result, more validators are able to attest to proposals, allowing blocks to be finalized more reliably. Once blocks are included on L1, their transactions are evicted from the mempool. So over time, as more blocks are finalized and transactions are mined, the mempool naturally shrinks and the network will recover on its own. 

If you are interested in running a sequencer node visit the sequencer page. Stay up-to-date on Noir and Aztec by following Noir and Aztec on X.