On Tuesday, February 6, the Solana blockchain network encountered a significant outage lasting nearly 5 hours, rendering the platform inoperative. While the issue has since been resolved, Matthew Sigel, head of digital assets at VanEck, offers insights into the root causes behind the outage. The Solana network stopped processing blocks. Five hours later, the developers fixed the error by rebooting 80% of the stack. The blockchain began operating normally from slot 246,464,041.
BFP Loader Failure Resulted in the Solana Outage
Matthew Sigel stated that the outage was caused by a malfunction in the BPF loader, or “Berkley Packet Filter,” which serves as the mechanism for deploying, upgrading, and executing programs on Solana. According to Sigel, the problem originated from a Solana Improvement Proposal (SMID) that introduced modifications, including the addition of a blocker to prevent the use of metadata in the BPF.
This modification was part of upgrade 0093. While a solution was developed after the glitch was identified on the testnet, it had not been implemented pending further testing. There is speculation that a manual trigger prompted the bug, resulting in the network’s downtime.
To resolve the issue, developers have rewritten the BPF code lines on the development network. This solution requires patching the core software utilized by all participants on the network before regular operations can resume.
Solana outage, what happened?
BPF loader, the “Berkley Packet Filter,” which is the mechanism to deploy upgrade and execute programs on Solana, failed. This seems to relate to a previous SMID (Solana Improvement Proposal) that altered some of the features including the adding a…— matthew sigel, recovering CFA (@matthew_sigel) February 6, 2024
The subsequent steps in restarting the network entail a community review of the patched core software. Validators will then create a snapshot of the last authenticated block, followed by a consensus process to validate the block.
Once consensus is reached, validators can begin executing the patched software. Although block production may initially occur without being appended to the chain until 66% of the network concurs on the blocks. Network activity will fully resume once 80% of the network agrees on the last block. However, there is a possibility of further interruptions if the fix proves inadequate.
Can the BFP Loader Fail Again?
Matthew Sigel assures that efforts to address the issue are already underway. Looking ahead, Sigel highlights potential second-order effects resulting from the restart of the Solana network. He anticipates a surge in decentralized finance (DeFi) activity as arbitrage bots capitalize on opportunities that arose during the downtime.
Estimates suggest that this activity could yield up to $25 million in Maximum Extractable Value (MEV). However, Sigel cautions that the influx of MEV-related activity could potentially trigger additional downtime, thus hindering innovation on the Solana network. He suggests that future Solana Improvement Proposals (SMIDs) may encounter heightened scrutiny and debate, citing the ongoing discussions surrounding fee markets as an instance of this evolving dynamic. Solana’s price has observed the losses after the initial dip. At the time of writing, SOL is trading at $95.
