Ethereum All Core Developers Consensus Call #106 Writeup
On April 6, 2023, Ethereum developers gathered for their 106th All Core Developers Consensus (ACDC) call. Chaired by Ethereum Foundation Researcher Danny Ryan, the ACDC calls are a bi-weekly meeting series where Ethereum developers discuss and coordinate changes to the consensus layer (CL) of Ethereum. This week, developers gave updates on their progress around the Deneb upgrade and discussed open issues around recent MEV-Boost software releases, which appears to have caused a high number of orphaned blocks.
On April 12, 2023, the Ethereum CL will activate the Capella upgrade to enable staked ETH withdrawals. Given that staked ETH withdrawals require changes on both the EL and CL of Ethereum, the Capella upgrade will activate simultaneously with an upgrade on the EL dubbed Shanghai. To test the final client software versions that have been released for both upgrades, developers launched their third mainnet shadow fork on Wednesday, April 6. As background, shadow forks are a specific type of test network that is created from forking an existing blockchain. The benefit of shadow forks is the ability to replicate state and replay transactions from an existing blockchain in a test environment.
Mainnet Shadow Fork 3 (MSF3) for Shanghai and Capella, or Shapella, went smoothly according to Paritosh Jayanthi, a DevOps Engineer for the Ethereum Foundation. On MSF3, developers tested two MEV block builders and the standard procedure for staked ETH deposits, as well as withdrawals, and withdrawal credential changes. Jayanthi noted there were a few problematic MEV-Boost block signatures discovered on the testnet, though the root cause of this has not yet been identified.
Before moving on to Deneb progress updates, Marius van der Wijden, a developer for the Geth (EL) client, raised a housekeeping item around standardizing transaction types for mainnet. “I would like to modify the block transaction type from 0x5 to 0x3. I think it was chosen in that way because there were some different transaction types of on some Layer 2s or something. I don’t think we should create the expectation that mainnet would care about transaction types on other networks and [so] I would like to modify the transaction type.” Developers were largely in agreement with this sentiment to standardize how transaction types are set on mainnet. Ansgar Dietrichs, Researcher for the Ethereum Foundation, noted that developers should check to ensure the proposed change to the mainnet transaction type does not impact and cause problems to existing applications or wallets. Developers agreed to coordinate with others outside of the call and schedule the change in an upcoming code release.
Then, developers went around the horn and have updates on their status for the Deneb upgrade, which is the name of the next CL upgrade after Capella.
Prysm: Working on gossip features, blocks by range, and blocks by route. Does not yet have backfilling functionality. Ready to pair and interop with other clients on testnets.
Lighthouse: Working on single blob lookups and pre-processing functionality. Gearing up to releasing a testable client version that could run on a testnet.
Lodestar: Ready to pair and interop with other clients on testnets.
Teku: In a building phase. Working on the API for signing, cashing, gossiping, and deconstructing bundles from blobs and blocks. Not ready to interop with other clients at the moment.
Danny Ryan and Mikhail Kalinin from the Teku (CL) client team raised two minor features that could be included in Deneb as minor improvements to CL specifications. Given that neither of them are critical updates, Ryan and Kalinin agreed to source more feedback from client teams outside of the call. Ryan also highlighted PR #3307, which proposes reusing the validator index numbers of validators that have fully exited from the Beacon Chain. This would ensure that there are no large gaps in the validator index and prevent additional computational resources from being expended when CL nodes scan the validator set by index number. Ryan encouraged developers on the call to review the PR and leave comments.
Then, Chris Hager from the Flashbots team gave an update on the latest MEV-Boost releases, which were created in response to a rogue validator stealing MEV from searchers on Sunday, April 2. Read this week’s Galaxy Research Brief for the full explanation of the MEV-Boost exploit. Two important changes that are pushed to MEV relay software were:
Enforcement of block validation before broadcasting
Implementation of block submission cutoff times, ~4 seconds
Due to the additional latencies around block production created by the first change, the number of orphaned blocks on Ethereum jumped from ten per day to four per hour. Ryan questioned whether the blocks being rejected were ones produced by honest block proposers or dishonest ones trying to game the system. Hager responded that further investigation around this was still needed. “I do think that providing more visibility into honest, late [block] reorgs would be valuable because I do think that a number of entities might have begun trying to modify behavior to get [blocks] and broadcast late. And this combined with modified relay behavior may be amplifying the problem. So these actors will probably figure it out themselves but just making clear that essentially there’s a modification that is utilized by the majority of the network I think would help,” said Ryan.
To this, Hager agreed that more monitoring was needed and that collaboration with relay operators would be helpful to identifying and analyzing data. Hager also pointed out that in general more security research on the MEV-Boost codebase is needed. He suggested perhaps “a coordinated bug bounty program that is not solely funded by Flashbots.” To this, Ryan resurfaced an open pull request on GitHub that specifies certain fork choice rules around MEV-Boost that more than 70% of node operators utilize already. On the topic of moving MEV-Boost specifications more generally into CL specifications, Marius van der Wijden said that he wasn’t opposed to the idea or the idea to move MEV-Boost into the general bug bounty program funded by the EF. However, “I’m very much opposed to painting this as something that was Ethereum’s fault or something. This was definitely an issue of the service providers and not like the general Ethereum ecosystem,” said van der Wijden.
On the topic of making it easier to verify blocks, a Prysm developer by the name of “Potuz” mentioned earlier discussions with client teams about creating a RPC endpoint through which blinded blocks could be submitted and checked for validity. Blinded blocks refer to blocks where the transaction contents cannot be seen. “My worry is that by adding things that make the situation continue on is simply diverting us from realizing that we need [proposer-builder-separation] in-protocol. Facilitating the continuation of the status quo is not something that I’m happy about,” said Potuz. Ryan suggested that a pull request specifying the RPC endpoint be created for further discussion next week, alongside any new data and analysis on block proposer behaviors.