Ethereum All Core Developers Consensus Call #103 Writeup
On February 23, 2023, Ethereum developers gathered for their 103rd All Core Developers Consensus (ACDC) call. Chaired by the Ethereum Foundation’s Danny Ryan, the ACDC calls are one of two bi-weekly meeting series where Ethereum developers discuss and coordinate changes to the protocol of Ethereum. ACDC calls focus primarily on development for the consensus layer of Ethereum, while the other meeting series, All Core Developers Execution (ACDE) calls, focuses on coordinating code changes to the execution layer of Ethereum. To read a summary of last week’s ACDE call, read this Galaxy Research report.
This week, consensus layer (CL) client teams discussed progress on testing for the Shanghai and Capella upgrade. Developers have started to test MEV-Boost, builder, and relay software a few test networks where the Shanghai upgrade has been activated like the Zhejiang testnet and Devnet 7. Tim Beiko, Chair of the ACDE calls, has published the official blog post announcing the date and final client releases for the upcoming Shanghai activation on the Sepolia testnet. Beiko also shared that next week EIP 4844 Implementers’ Call will be cancelled. This bi-weekly meeting series which focusses on development for the Deneb upgrade will reconvene on March 7.
Testing Shanghai & Capella
Barnabus Busa, a DevOps Engineer at the Ethereum Foundation, said Devnet 7 was deprecated Thursday morning, February 23, after processing 359,000 validator withdrawal credential changes and testing a number of different client releases. He stressed that the client versions running on Devnet 7 were not the final versions put out by the various CL and EL teams. Therefore, Busa recommended spinning up Devnet 8 in early March after the Shanghai upgrade on Sepolia to test final client releases. Busa also mentioned that Devnet 7 was useful for catching minor issues and bugs in the clients.
Then, Paritosh Jayanthi, another a DevOps Engineer for the Ethereum Foundation, gave an update on how testing is going on the Zhejiang testnet. He highlighted that MEV-Boost software is being tested alongside staked ETH withdrawals and so far, no issues have been identified. The next step, according to Jayanthi, is to test MEV-Boost software under edge case conditions on a dedicated devnet. To that end, Mainnet Shadow Fork #2 (MSF#2) for Shanghai testing was launched yesterday, February 22, and sometime next week, Shanghai will be activated on MSF#2, where developers will start to stress test MEV-Boost software under more extreme network conditions.
More MEV-Boost Testing
Mario Vega who is also employed by the Ethereum Foundation and assisting with the testing efforts for the Shanghai upgrade shared an update on new Hive tests for Ethereum clients. As background, Hive is a system for running integration tests or simulations to test client logic and behavior under specific conditions. During last week’s ACDE call, Vega had said he would work on creating Hive tests to assess the behavior of clients in the event MEV-Boost software fails. Vega affirmed that the circuit breaker mechanism, which is the mechanism that ensures validators can fall back on local block production in lieu of MEV-Boost blocks, for the Lighthouse, Teku, and Prysm (CL) clients were working. However, there were issues with this logic when tested on the Nimbus and Lodestar (CL) clients. Full results from Vega’s Hive test can be read here.
Danny Ryan highlighted that the issues could be because technically the circuit breaker logic of falling back on local block production after a certain number of consecutively missed slots is not implemented uniformly across all clients. “We decided before Bellatrix that the circuit breaker logic would be left client to client on the parameters and how they perform that. So, it’s not specified that it is a must and it’s not specified exactly how many slots [before the circuit breaker logic kicks in],” said Ryan on the call. A representative from the Lodestar team affirmed that the Lodestar client does implement circuit breaker logic for MEV-Boost and would investigate why they still failed to pass Vega’s Hive test. A representative from the Nimbus team affirmed that the Nimbus client does not implement the circuit breaker logic as of yet.
Terence Tsao from the Prysm (CL) client team said that it would be useful to also create Hive tests for assessing client logic in block value comparisons. “For Prysm, we’re currently comparing block value that [if] they use MEV-Boost, we’ll compare the block value from your local block and the builder block and then choose the one that’s the highest value. I’m wondering if any other [client] plans on doing that and whether we can actually integrate this type of test case into Hive because I think this is a great test,” said Tsao. A representative of the Teku (CL) client team affirmed that their client also compares local and MEV-Boost builder block values. The Teku client team is also looking at extending this logic to include multipliers that allows validators to choose MEV-Boost builder blocks only if the block value is two times or some multiple higher than a local block’s value. Based on the amount of support for this type of logic in CL clients, Vega agreed to add a new Hive test for assessing the logic in block value comparisons.
Chris Hager from the Flashbots team also gave an update on new releases of MEV-Boost software for validator node operators in lead-up to Shanghai. The primary maintainer of MEV-Boost software is Flashbots. Hager said his team was wrapping up minor updates and pull requests to the MEV-Boost relay repository and a final release for the software would be ready by early next week. He encouraged client developers on the call to take a look at their documentation. Hager said the final releases for MEV-Boost would be tested on both the Sepolia and Goerli testnets before Shanghai activation on mainnet Ethereum.
Planning Cancun & Deneb
After discussions on Shanghai and Capella testing, developers briefly discussed confusion in the versioning for Beacon API specifications. Paul Harris, a blockchain protocol engineer at ConsenSys, posted on the meeting agenda saying he’d like to release a checkpoint for Beacon API specifications for the Capella upgrade but several changes to those specifications for the Deneb upgrade are already bleeding into the documentation. Danny Ryan said that he would reach out to Harris separately and discuss the matter further.
On the topic of planning for Deneb, the next upgrade after Shanghai & Capella, CL client teams have put out a new release for the upgrade, which includes cryptography and naming updates, as well as new test cases. Danny Ryan also highlighted ongoing work on PR 3242 which aims to remove extra code logic for handling empty blob transactions. Ryan encouraged developers on the call to chime into the discussion if the logic around empty blob transactions should be changed. Finally, Beiko reminded developers that the next EIP 4844 Implementers’ Call would be cancelled and reconvene on March 7.