Ethereum All Core Developers Consensus Call #127 Writeup
On February 8, 2024, Ethereum developers gathered over Zoom for All Core Developers Consensus (ACDC) call #127. Chaired by Ethereum Foundation Researcher Danny Ryan, the ACDC calls are a bi-weekly meeting series where developers discuss and coordinate changes to the consensus layer (CL) of Ethereum. This week, developers scheduled mainnet activation of the Dencun upgrade for March 13, 2024. They also discussed learnings from an incident on mainnet resulting in 9 missed slots. The incident occurred on Tuesday, February 6 and has reinvigorated conversations about removing validator dependencies on trusted relays through an upgrade known as enshrined proposer builder separation (ePBS). Separately, developers also agreed to reconsider code changes like maxEB and inclusion lists in the Electra upgrade.
Dencun Testing
Ethereum Foundation Developer Operations Engineer Parithosh Jayanthi said the activation of the Dencun upgrade on the Holesky testnet on Wednesday, February 7 went smoothly. “There’s been nothing we’ve noticed at least on Holesky,” said Jayanthi. “We’ve passed the blob expiry window for Goerli so we ran a bunch of nodes and are doing a mixture of genesis sync, as well as checkpoint sync [tests].”
Prysm developer Terence Tsao mentioned that the fork transition block was missed during the Holesky upgrade. While “not a big deal,” Tsao said the incident led to an 11 second block delay for his node. He recommended that client teams double check to see if their implementations had somehow caused this delay during the upgrade.
Lighthouse developer “Sean” said that the Lighthouse team has implemented logic in their client related to node recovery in a scenario where the chain has not finalized blob transactions. In such a scenario, a node could recover by relying on a checkpoint sync from an unfinalized checkpoint. However, implementing this logic, Sean said, was more involved than his team had anticipated and encouraged CL client teams to reach out if they come across similar difficulties.
Nethermind developer Marcin Sobczak said that his team is continuing to investigate a potential bug in their client mentioned during last Thursday ACD call. Sobczak his team has started spamming the Goerli network with blob transactions and so far, have found no issues. He said that the tests on Goerli should wrap up in a few hours.
Dencun Mainnet Activation
Ethereum Foundation Protocol Support Lead Tim Beiko shared the following mainnet activation dates and times for the Dencun upgrade:
Caption: Dencun Fork Times
Source: YouTube, @EthereumProtocol
Beiko mentioned that he had reached out to the teams behind the top 10 Ethereum rollups on L2Beat.com to evaluate their readiness for Dencun. “All the teams are pretty much in testing phases at various stages. I think teams will be ready on the L2 side to use 4844 on mainnet around early to mid-March,” said Beiko. “I don’t think we should block anything based on where L2 teams are at.”
Tsao said from the Prysm side his team would appreciate two weeks to prepare a final mainnet release for the Dencun upgrade. Sean representing the Lighthouse client team agreed with this timeline. Other CL client teams such as Teku and Lodestar said that a runway of two weeks to prepare final mainnet client releases would also work for them.
Based on this sentiment, Beiko initially suggested a mainnet upgrade date for either Thursday, March 8 or Tuesday, March 12. However, developers pushed back on the tight turnaround time between final client releases and the proposed hard fork date in the Zoom meeting chat. Developers agreed then to leave three weeks between the release of final client software versions and the upgrade date. Developers will prepare final client releases by the next ACDC call on Thursday, February 22. Beiko will then publish a blog post formally announcing the mainnet activation of Dencun the following day on Friday, February 23. Mainnet node operators will have roughly three weeks from then to upgrade their software. The Dencun hard fork will occur on March 13, 2024.
Mainnet Missed Blocks Incident
On Tuesday, February 6, the Bloxroute Max Profit relay delivered 9 blocks to validators that failed to be added to the Ethereum blockchain. This was due to a bug in the relay that did not work to correctly demote the block builder responsible for submitting the faulty blocks. Bloxroute has since patched their relay and reimbursed validators for lost block rewards.
The incident has reinvigorated conversations about how to best mitigate validators’ reliance on relays for blocks containing MEV. Prysm developer “Potuz” said that an immediate solution to the problem could be add a new heuristic to validators’ circuit breaker functionality that will check for invalid blocks and fall back to local block production if a relay consistently sends two or three invalid blocks. Ethereum Foundation researchers Danny Ryan and Dankrad Feist were concerned that the addition of such a heuristic could be easily gamed by sophisticated MEV actors that could purposely trigger the circuit breaker to temporarily earn all the MEV in blocks for themselves. Sean from Lighthouse noted that the precise implementation of the heuristic may differ from client to client and thereby make it harder for malicious actors to game but to this, Ryan suggested that there will still likely be a norm for the heuristic that most client implementations end up following.
Ryan then suggested a “gossip channel” where validators could listen and drop in information about block signatures by builders to notify the entire network of a faulty builder more quickly and without reliance on a relay. However, Potuz pushed back on this idea saying that instead of a gossip channel, developers should dedicate resources to delivering ePBS, which is a code change that will completely remove the need for trusted relays and support direct interaction between block builders and validators. Before diving deeper into the discussion of whether ePBS should be prioritized ahead of a short-term solution like a dedicated gossip channel for block verifications, Danny Ryan recommended first evaluating a few other code changes that have been proposed for Electra.
Electra Portmanteau
As a brief aside, Ryan asked developers whether they supported a combined upgrade name of “Pectra” for Prague/Electra. Developers on the call did not appear to have a strong opinion about the portmanteau. Ryan moved on with the discussion about what code changes should be prioritized for Electra.
Electra EIP 7549
During ACDC #126, developers agreed to include EIP 7549 in the Electra upgrade. EIP 7549 is a code change impacting only the CL of Ethereum that makes the aggregation of validator attestations more efficient. Developers had agreed to go with the simplest design of EIP 7549 for inclusion in Electra. However, based on more conversation among developers on GitHub about the code change, there is appetite to increase the complexity of the code change such that it can have wider impacts on the network. Teku developer Mikhail Kalinin explained, “This is one more step on top of what was originally proposed by EIP 7549. As Danny said, it allows us to pack attestations tightly on chain. What’s nice about it is that considering the current validator set size, we can increase the block space in terms of attestations up to four times. Currently, we can keep attestations for two slots if they’re ideally aggregated. This change would allow us to do this for eight slots instead of two without increasing the block size in bytes.”
Danny Ryan agreed this was a good change to the EIP but recommended careful consideration of the tradeoffs that it might present to the protocol. “There’s a tradeoff because you increase the capacity but you decrease the potential diversity of that capacity, which it seems like is a reasonable trade off but one to consider the risks,” said Ryan. Kalinin agreed and said that he would do some further analysis and offer new specifications changes for the code change under the same EIP number.
Electra SSZ
As discussed on ACDC #126, developers are considering the inclusion of five EIPs related to SSZ formatting. Sean from Lighthouse said that he would need to evaluate the code changes in more detail but initially from his perspective the code changes are “a good thing to have”. Another developer reportedly wrote in the Zoom chat that they would like to see the SSZ formatting changes bundled as one large change to the protocol as opposed to implemented piecemeal. Ryan recommended that client teams do more due diligence on understanding the SSZ changes proposed by Nimbus developer Etan Kissling and re-discuss the topic on the next ACDC call.
Electra ePBS
Prysm developer “Potuz” made the case for ePBS in the next major Ethereum upgrade. “I think the problem that we've seen on mainnet is not like a live problem or a minor problem. It's not the issue that nine blocks went missing or that validators got refunded or not, but really the problem is that we need to trust the relay for this. We don't even know what the check was. We don't even know what the fix was. This is closed source software and this development is happening in a black box,” said Potuz, adding, “These are five players that are relaying all of our blocks or 90% of our blocks and 10 players maximum that are building these blocks. I think what we need to do is to decide that this is a priority, that we should not have in Ethereum a trusted player making decisions with closed source software, being the one that is responsible for paying or not, for refunding validators, or even deciding which transactions are censored or not. Once we make this a priority, then we can discuss whether or not there are viable designs for ePBS.”
Potuz urged developers to consider scoping out the Electra fork for 2025, instead of 2024, and using the “Interop event”, which is presumably an Ethereum core developer meetup happening in May 2024, to flesh out the finalized design for ePBS. Tsao said that ePBS is only “one solution to the problem.” He highlighted that the Builder API is an important piece of software that is not “incentive aligned” with the Ethereum protocol and needs updating. Sean from Lighthouse encouraged developers to consider solutions outside of ePBS, which would enshrine builders in the Ethereum protocol, to address the issue of trusted relays. Ethereum Foundation Researcher Dankrad Feist said that ePBS in his view was an extremely large change that should not be rushed. One of his main concerns about removing relays is opening the potential for validators to steal MEV from builders. At present, relays are trusted by not only validators but also builders. Builders trust relays not to front run their transactions and they may not have the same degree of trust in validators to do the same without appropriate guardrails in the protocol.
Potuz pushed back on the sentiment that ePBS is a large code change, arguing that the current designs for ePBS are not as large of an engineering change as inclusion lists or maxEB. Danny Ryan chimed in saying, “Whenever I see this [topic] opened up, there’s a lot of questions as to what are we even optimizing? What is the right end goal of this? It seems like there’s a lot of varying opinions on that. The decision to include this first becomes the decision to figure out what is the design.” To this end, Ryan recommended a break out session to discuss the design of ePBS and see if developers can come to an agreement about its complexity and dependencies on other code changes.
The breakout room for ePBS discussions will be held on Tuesday, February 13 at 9:00am (ET)/14:00 (UTC).
Electra Inclusion Lists
Then, developers discussed EIP 7547, inclusion lists. During ACDC #126, developers were not keen on prioritizing the proposal for Electra. However, Ryan said that since the last ACDC call there was desire from developers to reopen discussion for the proposal again. Ethereum Foundation Researcher Mike Neuder said that a simplified design for inclusion lists has been created. He also pointed to a new document detailing what the potential specifications for inclusions lists could looks like in clients. Ryan said that he had reviewed the specifications and agreed that it was not as complex as he had initially thought. Sean agreed inclusion lists could be feasible to include in Electra but that maxEB was in his view a more important code change. “I think right now I would lean toward just trying to include both maxEB and inclusions lists and not do the full ePBS,” said Sean. He added that data availability sampling (DAS) could be worked on in parallel with these other changes.
Like ePBS, developers agreed to discuss inclusion lists in more detail in a separate breakout call. The breakout meeting for inclusion lists will be held on Friday, February 16 at 9:00AM (ET)/14:00 (UTC).
Electra peerDAS and maxEB
The final two proposals that developers discussed for potential inclusion in Electra was peerDAS and maxEB. On the topic of peerDAS, Danny Ryan said that a number of client teams have expressed their intention to work on this code change in parallel to their work on the Electra upgrade. Because it does not strictly require a hard fork, peerDAS can be shipped independently of the Pectra upgrade. Without a hard fork upgrade, peerDAS would not result in any increases to the blob gas limit, that is the maximum number of blob transactions per block. However, Ryan mentioned that a small hard fork only changing the blob gas limit could be scheduled once peerDAS is ready for implementation. Therefore, Ryan recommended that developers work on peerDAS in parallel to Electra. However, Sean said that he could see peerDAS ready for implementation faster than the Electra upgrade and would like to see it implemented at the latest by the Electra fork. To this, Ryan recommended then a tentative inclusion of peerDAS in the Electra upgrade and removing it if the code change ends up being a blocker to the rest of the items included in the Electra fork. Developers were not able to come to a consensus about this proposal. Ryan recommended revisiting this conversation once the ePBS and inclusion breakout calls have happened. Ryan recommended the same for discussions on the inclusion of maxEB in Electra. Developers agreed to revisit these discussions on the next ACDC call in two weeks.
Legal Disclosure:
This document, and the information contained herein, has been provided to you by Galaxy Digital Holdings LP and its affiliates (“Galaxy Digital”) solely for informational purposes. This document may not be reproduced or redistributed in whole or in part, in any format, without the express written approval of Galaxy Digital. Neither the information, nor any opinion contained in this document, constitutes an offer to buy or sell, or a solicitation of an offer to buy or sell, any advisory services, securities, futures, options or other financial instruments or to participate in any advisory services or trading strategy. Nothing contained in this document constitutes investment, legal or tax advice or is an endorsementof any of the digital assets or companies mentioned herein. You should make your own investigations and evaluations of the information herein. Any decisions based on information contained in this document are the sole responsibility of the reader. Certain statements in this document reflect Galaxy Digital’s views, estimates, opinions or predictions (which may be based on proprietary models and assumptions, including, in particular, Galaxy Digital’s views on the current and future market for certain digital assets), and there is no guarantee that these views, estimates, opinions or predictions are currently accurate or that they will be ultimately realized. To the extent these assumptions or models are not correct or circumstances change, the actual performance may vary substantially from, and be less than, the estimates included herein. None of Galaxy Digital nor any of its affiliates, shareholders, partners, members, directors, officers, management, employees or representatives makes any representation or warranty, express or implied, as to the accuracy or completeness of any of the information or any other information (whether communicated in written or oral form) transmitted or made available to you. Each of the aforementioned parties expressly disclaims any and all liability relating to or resulting from the use of this information. Certain information contained herein (including financial information) has been obtained from published and non-published sources. Such information has not been independently verified by Galaxy Digital and, Galaxy Digital, does not assume responsibility for the accuracy of such information. Affiliates of Galaxy Digital may have owned or may own investments in some of the digital assets and protocols discussed in this document. Except where otherwise indicated, the information in this document is based on matters as they exist as of the date of preparation and not as of any future date, and will not be updated or otherwise revised to reflect information that subsequently becomes available, or circumstances existing or changes occurring after the date hereof. This document provides links to other Websites that we think might be of interest to you. Please note that when you click on one of these links, you may be moving to a provider’s website that is not associated with Galaxy Digital. These linked sites and their providers are not controlled by us, and we are not responsible for the contents or the proper operation of any linked site. The inclusion of any link does not imply our endorsement or our adoption of the statements therein. We encourage you to read the terms of use and privacy statements of these linked sites as their policies may differ from ours. The foregoing does not constitute a “research report” as defined by FINRA Rule 2241 or a “debt research report” as defined by FINRA Rule 2242 and was not prepared by Galaxy Digital Partners LLC. For all inquiries, please email [email protected]. ©Copyright Galaxy Digital Holdings LP 2024. All rights reserved.