Ethereum All Core Developers Consensus Call #123 Writeup
On November 30, 2023, Ethereum developers gathered over Zoom for All Core Developers Consensus (ACDC) call #123. Chaired by Ethereum Foundation Researcher Danny Ryan, the ACDC calls are a bi-weekly meeting series where developers discuss and coordinate changes to the consensus layer (CL) of Ethereum. This week, developers shared Cancun/Deneb testing updates, which included:
The launch of Devnet #12: Teku, Lodestar, and Lighthouse client software, along with all execution layer (EL) client software, is currently being tested on Devnet #12. The Prysm team expects to have their software ready for testing on the latest Devnet in two to three weeks.
Validator exit issue on Devnet #11: Developers identified an issue with validator exits on Devnet #11 that is being resolved by the Nimbus client team. Devnet #11 will remain up and running until the issue is fixed.
Networking specifications clarification: Developers discussed clarifying specifications for the “BlockByRoot” and “BlobSidecarsByRoot” requests to clearly mention whether CL nodes should respond to these requests in a specific order.
Aside from updates to Cancun/Deneb, developers also discussed a few process items raised by Ethereum Foundation Protocol Support Lead Tim Beiko to improve coordination for Ethereum upgrades.
Devnet #12
The Cancun/Deneb upgrade was activated on Devnet #12 on Wednesday, November 30. Mario Vega, on the testing team at the Ethereum Foundation, said that there has been “no major issues so far” identified in the three CL clients that are running on the test network. Barnabas Busa, a DevOps Engineer at the Foundation, said that between Lighthouse, Teku, and Lodestar, there is a total of 225 validators running across three nodes. Given that Devnet #12 is stable, Parithosh Jayanthi, also a DevOps Engineer at the Foundation, asked whether developers should start planning for a Goerli shadow fork to further test Cancun/Deneb. However, given that the most popular CL client, Prysm, has yet to join Devnet #12, developers agreed to hold off on planning a Goerli shadow fork until Prysm client software is ready for testing. Beiko recommended aiming for a Goerli shadow fork some time before the end of the year.
Devnet #11
A Nimbus developer by the screen name “Dustin” chimed in with details about the issues his team are resolving on Devnet #11. The issues were first identified when developers were exiting a third of validators from Devnet #11 for use on Devnet #12. Ryan asked Dustin whether the issues that were identified could have been caught with additional testing. Dustin responded saying that in his view the nature of the bugs was due to implementation details outside of the scope of consensus specifications. However, he also acknowledged an “obvious tension” between writing client software in strict accordance with CL specifications for the benefit of test coverage and straying into the grey area of specifications for the advantage of better node performance.
“I mean obviously more testing is always good but more systematically figuring out how to incorporate more client functionality into tests that are run, maybe more automated ways, whether it's Hive or Kurtosis or something else. … If these can be figured out or things can be factored well, so that more of these tasks can be brought in, I think that would be definitely useful,” said Dustin, adding that another challenge CL developers should consider tackling is figuring out to what level of detail CL specifications should specify and standardize node behavior. “The other challenge here is like how standardized - and this is actually not just like a software engineering questions - how standardized should the behaviors be?” asked Dustin, adding that all CL clients pass the tests examining basic behaviors but any behaviors outside of the scope of these tests are ambiguous. “Are these things that are intentionally kind of left ambiguous? What should be really nailed down by a spec versus what should be left deliberately vague?” asked Dustin.
At the very least, developers agreed to add more testing for high volumes of validator exits on future devnets and testnets for Cancun/Deneb. Ryan also acknowledged that there is room for more rigorous and comprehensive test coverage when it comes to CL clients and CL specifications implementation. Vega recommended that Dustin create a HackMD post detailing his concerns and bring up this topic for discussion on the next Cancun/Deneb testing call. Based on the last few issues identified on Cancun/Deneb devnets, Jayanthi added that developers clearly do need tools that can systematically execute a certain number of integration-related behaviors on-chain such as staked ETH withdrawals, validator exits, and more. For this, Jayanthi recommended using a mixture of Hive and Kurtosis test suites to build out such a tool.
Speaking of new testing tools for the Cancun/Deneb upgrade, Jayanthi noted that developers were working on a tool to reliably activate chain reorgs on devnets and testnets. To ensure that the tool is working properly, Jayanthi asked developers to share the details of known bugs that trigger chain reorgs on Ethereum. Jayanthi explained that he will use these bugs to test the tool and ensure that it can reliably identify reorgs as they are happening and when they are resolved.
Networking Specifications Clarification
Developers briefly discussed an open pull request by Ethereum Foundation Researcher Justin Traglia about the order of responses that should be returned by a CL client if they receive a “BlockbyRoot” or “BlobSidecarsByRoot” request. Ryan asked how the functionality is already implemented across different CL client teams. No developers on the call appeared ready to answer this question. Ryan recommended resurfacing this discussion on the Ethereum Research and Development Discord channel with a broader audience of client developers. Ryan acknowledged that the specifications for these two requests were ambiguous and “might be cause of issue and weird edge cases.” “It’s worth clarifying and cleaning up,” affirmed Ryan.
Ryan also mentioned that he will be releasing a new version of CL specifications over the next few days. The latest version notably adds specification for when CL clients can serve blocks and blobs through the “byRoot” RPC request. For more context on the discussion about “byRoot” RP requests for retrieving missing blocks and blobs, refer to prior call notes here. Ryan stressed that the new additions to CL specifications contained within the latest release do not introduce any breaking changes to code that will impact validators already operating on Devnet #11 or #12.
Upgrade Planning Process
Then, developers discussed a variety of process items raised by Beiko. Beiko said that the blog post alerting Goerli testnet users of a forthcoming deprecation either 3 months after the Cancun/Deneb upgrade is activated on Goerli or 1 month after the upgrade is activated on Ethereum mainnet (whichever comes later) will be published on the Ethereum Foundation blog today, Wednesday, November 30. Since the call concluded, the aforementioned blog post has been published and can be read here.
Beiko recommended creating upgrade-specific folders under the Ethereum “pm” repository to organize miscellaneous documents related to a specific upgrade into one folder for posterity. Developers on the call agreed with Beiko’s recommendation.
The second process item Beiko raised was on the creation of a “Meta EIP” document to track the full scope of finalized network upgrades on Ethereum. “There currently is no good place to track the full scope of a network upgrade prior to it being deployed and announced in a blog post,” wrote Beiko in a post outlining his proposal for Meta EIPs. “For Dencun, we have EL EIPs in a hard to find markdown file 7 and CL EIPs as part of the Beacon Chain spec 3. This isn’t great, as both of these are somewhat hard to find, each of them uses a separate ‘format’ and it results in duplication. With ERC and EIPs now separate, I suggest (going back to) using Meta EIPs to track EIPs included in network upgrades.” Developers were in favor of creating these documents. Beiko said that he would draft one for review this week for the Cancun/Deneb upgrade.
Finally, Beiko raised an outstanding question about the utility of labelling proposed code changes, Ethereum Improvement Proposals (EIPs), as “considered for inclusion” (CFI). According to Beiko, CFI is a status that was used by developers historically as a “soft signal” that an EIP author should continue work on a proposal for potential inclusion in a future hard fork. It has only been used for EL-focused code changes and upgrades. “[CFI is] higher than random ideas from random people but prior to accepted in the fork,” said Beiko.
In the past, the label CFI has had little to no effectiveness in signaling which EIPs on the EL will be implemented in an upgrade or when. It has been applied to a wide range of EIPs possessing varying degrees of code readiness and client team support. In the case of the EVM Object Format (EOF) proposal, the label was applied by developers to signal their commitment to implementing EOF in the next immediate upgrade, Cancun/Deneb. However, EOF along with several other proposals that were also CFI’d were rejected from Cancun/Deneb and it remains unclear the status of these CFI’d EIPs for the next upgrade, Prague/Electra.
Beiko said that if the status is not helpful, developers should remove it but if they intend to keep the status, CL developers should also adopt the same label for code changes that they consider for implementation in a CL upgrade. It is unclear to what extent the CL EIP review process already mirrors the EL EIP review process and to what extent they should evolve moving forward identically. Generally, proposed code changes to CL specifications are discussed on ACDC calls while proposed EIPs to EL specifications are discussed on ACDE calls.
Danno Ferrin of Swirlds Labs also proposed the idea to create a placeholder field for all EIPs, CL and EL-related, identifying their status in the review process that includes the CFI status. There were no clear decisions made on this call about the topic of EIP status and the CFI label.
Legal Disclosure:
This document, and the information contained herein, has been provided to you by Galaxy Digital Holdings LP and its affiliates (“Galaxy Digital”) solely for informational purposes. This document may not be reproduced or redistributed in whole or in part, in any format, without the express written approval of Galaxy Digital. Neither the information, nor any opinion contained in this document, constitutes an offer to buy or sell, or a solicitation of an offer to buy or sell, any advisory services, securities, futures, options or other financial instruments or to participate in any advisory services or trading strategy. Nothing contained in this document constitutes investment, legal or tax advice or is an endorsementof any of the digital assets or companies mentioned herein. You should make your own investigations and evaluations of the information herein. Any decisions based on information contained in this document are the sole responsibility of the reader. Certain statements in this document reflect Galaxy Digital’s views, estimates, opinions or predictions (which may be based on proprietary models and assumptions, including, in particular, Galaxy Digital’s views on the current and future market for certain digital assets), and there is no guarantee that these views, estimates, opinions or predictions are currently accurate or that they will be ultimately realized. To the extent these assumptions or models are not correct or circumstances change, the actual performance may vary substantially from, and be less than, the estimates included herein. None of Galaxy Digital nor any of its affiliates, shareholders, partners, members, directors, officers, management, employees or representatives makes any representation or warranty, express or implied, as to the accuracy or completeness of any of the information or any other information (whether communicated in written or oral form) transmitted or made available to you. Each of the aforementioned parties expressly disclaims any and all liability relating to or resulting from the use of this information. Certain information contained herein (including financial information) has been obtained from published and non-published sources. Such information has not been independently verified by Galaxy Digital and, Galaxy Digital, does not assume responsibility for the accuracy of such information. Affiliates of Galaxy Digital may have owned or may own investments in some of the digital assets and protocols discussed in this document. Except where otherwise indicated, the information in this document is based on matters as they exist as of the date of preparation and not as of any future date, and will not be updated or otherwise revised to reflect information that subsequently becomes available, or circumstances existing or changes occurring after the date hereof. This document provides links to other Websites that we think might be of interest to you. Please note that when you click on one of these links, you may be moving to a provider’s website that is not associated with Galaxy Digital. These linked sites and their providers are not controlled by us, and we are not responsible for the contents or the proper operation of any linked site. The inclusion of any link does not imply our endorsement or our adoption of the statements therein. We encourage you to read the terms of use and privacy statements of these linked sites as their policies may differ from ours. The foregoing does not constitute a “research report” as defined by FINRA Rule 2241 or a “debt research report” as defined by FINRA Rule 2242 and was not prepared by Galaxy Digital Partners LLC. For all inquiries, please email [email protected]. ©Copyright Galaxy Digital Holdings LP 2023. All rights reserved.