Ethereum All Core Developers Consensus Call #119 Writeup
On October 5, 2023, Ethereum developers gathered over Zoom for All Core Developers Consensus (ACDC) call #119. Chaired by Ethereum Foundation Researcher Danny Ryan, the ACDC calls are a bi-weekly meeting series where developers discuss and coordinate changes to the consensus layer (CL) of Ethereum. This week, the ACDC call was primarily focused on Cancun/Deneb (Dencun) testing. Developers discussed the status and health of the latest developer-focused testnet, Devnet-9, and timing for the launch of Devnet-10.
Devnet-9
Devnet-9 launched on Friday, September 29. The network participation rate, which is a measure of the percentage of validators actively and correctly fulfilling their duties, is trending at 90%. This is a positive indicator of the health of the testnet. Barnabas Busa, a DevOps Engineer for the Ethereum Foundation, noted that a few bugs have been discovered in client software since the launch of Devnet-9, specifically in execution layer (EL) clients Reth, Besu, and Nethermind.
Aside from client software, developers are also testing MEV-related tooling on Devnet-9. Busa said that his team recently flooded the network with transactions containing MEV for builders and relays on Devnet-9 to package into blocks. One of the relays being tested on Devnet-9 is operated by MEV research and development company Flashbots. The activity of the Flashbots relay on Devnet-9 can be tracked on this website here. Busa said roughly 20% of validators on Devnet-9 are running MEV-Boost software.
Geth developer Marius van der Wijden said that he is working on updating the EL block generator on Devnet-9 to propagate “bad” blocks and blobs and thereby test the resiliency of validators. He also noted that the Ethereum JS EL client appears to be having some issues on Devnet-9. Representatives from Besu and Nethermind shared additional color on the call about the bugs discovered in their software on Devnet-9.
Devnet-10
After discussing activity on Devnet-9, developers discussed timing for the launch of the next and possibly final devnet, Devnet-10. The purpose of Devnet-10 is primarily to test the new maximum epoch churn limit defined in EIP 7514. Parithosh Jayanthi, another DevOps Engineer at the Ethereum Foundation, said that developers could try and launch Devnet-10 as early as next week if all client teams have fixed outstanding bugs and issues. “I would like to see all clients being able to participate and no more bugs on Devnet-9 before we proceed with the larger test. We should have a very stable, like 100% participation, before we move on,” added Busa.
Given that most of the bugs discovered on Devnet-9 are isolated to EL client software, Danny Ryan suggested that Devnet-10 could be launched even without some EL clients being ready since the primary purpose of the developer-focused testnet would be to test CL client logic, that is the validator churn.
Mario Vega, a developer on the Ethereum Foundation testing team, said that he is working on expanding the coverage of test cases for clients, especially for edge case scenarios where there is variability in the timing of blobs. Terence Tsao, a developer for the Prysm CL client, added that his team is working on new features to mitigate an issue related to the duplication of blobs by a malicious actor. Tsao asked whether testing teams and other client teams on the client were aware of this issue and working on test vectors for it. Ryan said that some of Vega’s work in updating the Hive test suite for Dencun was related to the issue mentioned by Terence. However, Ryan encouraged Tsao and other client teams on the call to do their own testing to address this issue as well. Enrico Del Fante, a developer for the Teku CL client, said that his team did recently test for the scenario of blob duplication and found issues in their software. These issues have since been resolved in the Teku client, but Fante encouraged other client teams to also devote dedicated testing resources for this scenario if they haven’t already.
A representative for the Lighthouse CL client with the screen name “Sean” added that his team was working on an implementation of the Lighthouse client that would intentionally propose invalid blocks, which may be useful for running on a future testnet. “We’re working on a similar tool [as Teku] that we can maybe use as a node that is proposing [bad blocks]. So, we could deploy [it] to a decent amount of the validator set on like a devnet or attack net. That could be pretty useful to get to a lot of scenarios pretty quick,” said Sean. Vega encouraged client teams on the call to drop in their suggestions and comments about testing blobs to this shared document here.
Dencun Testing Timeline Concerns
Going back to the discussion on timing for the launch of Devnet-10, Chair of the ACDE calls Tim Beiko restated the decision made on last Thursday’s ACD call to aim for a public testnet launch of the Dencun upgrade before the start of Ethereum developer conference Devconnect. Devconnect is scheduled to occur in Istanbul, Turkey this year in mid-November. During ACDE #171, developers agreed to aim for the launch of Dencun on the Goerli test network, a public Ethereum testnet, before Devconnect.
“So [October] 19th is really when we need to have [Devnet-10] done by if we want this to happen before Devconnect. Because if we take time on the next All Core Devs then [the Goerli testnet upgrade] could happen sometime in late October, the two weeks or so after [the next ACDE call]. And then, [launching Devnet-10 around] the CL call after that starts pushing [a potential Goerli upgrade] into like early November and we probably want to get this done before people start heading out for Devconnect,” Beiko said.
Regarding aspects of the Dencun upgrade that should be tested in final on Devnet-10, Andrew Ashikhmin, a developer for the Erigon EL client, asked about the status of the KZG ceremony for EIP 4844. For more information about what a KZG ceremony is, read this Galaxy Research report. Carl Beekhuizen, a researcher for the Ethereum Foundation, is in the process of creating the final output from all contributions to the EIP 4844 KZG ceremony. Busa said that the final output should be ready for testing in Devnet-10. A pseudonymous Geth developer by the screen name of “Lightclient” briefly mentioned in the chat room of this week’s Zoom call that his colleague “Peter” was also working on a custom transaction fetcher for blobs.
Because clients are still discovering bugs on devnets, and there remains important areas of testing for blobs that have not yet been run, Tsao said that in his opinion, it is too early for developers to be talking about upgrading public Ethereum testnets. “What I would want to see in a devnet before I can confidently fork on [public] testnets is that we test more relay infrastructure. Today, we have a relay that is set up … but I would like to bump up that number and test that slightly longer,” said Tsao, adding, “Another thing I would like to see play out more on the devnet is just the blob scenario testing that Mario [Vega] was sharing. If we can run more tests, we’ll have more confidence.” To this, Ryan suggested that developers re-evaluate the timeline for Devnet-10 over the next two weeks as client teams address bugs and the Ethereum Foundation testing team expands the Dencun test suite.
Enrico Del Fante echoed Tsao’s sentiments and added that testing still has not been done for client interoperability using the new block V3 API. Both Teku and Lighthouse have yet to implement the new API. Sean from the Lighthouse team mentioned that they would need a few more weeks of testing for the API before finalizing the new code for their client. “I think we still need another week to get [the API] in and then after that, we’d like to have a decent amount of time before we actually cut a release to do regression testing on that because it’s just like a massive set of changes,” said Sean.
Regarding test coverage for interoperability between clients using the block V3 API, Danny Ryan asked developers whether they should consider launching a dedicated short-term devnet once all clients have merged the API. “At least in the past, we haven’t really tested that on devnets. But if there was an issue, it usually spills out when we fork the first [public] testnet because there’s also like a million different combinations. You can run everything and there’s only so much you can test,” said Jayanthi.
Vega said that the API could be tested through the Hive test suite. He mentioned that he could start work on this immediately for Devnet-9. Sean from the Lighthouse team said that he would prioritize implementing the block V3 API. Ryan recommended revisiting the prioritization of block V3 API and client interoperability testing with it in a couple of weeks. A developer on the call with the screen name “Z” said that the area of testing for Dencun lacking the most attention was builder infrastructure. To this, Alex Stokes, a researcher for the Ethereum Foundation, highlighted there would be a MEV community call organized by Flashbots happening later in the day on October 5. He encouraged participation from client teams on that call to bolster collaboration on testing efforts for MEV-related tooling and software in Dencun.
Legal Disclosure:
This document, and the information contained herein, has been provided to you by Galaxy Digital Holdings LP and its affiliates (“Galaxy Digital”) solely for informational purposes. This document may not be reproduced or redistributed in whole or in part, in any format, without the express written approval of Galaxy Digital. Neither the information, nor any opinion contained in this document, constitutes an offer to buy or sell, or a solicitation of an offer to buy or sell, any advisory services, securities, futures, options or other financial instruments or to participate in any advisory services or trading strategy. Nothing contained in this document constitutes investment, legal or tax advice or is an endorsementof any of the digital assets or companies mentioned herein. You should make your own investigations and evaluations of the information herein. Any decisions based on information contained in this document are the sole responsibility of the reader. Certain statements in this document reflect Galaxy Digital’s views, estimates, opinions or predictions (which may be based on proprietary models and assumptions, including, in particular, Galaxy Digital’s views on the current and future market for certain digital assets), and there is no guarantee that these views, estimates, opinions or predictions are currently accurate or that they will be ultimately realized. To the extent these assumptions or models are not correct or circumstances change, the actual performance may vary substantially from, and be less than, the estimates included herein. None of Galaxy Digital nor any of its affiliates, shareholders, partners, members, directors, officers, management, employees or representatives makes any representation or warranty, express or implied, as to the accuracy or completeness of any of the information or any other information (whether communicated in written or oral form) transmitted or made available to you. Each of the aforementioned parties expressly disclaims any and all liability relating to or resulting from the use of this information. Certain information contained herein (including financial information) has been obtained from published and non-published sources. Such information has not been independently verified by Galaxy Digital and, Galaxy Digital, does not assume responsibility for the accuracy of such information. Affiliates of Galaxy Digital may have owned or may own investments in some of the digital assets and protocols discussed in this document. Except where otherwise indicated, the information in this document is based on matters as they exist as of the date of preparation and not as of any future date, and will not be updated or otherwise revised to reflect information that subsequently becomes available, or circumstances existing or changes occurring after the date hereof. This document provides links to other Websites that we think might be of interest to you. Please note that when you click on one of these links, you may be moving to a provider’s website that is not associated with Galaxy Digital. These linked sites and their providers are not controlled by us, and we are not responsible for the contents or the proper operation of any linked site. The inclusion of any link does not imply our endorsement or our adoption of the statements therein. We encourage you to read the terms of use and privacy statements of these linked sites as their policies may differ from ours. The foregoing does not constitute a “research report” as defined by FINRA Rule 2241 or a “debt research report” as defined by FINRA Rule 2242 and was not prepared by Galaxy Digital Partners LLC. For all inquiries, please email [email protected]. ©Copyright Galaxy Digital Holdings LP 2023. All rights reserved.