Ethereum All Core Developers Consensus Call #123 Writeup
On November 30, 2023, Ethereum developers gathered over Zoom for All Core Developers Consensus (ACDC) call #123. Chaired by Ethereum Foundation Researcher Danny Ryan, the ACDC calls are a bi-weekly meeting series where developers discuss and coordinate changes to the consensus layer (CL) of Ethereum. This week, developers shared Cancun/Deneb testing updates, which included:
The launch of Devnet #12: Teku, Lodestar, and Lighthouse client software, along with all execution layer (EL) client software, is currently being tested on Devnet #12. The Prysm team expects to have their software ready for testing on the latest Devnet in two to three weeks.
Validator exit issue on Devnet #11: Developers identified an issue with validator exits on Devnet #11 that is being resolved by the Nimbus client team. Devnet #11 will remain up and running until the issue is fixed.
Networking specifications clarification: Developers discussed clarifying specifications for the “BlockByRoot” and “BlobSidecarsByRoot” requests to clearly mention whether CL nodes should respond to these requests in a specific order.
Aside from updates to Cancun/Deneb, developers also discussed a few process items raised by Ethereum Foundation Protocol Support Lead Tim Beiko to improve coordination for Ethereum upgrades.
The Cancun/Deneb upgrade was activated on Devnet #12 on Wednesday, November 30. Mario Vega, on the testing team at the Ethereum Foundation, said that there has been “no major issues so far” identified in the three CL clients that are running on the test network. Barnabas Busa, a DevOps Engineer at the Foundation, said that between Lighthouse, Teku, and Lodestar, there is a total of 225 validators running across three nodes. Given that Devnet #12 is stable, Parithosh Jayanthi, also a DevOps Engineer at the Foundation, asked whether developers should start planning for a Goerli shadow fork to further test Cancun/Deneb. However, given that the most popular CL client, Prysm, has yet to join Devnet #12, developers agreed to hold off on planning a Goerli shadow fork until Prysm client software is ready for testing. Beiko recommended aiming for a Goerli shadow fork some time before the end of the year.
A Nimbus developer by the screen name “Dustin” chimed in with details about the issues his team are resolving on Devnet #11. The issues were first identified when developers were exiting a third of validators from Devnet #11 for use on Devnet #12. Ryan asked Dustin whether the issues that were identified could have been caught with additional testing. Dustin responded saying that in his view the nature of the bugs was due to implementation details outside of the scope of consensus specifications. However, he also acknowledged an “obvious tension” between writing client software in strict accordance with CL specifications for the benefit of test coverage and straying into the grey area of specifications for the advantage of better node performance.
“I mean obviously more testing is always good but more systematically figuring out how to incorporate more client functionality into tests that are run, maybe more automated ways, whether it's Hive or Kurtosis or something else. … If these can be figured out or things can be factored well, so that more of these tasks can be brought in, I think that would be definitely useful,” said Dustin, adding that another challenge CL developers should consider tackling is figuring out to what level of detail CL specifications should specify and standardize node behavior. “The other challenge here is like how standardized - and this is actually not just like a software engineering questions - how standardized should the behaviors be?” asked Dustin, adding that all CL clients pass the tests examining basic behaviors but any behaviors outside of the scope of these tests are ambiguous. “Are these things that are intentionally kind of left ambiguous? What should be really nailed down by a spec versus what should be left deliberately vague?” asked Dustin.
At the very least, developers agreed to add more testing for high volumes of validator exits on future devnets and testnets for Cancun/Deneb. Ryan also acknowledged that there is room for more rigorous and comprehensive test coverage when it comes to CL clients and CL specifications implementation. Vega recommended that Dustin create a HackMD post detailing his concerns and bring up this topic for discussion on the next Cancun/Deneb testing call. Based on the last few issues identified on Cancun/Deneb devnets, Jayanthi added that developers clearly do need tools that can systematically execute a certain number of integration-related behaviors on-chain such as staked ETH withdrawals, validator exits, and more. For this, Jayanthi recommended using a mixture of Hive and Kurtosis test suites to build out such a tool.
Speaking of new testing tools for the Cancun/Deneb upgrade, Jayanthi noted that developers were working on a tool to reliably activate chain reorgs on devnets and testnets. To ensure that the tool is working properly, Jayanthi asked developers to share the details of known bugs that trigger chain reorgs on Ethereum. Jayanthi explained that he will use these bugs to test the tool and ensure that it can reliably identify reorgs as they are happening and when they are resolved.
Networking Specifications Clarification
Developers briefly discussed an open pull request by Ethereum Foundation Researcher Justin Traglia about the order of responses that should be returned by a CL client if they receive a “BlockbyRoot” or “BlobSidecarsByRoot” request. Ryan asked how the functionality is already implemented across different CL client teams. No developers on the call appeared ready to answer this question. Ryan recommended resurfacing this discussion on the Ethereum Research and Development Discord channel with a broader audience of client developers. Ryan acknowledged that the specifications for these two requests were ambiguous and “might be cause of issue and weird edge cases.” “It’s worth clarifying and cleaning up,” affirmed Ryan.
Ryan also mentioned that he will be releasing a new version of CL specifications over the next few days. The latest version notably adds specification for when CL clients can serve blocks and blobs through the “byRoot” RPC request. For more context on the discussion about “byRoot” RP requests for retrieving missing blocks and blobs, refer to prior call notes here. Ryan stressed that the new additions to CL specifications contained within the latest release do not introduce any breaking changes to code that will impact validators already operating on Devnet #11 or #12.
Upgrade Planning Process
Then, developers discussed a variety of process items raised by Beiko. Beiko said that the blog post alerting Goerli testnet users of a forthcoming deprecation either 3 months after the Cancun/Deneb upgrade is activated on Goerli or 1 month after the upgrade is activated on Ethereum mainnet (whichever comes later) will be published on the Ethereum Foundation blog today, Wednesday, November 30. Since the call concluded, the aforementioned blog post has been published and can be read here.
Beiko recommended creating upgrade-specific folders under the Ethereum “pm” repository to organize miscellaneous documents related to a specific upgrade into one folder for posterity. Developers on the call agreed with Beiko’s recommendation.
The second process item Beiko raised was on the creation of a “Meta EIP” document to track the full scope of finalized network upgrades on Ethereum. “There currently is no good place to track the full scope of a network upgrade prior to it being deployed and announced in a blog post,” wrote Beiko in a post outlining his proposal for Meta EIPs. “For Dencun, we have EL EIPs in a hard to find markdown file 7 and CL EIPs as part of the Beacon Chain spec 3. This isn’t great, as both of these are somewhat hard to find, each of them uses a separate ‘format’ and it results in duplication. With ERC and EIPs now separate, I suggest (going back to) using Meta EIPs to track EIPs included in network upgrades.” Developers were in favor of creating these documents. Beiko said that he would draft one for review this week for the Cancun/Deneb upgrade.
Finally, Beiko raised an outstanding question about the utility of labelling proposed code changes, Ethereum Improvement Proposals (EIPs), as “considered for inclusion” (CFI). According to Beiko, CFI is a status that was used by developers historically as a “soft signal” that an EIP author should continue work on a proposal for potential inclusion in a future hard fork. It has only been used for EL-focused code changes and upgrades. “[CFI is] higher than random ideas from random people but prior to accepted in the fork,” said Beiko.
In the past, the label CFI has had little to no effectiveness in signaling which EIPs on the EL will be implemented in an upgrade or when. It has been applied to a wide range of EIPs possessing varying degrees of code readiness and client team support. In the case of the EVM Object Format (EOF) proposal, the label was applied by developers to signal their commitment to implementing EOF in the next immediate upgrade, Cancun/Deneb. However, EOF along with several other proposals that were also CFI’d were rejected from Cancun/Deneb and it remains unclear the status of these CFI’d EIPs for the next upgrade, Prague/Electra.
Beiko said that if the status is not helpful, developers should remove it but if they intend to keep the status, CL developers should also adopt the same label for code changes that they consider for implementation in a CL upgrade. It is unclear to what extent the CL EIP review process already mirrors the EL EIP review process and to what extent they should evolve moving forward identically. Generally, proposed code changes to CL specifications are discussed on ACDC calls while proposed EIPs to EL specifications are discussed on ACDE calls.
Danno Ferrin of Swirlds Labs also proposed the idea to create a placeholder field for all EIPs, CL and EL-related, identifying their status in the review process that includes the CFI status. There were no clear decisions made on this call about the topic of EIP status and the CFI label.