Ethereum Consensus Layer Call #93
On Thursday, August 11, Ethereum core developers concluded their 93rd fortnightly Consensus Layer (CL) call. Chaired by Ethereum Foundation researcher Danny Ryan, the CL calls are one of two recurring meeting series where Ethereum developers discuss and coordinate upgrades to the protocol layer of Ethereum. Over the last several months, these calls and for that matter, the All Core Developer (ACD) calls, have focused primarily on preparations for Ethereum’s Merge upgrade. For more information about what the Merge upgrade is, read this Galaxy Digital Research report.
On the 93rd CL call, developers picked a tentative date for mainnet Merge activation. They agreed on the call to aim for releasing final software, also called client versions, for the upgrade by August 23, 2022. Thereafter, they discussed scheduling the first of two hard forks, the Bellatrix hard fork, to activate at epoch 144896 on the Beacon Chain, which is Ethereum’s proof-of-stake blockchain. Epoch 144896 is expected to hit around September 6, 2022. Finally, developers also agreed that 10 days would be a sufficient window for the second hard fork, called Paris, to activate on Ethereum mainnet at a total terminal difficulty value (TTD) of 58750000000000000000000. While TTD is an extremely hard value to estimate, it is currently expected to hit around September 16, 2022. For more background on what TTD is, read this Galaxy Digital Research report.
Other discussion items raised during the call included:
Goerli Merge post-mortem
Goerli Merge Post-Mortem
The Goerli testnet was the third and final major Ethereum testnet to undergo the Merge upgrade. Developers had discussed during last Thursday’s developer call that if all went well with the Goerli Merge activation, they would feel confident about picking a tentative date for mainnet Merge activation this week. The Merge activated on Goerli roughly 12 hours in advance of the call on August 11th. Ethereum Foundation’s Parithosh Jayanthi gave a recap of what he saw on-chain. “One of the things that was peculiar about the Merge this time is that we had two terminal blocks,” said Jayanthi on the call. “We were looking at about 90% participation before the Merge and then we dropped to roughly 70% and for a couple of epochs we were below the required 66% so we didn’t finalize [the chain] for about 4 or 5 epochs.”
The activation of the Goerli Merge was less than ideal. Some nodes on the network were confused as to which block was the terminal block, that is the block created after the TTD threshold is met. In addition, the participation rate of validators dropped significantly through the upgrade causing chain finalization to be delayed for about 25 to 30 minutes. Part of the issue around validator participation rate was due to wrong node configurations. Jayanthi said that the Nimbus (CL) and Lodestar (CL) client team running nodes had not upgraded their software to the latest client release versions. In addition, there were a handful of other client-related issues impacted other nodes such as Nethermind (EL) and Erigon (EL) nodes. Further investigation is still needed on the root causes of these issues with the Nethermind and Erigon clients. For a more detailed explanation of the issues, watch the livestream of the developer call here.
On the positive side, Paul Hauner from the Lighthouse (CL) team noted that regarding his investigation of Ethereum’s fork choice rule under a PoS consensus protocol, validators had no issues sorting themselves out amid a low turnout and continued to correctly progress and build atop a canonical chain. There were some concerns around peer scoring, which is the mechanism that helps nodes judge other nodes based on their health and quality and connect to the ones with a high peer score. But overall, developers agreed the Goerli Merge was a success given that the chain did end up reaching finalization, which is a technical term to describe the completion of 2 epochs in a row, with at minimum 66% participation from validators in each epoch. It is likely that minor issues like the ones seen during the Goerli Merge activation will be seen on mainnet Ethereum. However, these issues that were identified did not prevent the chain from finalizing and as such are not expected to cause major disruptions when activated on Ethereum in September.
Following the post-mortem on Goerli, developers discussed their expectations around timing for mainnet Merge activation. As mentioned at the top of this writeup, developers ultimately agreed on a September 6th date for activating the Bellatrix hard fork and a September 16th date for activating the Paris hard fork. However, it is important to note that not all client teams felt comfortable with this decision. Andrew Ashikhmin from the Erigon (EL) client team said that he would appreciate more time to work on final software releases. To this, Chair of the CL call Danny Ryan said that EL client teams could issue a second release after the Bellatrix hard fork. This would give EL client teams at least four weeks, instead of two to prepare final releases. They would still be required to put out a feature-complete release as suggested by August 23. However, they could then release a second version in time for the Paris upgrade that is “strongly suggested” to users running the former client version. Ashikmin did not object to this strategy and other Ethereum client teams were in favor of preparing their releases for mainnet Merge by August 23.
Developers will reconfirm the values for mainnet Merge during next Thursday’s ACD call. In preparation for final releases, developers also briefly discussed the Hive test suite for Ethereum clients. These are custom-made testing environments for double-checking the behavior of Ethereum clients under specific edge case scenarios. Developers on the call noted that there were aspects of the Hive test suite that needed updating for clients such as Nethermind to use. In addition, developers briefly discussed setting a placeholder value for TTD in execution layer clients even before August 23 so that node operators could start setting up their infrastructure for mainnet Merge in advance. This is an issue that Paul Hauner from the Lighthouse team has raised on previous developer calls. However, given that a tentative TTD value has been picked. It may be more likely that certain EL client teams simply update their releases with the actual mainnet Merge TTD. However, for any EL client teams planning on releasing version before August 23, Hauner encouraged them to use the placeholder TTD value already coded into consensus layer client software for consistency in the meanwhile until the tentative TTD value picked during Thursday’s call is confirmed.
Previously, developers discussed implementing a circuit breaker design to MEV-Boost. MEV-Boost is an optional software that validators on Ethereum can run post-Merge to earn additional revenue in the form of maximal extractable value. To learn more about MEV on Ethereum, read this Galaxy Digital report. To mitigate the impact of a malicious relay intentionally withholding blocks at the last moment from validators, certain Ethereum client teams such as Lighthouse and Prysm have moved forward with implementing a circuit breaker. For validators running a Lighthouse node, 8 missed slots will trigger the validator to start relying on their own block-building process as opposed to that from a third-party relay. Prysm has left the exact number of missed slots needed to automatically disable MEV-Boost undetermined but potentially randomizable. To prevent anyone from taking advantage of a circuit breaker function, developers agreed that the exact threshold for missed slots should be different across each Ethereum CL client team. Ethereum Foundation researcher Alex Stokes emphasized that it would be ideal to have this functionality implemented before the Merge date. In addition, Stokes pointed out one minor change that would be added to the code specifications of MEV-Boost. Specifically, the specifications for third-party builders will state explicitly that the validator should be building their own block locally in parallel to the blocks they are expecting to receive from an external builder network, that is a relay.
Developers ran out of time during Thursday’s call to discuss Ethereum Improvement Proposal 4844 in depth. However, it was raised as a discussion item for future calls. EIP 4844 is a code change that will likely be included in the hard fork following the Merge called Shanghai. It is designed to improve Ethereum’s scalability through the use of rollups and more specifically, by making the cost of using rollups cheaper on Ethereum. To learn more about rollups, read this Galaxy Digital report. The items for discussion were around how the network would handle the new transaction types introduced by EIP 4844. In specific, how these new transaction types should be synced and priced through Ethereum’s fee market.
One other area of discussion on today’s call was on the terminal block hash override. This was a minor change to the specifications of Ethereum execution layer clients proposed by developer Mikhail Kalinin. Most EL teams including Geth, Nethermind, and Erigon have not yet implemented the code change. As such, Danny Ryan suggested that this code change not be pushed into final client releases for the Merge. Rather, the code change should be ready for implementation in case of a network emergency. For more background on the edge cases around terminal blocks at the time of the Merge, refer to the call notes from CL call #92.