Ethereum Consensus Layer Call #91 Writeup
On Thursday, July 14, Ethereum core developers concluded their 91st fortnightly Consensus Layer (CL) call. Chaired by Ethereum Foundation researcher Danny Ryan, the CL calls are one of two recurring meeting series where Ethereum developers discuss and coordinate upgrades to the protocol of Ethereum. Over the last several months, these bi-weekly calls and the All Core Developer (ACD) calls, which occur every other Thursday, have focused primarily on preparations for Ethereum’s Merge upgrade. For more information about what the Merge upgrade is, read this Galaxy Digital Research report.
On the call developers discussed:
Outcomes from mainnet shadow fork 9;
Preparations for mainnet activation on the Goerli testnet;
Potential mainnet merge activation timing and process;
Updates around MEV-Boost.
Mainnet Shadow Fork 9
Developers successfully spun up their 9th mainnet shadow fork on Thursday, July 14 to test the Merge upgrade. For details around what a shadow fork is and how it is used to test the Merge upgrade, click here. Ethereum Foundation’s Parithosh Jayanthi explained on the call that total terminal difficulty (TTD) was hit on the devnet roughly 8 hours earlier than expected. For details around what TTD is and how it works, click here. This despite re-running estimations for TTD twice. Developers are unsure about why their estimations were so off. However, this does go to show how unreliable and unpredictable estimating Merge timing is based on a TTD value given that network hashrate activity is variable across mainnet and Ethereum testnets.
Other than the TTD being hit earlier than expected on mainnet shadow fork 9, Lighthouse (CL) nodes ran were falling out of sync with the network post-Merge. The bug causing this behavior has since been patched by the Lighthouse team. Besu (EL) nodes were also producing invalid blocks due to an ongoing data storage issue, which is not related to Merge code but rather an optional storage format unique to the Besu client software. For more background on this issue, click here. Nethermind (EL) nodes were purposefully kept offline through the Merge upgrade to debug a separate issue that was first identified during mainnet shadow fork 7. Erigon (EL) nodes were not able to sync properly to the head of the chain. It appeared they simply need more time, though developers were unsure why the syncing was so slow. Finally, Nimbus (CL) nodes had a slight configuration issue that has since been resolved.
So, all in all, rocky start for mainnet shadow fork 9. However, most issues identified above are considered minor by developers and many have already been patched. There remains some debugging work to do particularly with the Nethermind software client and that will occur asynchronously between developers throughout next week.
Goerli Merge Preparations
Given that developers during last Friday's All Core Developers’ call discussed potentially picking a date for the Goerli Merge activation soon, Jayanthi suggested doing a shadow fork on the Goerli network next week. As background, the Goerli network is a proof-of-authority blockchain supported by multiple Ethereum clients. It has been around since March 2019. The Goerli network will soon undergo the Merge upgrade by fusing with the Prater Beacon Chain, which is an Ethereum proof-of-stake (PoS) testnet. Goerli is one of two testnets that Ethereum developers will actively maintain alongside Sepolia post-Merge.
Developers discussed the naming around the Prater Beacon Chain after its Merge activation. There are several flags and documentation that refer to the Prater Beacon Chain as Prater but given that it will merge in the coming weeks with the Goerli network, developers agreed to start adding in a Goerli flag to Prater documentation. They decided not to deprecate the Prater flags, however, to avoid creating any breaking changes or reasons to resync existing Prater nodes.
Mainnet Merge Preparations
Then developers briefly discussed a minor change to the mainnet Merge activation process, which will be replicated on the Goerli testnet as well. For a full overview on the Merge activation process, read this Galaxy report. Basically, instead of two TTDs being released during the Merge activation process (as detailed in the linked report), there will only be one which is baked into the EL and CL client software. The reason for this change is to reduce the complexity of the upgrade and improve the user experience for node operators following along through the Merge activation. For a complete breakdown for pros and cons to the TTD strategy, read this writeup by Chair of the ACD calls Tim Beiko.
About Merge timing, developers were optimistic that if all client teams are ready to cut releases for the Goerli Merge activation by the last week of July, then the Goerli Merge can be scheduled for the week of August 8th. Assuming the Goerli Merge goes smoothly, developers anticipate scheduling mainnet Merge activation in late August and executing it throughout the month of September. I say throughout the month of September because the Merge is a bundle of two hard fork upgrades which will be scheduled roughly 2 weeks apart. Keep an eye out on GithHub and the Ethereum Research and Development Discord Channel for updates on a TTD value for the Goerli Merge activation in the coming days!
Finally, developers discussed ongoing testing and preparations for activating MEV-Boost software. For more information on what MEV-Boost is, click here. Teku (CL) and Prysm (CL) client teams have finished implementing their version of the software. Lighthouse (CL) is close. MEV-Boost has recently undergone a third-party security audit which uncovered some minor and medium-level issues. Read more about those here. MEV-Boost is being actively tested across various shadow forks and testnets.
Developers agreed on Thursday's call to bake in a slight delay to the activation of MEV-Boost builder specs in CL client software. This is again to reduce the complexity of the Merge upgrade. Instead of activating this software right at Merge, validators will be encouraged to run MEV-Boost after the network is considered finalized, meaning after the network has run through 2 epochs, roughly 12.8 minutes. Of course, users can still override this delay. However, it is a decision that will likely be honored also by the Flashbots team who will be operating a relay, which connects third-party block builders to validators running MEV-Boost software. I talk about MEV-Boost in slightly more detail in this week's Galaxy Digital newsletter. Read our full newsletter which was published today here.
The way CL client software is built, it is easier to bake in a delay based on network finalization than a specific number of epochs passing from the time of the Merge. This is because CL nodes newly syncing to the network are not required necessarily to know exactly how many epochs have passed since Merge activation. There was an interesting back and forth between developers about the most straight-forward approach to embargoing MEV-Boost, which I encourage you to check out through the livestream. It's basically the last 10 minutes or so of the recording.
For another perspective on yesterday's call, check out Ben Edgington's summary here.
Bonus: Merge Community Call #5 Notes
Today, a few prominent Ethereum community members and developers hosted a AMA about the Merge upgrade. Check out the full agenda and pre-call resources here. A recording of the AMA will be uploaded shortly to YouTube. For me, I took today's community call as an opportunity to ask a bunch of questions. Here are some highlights from the call:
Setting up your validator or node to run on a post-Merge testnet is the exact same process as setting up your node pre-Merge, so all validators and relevant network stakeholders are encouraged to start testing out their set-ups now even before the Goerli Merge activation.
For stakeholders who want to test out set ups during the Goerli Merge, keep in mind that the upgrade is expected to happen in early August and node operators will only be given roughly a week's notice. For mainnet, node operators will be given roughly two weeks heads up. It's a limited window because estimating TTD several weeks out in advance is hard.
On the topic of consensus layer (CL) client software distribution, Lighthouse nodes have surpassed the 1/3 mark. Prysm remains the dominant client but not by much with 39% of CL nodes running Prysm. More data on client diversity here and more info about why this is an important issue for Ethereum security here.
Kiln, Ropsten, and Rinkeby are all Ethereum testnets that will be deprecated sometime after the Merge. Kiln most quickly. Then Ropsten probably sometime before the end of the year will get deprecated. Then Rinkeby perhaps are year out after the Merge. While these three testnets are up, they will not be upgraded as post-Merge Ethereum undergoes upgrade. The only two testnets devs will actively maintain post-Merge are Sepolia and Goerli.
MEV-Boost is kind of like a side car that runs alongside the main car, which is the consensus client (CL) software, that validators can choose to run to earn MEV. Watch the recording of this AMA for Danny Ryan's full explanation of MEV-Boost and its components.
The Engine API is a piece of software that unifies many aspects of the core Merge specifications and creates a cohesive interface for setting up EL/CL node communication. 100% of Ethereum software clients use this API. However, other can and will be built.
The Merge is coming sooner than you expect if you're not paying attention and trying to test stuff ahead of time. It is highly encouraged that all node operators test out their setup AT LEAST once before mainnet Merge activation.
Wen Merge? Follow along through this readiness checklist.