Ethereum All Core Developers Consensus Call #120 Writeup
On October 19, 2023, Ethereum developers gathered over Zoom for All Core Developers Consensus (ACDC) call #120. Chaired by Ethereum Foundation Researcher Danny Ryan, the ACDC calls are a bi-weekly meeting series where developers discuss and coordinate changes to the consensus layer (CL) of Ethereum. This week, developers discussed:
version 1.4.0-beta.3 of CL specifications,
launch conditions for Devnet-10,
and blob latency analysis by Gajinder Singh, a software developer who maintains the Lodestar and EthereumJS clients.
Danny Ryan announced the release of new CL code specifications for the Cancun/Deneb (Dencun) upgrade, dubbed “The Summoning.” The release, formally tagged as version 1.4.0-beta.3 in the CL GitHub repository, features two main changes:
Mainnet KZG configurations: The formatting work required to finalize the output from Ethereum’s trusted setup ceremony has been completed and included in the latest CL spec release. For background on what KZG is and the role it plays in EIP 4844, read this Galaxy Research report.
New gossip rule: Teku developer Enrico Del Fante has created a new gossip rule to ensure that CL nodes do not propagate a higher number of blobs than the maximum number of blobs per block, which is currently defined in the specifications as six blobs per block. This will ensure that validators cannot spam the network with invalid messages exceeding six blobs per block.
The changes in version 1.4.0-beta.3 of CL specifications will be tested on the next devnet, Devnet-10. However, there are a few blockers to the launch of Devnet-10. Barnabas Busa, a DevOps Engineer for the Ethereum Foundation, said that he is still waiting on client teams for new software releases. Once these are ready, Busa hopes to launch the next devnet sometime tomorrow on Friday, October 20th.
A pseudonymous developer for the Prysm client who goes by the screen name “Potuz” noted that the mainnet KZG configurations included in the latest CL specifications cannot be merged to Geth and therefore Prysm until the changes are made to “go-kzg,” a separate code repository implementing the KZG commitment scheme into the Go programming language. Developers expressed frustration on the call about the dependencies between go-kzg, Geth, and Prysm repositories. This is not the first time Potuz and other developers have raised concerns about the challenges of using KZG libraries for EIP 4844.
Parithosh Jayanthi, also a DevOps engineer at the Ethereum Foundation, shared that developers could launch Devnet-10 without mainnet KZG configurations and new client releases but this would mean developers are not testing any new code on Devnet-10, but rather re-testing the code launched on Devnet-9.
This week, developers discovered an issue in CL client implementations on Devnet-9. Mario Vega who is on the testing team at the Ethereum Foundation explained validators are not correctly pruning invalid blobs. If a validator maliciously broadcasts a correct blob to certain peers and an invalid blob to other peers, the peers that had received the invalid blob will be unable to follow the head of the chain, Vega said. The precise conditions that result in such behavior have already been created into a hive test, meaning these conditions can be easily reproduced and tested against new client releases so that developers can know immediately whether their fixes for this issue are working or not.
Related to this issue, Enrico Del Fante mentioned that the Teku client will not import a block under certain conditions if the block contains a blob or number of blobs with an index that doesn’t match any existing blob commitments in the block. Ethereum Foundation Researcher Dankrad Feist noted that the validator proposing blocks intentionally with invalid blobs would get to no benefit from doing so other than potentially increasing the computational load of Ethereum’s peer-to-peer network and causing certain validators receiving the block to fall out of sync with the network. Feist emphasized that there is no economic gain from such behavior and even if a validator were to do this, the additional computational load on Ethereum’s peer-to-peer layer would be capped by the maximum number of blobs per block.
Even so, to dissuade this type of behavior from validators, developers discussed potentially adding a new slashing condition. The new slashing condition would try to monitor for the inclusion of invalid blobs into a block and dissuade validators from intentionally causing strain on the peer-to-peer layer. However, due to the high cost in terms of research and analysis needed to change Ethereum’s validator economics, Ryan suggested that this proposal be discussed further in context of the next upgrade after Dencun, Prague/Electra.
Ryan added that developers could also specify in CL specifications for Dencun the correct behavior under this scenario, which is that validators should still import blocks regardless of whether blob indexes match block commitments. Ryan suggested that since there is no economic rationale for a validator to propagate blocks with invalid blobs in the way described by Del Fante and the maximum strain on network load such irrational behavior could cause is capped by the max blobs per block limit, the minor change to CL specifications should be enough to address the issue in the short-term while developers consider more permanent solutions like a new slashing condition for future upgrades.
Jayanthi reconfirmed with Ryan that developers will not launch Devnet-10 until mainnet KZG configurations are setup in clients and the issue identified by Mario Vega is resolved in at least the Prysm client. Ryan reaffirmed that the conversation about a new slashing condition is only in context for the Prague/Electra upgrade, not Dencun, and the change to CL specifications about the importing of blocks with invalid blobs should not be a blocker for the launch of Devnet-10. Representatives from the Prysm and Lighthouse client teams said that they will have an updated release for their software by tomorrow, October 20.
Block Latency Analysis
On Saturday, October 14, Gajinder Singh, maintainer of Ethereum clients Lodestar and EthereumJS, shared new analysis on block import latency based on number of blobs gossiped. Singh noted that from his experiments on Devnet-9, the latency of blocks increases significantly, the higher the number of blobs received by the validator.
The following is a table summarizing Singh’s findings. The first column lists the percentage of full block arrivals. The values in the table are the seconds it takes for the block to be imported.
Caption: Block import latency in seconds by number of blobs included in a block.
Source: Gajinder Singh, Twitter
Singh tweeted that while the latency introduced by one blob per block is negligible, any number higher than two introduces significant latency. Dankrad Feist said that the numbers from Singh’s analysis were very different from his experiments propagating large blocks on mainnet Ethereum. “It seems like this data can’t be correct,” asserted Feist, adding that blobs are processed in parallel with blocks so adding the latency from blob processing to block times are an inaccurate way to create projections about block propagation times on mainnet. Singh’s projections about block propagation with blobs can be found here. Feist called Singh’s analysis “overly pessimistic.”
Despite the disagreement, both Feist and Ryan admitted that Singh’s analysis did warrant a replication by another client team. Ryan encouraged another CL team to try and reproduce Singh’s experiment on Devnet-9 to see if they get similar values and results. If there is new data to present about block latencies with the introduction of blobs, Ryan encouraged developers to re-discuss the topic on next week’s ACDE call.