Client Team Updates

Merge coordinator

Validator Comms Lead @ Gnosis

Core Devs May 10

For those who missed the live meeting, a full recording is available on YouTube.

May 10, 2023

EL

Nethermind: Not much. Still working on Shutter
Erigon: Fixed an issue where trace_block was not returning correct rewards because of the block rewards contract. Was merged yesterday.
Geth: Small update: Struggling with the rewards computation code. We'll be in touch with Andrew.

CL

Prysm: No updates.
Nimbus: No updates.

Chain Infra

Gateway : All set up for EDCON (May 19-23) Shared the API key yesterday, hasn’t received any updates from Burner Wallet. Would like to get a version of the wallet in order not to have any updates. Gateway is fine with increasing rps and nodes. Need information from Gnosis. Deposit UI needs to be updated for the hard fork.

Devnet

Gnosis ran manual tests, everything seems to be ok. No updates from Nethermind.

Testnet

No objections from Nethermind. Should be doable on Erigon’s side as well

Suggested slots Conditions. Slot mod 8192 = 0 (aligns with historical beacon vector) Is weekday? Is between 14:00 and 20:00 CET? Within the next 2-4 weeks Matches

Slot 3874816: 22/05/2023, 16:41:20
Slot 3891200: 23/05/2023, 15:26:40
Slot 3907584: 24/05/2023, 14:12:00
Slot 3997696: 29/05/2023, 19:21:20
Slot 4014080: 30/05/2023, 18:06:40
Slot 4030464: 31/05/2023, 16:52:00
Slot 4046848: 01/06/2023, 15:37:20
Slot 4063232: 02/06/2023, 14:22:40
Slot 4153344: 07/06/2023, 19:32:00

Can we try for May 24, 2023? Consensus chain variables. We’ll keep the same ones as Ethereum mainnet. How many withdrawal blocks should we accept?. On Ethereum it’s 16. Worst case it’s around 1.3 million gas per block. This would iterate over all partial withdrawals in ~8 hours, which is very fast. We could do 8 withdrawals per block. In any case we’ll add 550k gas per block. We have to define a list of KPI we want to reach on Chiado to consider it a success.

Research

Shutterized Beacon Chain No updates.

Withdrawals: Right now it’s fairly complicated to track information regarding withdrawals. We might need to change the signature of the withdrawal function. Should not be too complicated for Nethermind and Erigon. The contract would also require an update to track these changes

Tests

Hive:Added the integration with the withdrawal contract. Adding the way to test the balance of the contract and the amount on the accounts that have done withdrawals. Should be done by end of this week No ETA for the rest yet.

Core Devs Call - 2023/05/03

May 3, 2023 · 3 min read

Merge coordinator

Validator Comms Lead @ Gnosis

For those who missed the live meeting, a full recording is available on YouTube.

May 3, 2023

Client Team Updates

EL

Confirming EIPs

EIP-3651: Warm COINBASE
EIP-3855: PUSH0 instruction
EIP-3860: Limit and meter initcode
EIP-170 is added as well because it was still missing https://eips.ethereum.org/EIPS/eip-170
Regarding max code size
EIP-6049: Deprecate SELFDESTRUCT
EIP-4895: Beacon chain push withdrawals as operations** Gnosis modified. These need to be added to our specs repo https://github.com/gnosischain/specs/issues/5
Nethermind: Started work on the shutter chain. Nothing specific to show yet. Running 50% Erigon nodes
Erigon: No updates
Geth: I got fullsync to kickstart after the database reconstruction: code was missing. I fixed this and now I am debugging the "system call" at the end of the block, where the block reward is distributed. Once this is fixed, and since it's the last thing that gets executed in the block, fullsync should work.

CL

Prysm: No updates.
Nimbus: No updates.

Chain Infra

Gateway : Nothing specific. Igor is participating in a hackathon today

Devnet

No updates from Nethermind, Erigon or Gateway. Nethermind, ran a tool that generates bad blocks to test if the network handles them well. Everything is working well. Tests on Gnosis’ side; we haven’t tested a lot on our side yet. Defined a todo list of what we are going to test this week. https://github.com/gnosischain/consensus-deployment-ansible/issues/67

Research

Shutterized Beacon Chain Being implemented as a Nethermind plugin

Tests

Hive Last week’s issue was solved. Related to geth being hardcoded. Trying to make this more universal. It was recalculating a hash instead of just taking one received by the API. Ensure that we reach the TTD. Generating some post-shanghai blocks. During each step, we check if the chain is working correctly. Should have some withdrawal tests for end of next week.Focused on integrating the withdrawal contract this week

Core Devs Call - 2023/04/26

April 26, 2023 · 2 min read

Merge coordinator

Validator Comms Lead @ Gnosis

Greetings everyone, and welcome to the weekly Gnosis Core Devs Call. Just a quick reminder that this meeting takes place every Wednesday.

Participants: Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team and the contributors.

April 26, 2023

Client Team Updates

EL

Nethermind: Got close to releasing the first hive tests. Maybe later today or tomorrow. Preparing a new major release. Found some issues, mostly optimizations and bug fixes. Probably going to be released next week. Will send the release notes once they have them
Erigon: New release (2.43) Replaced the light consensus client with a more experimental full consensus version. As it’s a full blown CL implementation, it should also be able to work for Gnosis in the future--externalcl is now default (and the flag was removed).Use --internalcl now to use the new experimental CL. Looking to hire a new core developer. Gnosis HR has been helping. New diagnostic tool https://github.com/ledgerwatch/diagnostics. Idea is to also use it as a recruitment tool
Geth: Still has a sync problem. Nethermind hangs up without telling why. Spent some time with Nethermind to fix this. Related to a wrong chain ID. Resulted in the need to resync everything

CL

Prysm: No updates.
Nimbus: No updates.

Chain Infra

Gateway : No peers found on Devnet (probably just the different genesis).

Devnet

No major updates. Withdrawals look good. Activated ~80% of the keys and withdrawals work. Failure scenario works as well.Exiting worked a few times as well.

Research

Shutterized Beacon Chain The EL specs seem to be clear enough. Nethermind can start prototyping something until libp2p is made available.

Tests

Hive Ran into issues regarding the config. Five basic withdrawal tests should be delivered today or tomorrow. The blocker is regarding the payload. The payload isn’t communicated correctly between CL and EL.

Core Devs Call - 2023/04/19

April 19, 2023 · 2 min read

Merge coordinator

Validator Comms Lead @ Gnosis

Greetings everyone, and welcome to the weekly Gnosis Core Devs Call. Just a quick reminder that this meeting takes place every Wednesday.

Participants: Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team and the contributors.

April 19, 2023

Client Team Updates

EL

Nethermind: No updates.
Erigon: No updates.
Geth: Was able to import the state (simulate snap sync)Is now starting the node in full sync, but it’s not connecting / downloading blocks. Once this is done, Guillaume should have a working prototype

CL

Prysm: No updates.
Nimbus: No updates.

Chain Infra

Gateway

No peers found on Devnet (probably just the different genesis).

Devnet

Wrong configuration was shared. There’s now a mismatch for outside nodes. There’s no Erigon node running at this point. 80% attestation rate (all nodes ran by Nethermind) Shapella went through Lion: does Nethermind have metrics for the network? I.e. withdrawal amounts per block etc. Apparently not. Might potentially be added. Most credentials are still 0x00. Meaning that no withdrawals are currently happening.

Research

Churn coefficient https://hackmd.io/@dapplion/gnosis-churn-limit Defines the number of validators that can join or exit the network in a specific time frame. On Gnosis that number is a bit high, meaning that it would theoretically be possible to get a big stake in the network very quickly. This would require a consensus client code change. Technically a change on the deposit contract could also limit inflow. But that break a lot of assumptions. It doesn’t change anything in the case of outflows.Attacking the light client bridge is exclusively an inflow problem

Shutterized Beacon Chain: is there progress? We’ve got more precise specs on a PR https://github.com/gnosischain/specs/pull/4 Lion: how specific do you need the spec to be? It seems like EL clients have more lax requirements for specs, is that true? On CL everything is specced quite precisely.

Tests

Hive

Will start writing Hive tests by the end of this week Improving the genesis loading for Erigon and Nethermind, but hasn’t been tested on Erigon yet. Lion: can we agree on which conditions tests would be considered good enough to move to testnet / mainnet? Ruben thinks it would be enough if we can just run the contract specific parts (balances etc)

Core Devs Call - 2023/04/12

April 12, 2023 · 2 min read

Merge coordinator

Validator Comms Lead @ Gnosis

Greetings everyone, and welcome to the weekly Gnosis Core Devs Call. Just a quick reminder that this meeting takes place every Wednesday.

Participants: Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team and the contributors.

April 12, 2023

Client Team Updates

EL

Nethermind: No updates. Steadily progressing with Hive tests
Erigon: No specific updates, waiting for Shanghai to happen. Marcos was asking for an Erigon config for Gnosis. Erigon doesn’t support single file configs for AuRa chains. It will require some code change. Is this required for Hive tests? Not absolutely necessary, but would be useful so that it’s generic for every client Andrew will take a look at it this week.
Geth: No update

CL

Prysm: No update
Nimbus Started working on a DAppNode package

Chain Infra

Gateway

Bridge validators up and running and there are successful transactions for both (native and AMD) Helped Giveth team to setup their bridge validators by sharing steps via notion. Looked into the shapella issue on devnet

Plausible scenario: Lighthouse lost all peers just before it was its time to propose the block, so it began a new chain, then it couldn’t re-connect to reorg so it only proposed blocks when it was its time

Doesn’t seem to be the issue with Erigon or Nethermind Unlikely to repeat in real life because we won’t have that little actual nodes and that many validators. But we need to be careful with Chiado, because the setup is similar

Waiting for the next steps for the next devnet.

Gnosis

Setting up network metrics and alerts Grafana and Prometheus running. Doing some tests. We will follow up with new updates next week. Also have Telegram notifications working. We will invite appropriate teams in due time.

Devnet

We have consensus on the current broken devnet. We’re going to spin a new one up. Mainnet hardfork is happening today, so it might be a blocker. To be confirmed by Carlos

Tests

Hive

How to open up the network to outside validators? GNO faucet? New testnet for validators to keep Chiado stable for other testing?

Core Devs Call - 2023/04/05

April 5, 2023 · 4 min read

Merge coordinator

Validator Comms Lead @ Gnosis

Greetings everyone, and welcome to the weekly Gnosis Core Devs Call. Just a quick reminder that this meeting takes place every Wednesday.

Participants: Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team and the contributors.

April 5, 2023

Client Team Updates

EL

Nethermind:

Merged withdrawals to the master branch

Erigon:

No updates

Geth:

Was still syncing on Monday (~1 month left to sync)

CL

Prysm:

No updates

Chain Infra

Gateway

Has been helping quite a few teams. Giveth to setup their bridge validator. Max helped Marcos with Hive tests. Took all the traffic from the main Gnosis RPC because Gnosis was experiencing issues. Everything went fine

Gnosis

Had an outage on the public RPC. Redirected all the traffic to Gateway

Devnet

Gateway (on 4 nodes for 5k validators) that 3 validators are on the right chain and working correctly
- 2 have Nethermind
- 1 has Erigon: The last Erigon forked away on the block that it created by itself (#88092). A previous block by the same node has been accepted by the network. Except by the broken Erigon node. The failed block has a different state root and an additional withdrawal. This specific node rejected a block that was considered valid by all other nodes. We don’t exactly know why the broken node rejected the valid block. There was an unwind right before on the broken node. Logs for both Erigon nodes are in the Telegram group
Nethermind (5-6 Nethermind nodes, out of which 2 had issues)
Lodestar: Seemingly peering issue Ping-pongs between having enough peers and not having enough, so it wasn’t able to keep in sync
Teku (although another one was working fine) Started sending FCUs of the same blocks over and over again around 1k blocks before it lost sync Might have been connected to Gateway’s broken Erigon node
Gnosis Experiencing issues as well (running Erigon, Nethermind, Teku and Lighthouse, 2.5k keys total).Nodes are stuck on different block numbers. Currently debugging more in depth. None of the 4 validator combinations are working correctly and are on different forks. Giacomo and Igor will keep in touch to debug this.
Next steps
- Find the issue (Andrew)
- Launch a new devnet Alerts (Lion)

The objective is to track everything that can go bad and send alerts to the relevant parties. Has a private repo with a bunch of alerts. Covered basically everything on the consensus side. Will add the relevant people on the repo async. Will be set up by Gnosis DevOps, Giacomo knows about it already. Ruben tested failed withdrawals on Apr 4, 2023

The current implementation executes withdrawals transactions after standard transactions. This means that the event logs are missed. We might want to drop them, as they are thus useless. To track specific parameters we could add more variables. Erigon behaves in the same way. Instead of using events to check if a withdrawal was successful, we could check the state of the contract to see if the transaction was added to the failed transactions. If not, it should be considered successful This is not super accurate though and doesn’t give great visibility.

Lion thinks it might be possible to do it with the current withdrawal contract implementation, but if not we can add some primitives. There might be something with public variables on the contract:

failedWithdrawalsPointer numberOfFailedWithdrawals

Lion want to know how exactly the devnet was bootstrapped. ruben@nethermind.io will send this offline

Tests

Hive

Had a call last Thursday regarding what our objectives were defined a overview of what things should look like. Hopefully more information next week.

Withdrawal Contrat

No updates

Core Devs Call - 2023/03/29

March 29, 2023 · 2 min read

Merge coordinator

Validator Comms Lead @ Gnosis

Greetings everyone, and welcome to the weekly Gnosis Core Devs Call. Just a quick reminder that this meeting takes place every Wednesday.

Participants: Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team and the contributors.

Mar 29, 2023

Client Team Updates

EL

Nethermind:

Introducing Marcos for Hive tests. Will have a call tomorrow regarding those tests.

After Shapella, Nethermind will probably release a very big version with a lot of changes. Huge amounts of optimizations is coming.. Potential new light mode that would reduce memory consumption a lot

Improving attestation performance Better multi-threading Potentially (but way later) snap sync

Erigon:

Canceled their agreement with Gateway for developers, thus Max is no longer an Erigon dev. They want to hire new devs, but will take time. Added the max failed withdrawal parameters to the withdrawals system call, which should now work

Geth:

Guillaume has worked around the issue from last week, and is now importing the data to start syncing the chain. It has been running for two days, and there’s probably one more day to go, after which he’ll be able to give an update.

CL

Prysm:

No updates. Guillaume is focused on Geth

Nimbus:

We’ve written and reviewed our docs to be published soon After this, we’ll publish client diversity docs, so that validators can start using Nimbus.

Chain Infra

Gateway

A mainnet bridge validator is now running and has already processed a few transactions.Extra monitoring was added there.

Devnet

Withdrawals are working again. Tried partial withdrawals, not full exit yet. Gnosis mostly deployed Nethermind nodes. No one seems to know if Erigon nodes still sync. Andrew’s Erigon nodes still work

There’s close to 100% attestation participation.Ruben can do a full exit on 1024 nodes to drain the GNO in the deposit contract.

There’s a faucet for GNO afterwards.Ruben expects Gnosis to be testing this a bit more

Lion will run Truffle scripts

Tests

Hive

There’s a call tomorrow, invite in Telegram Happening at 3pm CET

Withdrawal Contrat

Adam’s audit is done, and another external audit should be done by early next week. The minimum amount to deposit is 1/32 GNO, which could be an attack vector. Looking into it. We might want to increase that minimum.

Core Devs Call - 2023/03/22

March 22, 2023 · 3 min read

Merge coordinator

Validator Comms Lead @ Gnosis

Greetings everyone, and welcome to the weekly Gnosis Core Devs Call. Just a quick reminder that this meeting takes place every Wednesday.

Participants: Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team and the contributors.

Mar 22, 2023

Client Team Updates

EL

Nethermind: Bad block fix was shipped.
Erigon: No updates
Geth: Imported some blocks, but there’s issues with the freezer, hopefully that’s the last hurdle

CL

Prysm: No updates

Chain Infra

Gateway

Increasing mainnet RPC to 75% traffic. Implementing monitoring for the Chiado bridge. As soon as this is done, they’ll deploy both bridges for mainnet

Devnet

Some issues: 3 phases

Launching chain
- No problem
Merge
- Issues caught before the merge and fixed in time
Shapella hard fork
- Erigon nodes got stuck with “known bad hash”. Erigon now stuck with another bug
Gateway switched Erigon nodes to Nethermind
- Issues regarding sync committees Mar 22 13:01:07.055 WARN Error processing HTTP API request method: POST, path: /eth/v1/beacon/pool/sync_committees, status: 400 Bad Request, elapsed: 4.13746ms
Erigon
- Was a good test, revealed some issues. DDoS protection from geth was problematic but is fixed. Missing gas limit implementation. The withdrawal contract reverts on Erigon, but works fine on Nethermind. Missing all the slots (or most?) Bad blocks on Sepolia, but works on Goerli. If any bad block is detected on Goerli, please tell Andrew. Could be linked to an old version of Erigon. Andrew will check on their Sepolia validators
Nethermind
- Consensus issues need to be investigated separately. We need consensus between both clients to test client diversity / parity

Igor: where do (automatic) withdrawals go for genesis validators? Do they have a withdrawal address? Lion: the default tooling sets addresses to 0 Jorge: they sometimes set withdrawal addresses on genesis or after, depending on what’s to be tested Ruben: can help on how to check if withdrawals are happening or not The failed withdrawals per slot are fixed to 4 on Nethermind’s side for now Lion: this should be a consensus config Marek: how can we be sure that we are triggering all testing paths? Lion: we could add coverage tools from Solidity to automate those tests Igor: we need tooling to automatically test withdrawals, because they happen on both consensus and execution Lion: for the withdrawal contract we can easily test it on chain now because it’s insolvent at genesis anyways, as the validators are created in genesis and thus there’s no GNO in the contract Jorge: instead of deploying a new devnet or messing with the current one, we can create a shadow fork and modify some storage there to test specific scenarios I.e. removing the GNO in the contract for example Marek: downside is that this is not automated testing for the future Jorge: indeed, but we don’t have automated tests yet and that’s the best we have

Testnet

Was supposed to hard fork next week: still realistic? Igor: probably wise to postpone until Erigon is ready

Tests

Hive

Max not present: no update

Withdrawal Contrat

mGNO is getting removed Would require a contract upgrade on devnet. Will be tested on devnet by triggering actions that provide good solidity coverage

Core Devs Call - 2023/03/15

March 15, 2023 · 5 min read

Merge coordinator