Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 18, 2023

EL​

• Nethermind:

  • Finished some Gnosis-specific features
    • Configurable EIP-4844 configurations
  • Cancun merged to main branch
  • Kickstarted an initiative to rework the eth_getLogs system to optimize it

• Erigon:

  • Sync issues (OOM on 32 GB nodes also running CL)
    • Exclusively happens in Docker, GOMAXPROC should be a workaround (4 instead of 24 for example)
  • High disk usage (400 GB with --prune=hrtc vs 170 GB on Nethermind for the standard gnosis preset)
  • Fairly unreliable (I have to restart it every few days at this point, maybe too many attached validators? I miss quite a few epochs on mainnet too but don’t need to restart at least.)
  • Configurable EIP-4844 configurations are merged
  • Working on Erigon 3

• Geth:

  • Tried the static peers and it seems to work better
  • Still not getting a connection
    • The old branch systematically uses eth/66 and that works
    • The new branch gets disconnected when trying to connect to eth/66

Dencun

• Config
  • Max churn limit of 2
  • Target and max blobs per block
  • Security first: 1/2
    • Thoughts of EL devs?
  • https://github.com/gnosischain/specs/pull/24
  • BLOB_GASPRICE_UPDATE_FRACTION=1112826

Chain Infra

• Gateway
  • Discovered a bug in the load balancer that resulted in 0.5 to 1% of errors at the current load
    • Sudden increase in eth_getLogs, which are typically quite slow on the nodes, which were subsequently falling behind
    • The solution was to account for this slowness

Gnosis

• The public RPC was down during the weekend, but that was because of Gnosis.

Hive Tests

• Solved the configuration issues for Dencun
• Working on an error when sending blobs

Devnet

• Has Erigon joined?
  • thorax/erigon:docker_gno_dencun_devnet_0
  • --chain=gno_dencun_0
  • Issues with the internal CL
  • Not tested yet

Research

• Automatic sync tests

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams such as:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 11, 2023

EL​

Nethermind:

• Working on internal dencun specific things for Gnosis
• Right now 4844 variables are hardcoded in Nethermind and they’re working on making this configurable
• A new release candidate is underway, majorly integrating cancun features along with some fixes.

Erigon: No one present

Geth:

• Not much progress since last week, mostly investigating
• 2 branches
  • One that extracts the data from the Nethermind DB and creates the DB to use by the next branch
    • Figured out why it was running out of memory: memory leak in the transaction pool, but didn’t spend too much debugging because of the new branch
  • One that is more modern and supports withdrawals, but doesn’t currently sync.
    • Has issues connecting to other nodes in the network.

Chain Infra

Gateway No one present

Hive Tests

• Fixed the withdrawal issues after merge.
• Working on the dencun tests.

Devnet

• The devnet nodes have been stable and the network is finalizing
• Lighthouse requires the preset to be built in?
• The blob transaction spammer has some issues, it would be better to try to internal Nethermind tool
• Running with the min gas per blob preset since Friday
• Two more parameters need to be updated, and we need to spin up a new devnet with Erigon from the get go

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

October 4, 2023

EL​

• Nethermind: Working on Shutter No more xdai(_archive) configs in the new Nethermind release (1.21.0) Potential issues in Avado

• Erigon: The header download functionality is acting up a bit from time to time Somnath is working on it Transaction pool should work on Gnosis but hasn’t gone through extensive testing Same for validating

• Geth: Was able to sync 1.5m blocks Geth is now running out of memory on the old branch Did a rebase and is cleaning up some code, but that new branch isn’t syncing

Chain Infra

• Gateway Is running 100% of the RPC traffic of the Gnosis RPC

Dencun

• EIP-4788
  • We believe that the HISTORY_BUFFER_LENGTH variable doesn’t matter that must
    • Having 1.14 days or 0.47 days give roughly the same UX
• We’re going to keep that number and thus deploy the exact same contracts
  • Gnosis: https://gnosis.blockscout.com/address/0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02
  • Chiado: https://gnosis-chiado.blockscout.com/address/0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02

EIP-4844

• Big block experiment
  • The network suffers from big blocks (we spammed it super cheaply for a 10 - 20 minutes)
  • The head participation went down quite a bit (close to 20%)
• Min gas price will be 1 gwei to be on the same scale as normal transactions (1 gwei minimal priority fee)
• Nethermind wrote a tool to spam blobs and it broke a lot of clients
  • These might just have been early implementation bugs
• EIP-7514
  • https://hackmd.io/jWVHqjmrQ1aXtpt6iTNTdw?view
  • An attacker could get a 67% share in the network in 2.5 days because the entry queue scaled way too quickly

Hive Tests

• Merged the dencun tests from upstream
• Will work with Egor to catch up as fast as possible

Devnet

• We have a Devnet with Nethermind, Teku and Lodestar
  • Lighthouse doesn’t work for custom configs anymore, Carlos is in communication with them
• There are some issues, probably related to peering
• Also deployed the blob scanner and spammer

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

September 27, 2023

EL​

• Nethermind: Exploring Shutter and waiting for some answers Probably going to release a new version of Nethermind end of this week

• Erigon: No updates

• Geth: Found the previous problem thanks to Somnath The contracts weren’t properly imported, but the logic works correctly Was able to sync 145k blocks After this, one transaction has an invalid gas usage (more expensive on geth) https://gnosisscan.io/tx/0x05a1fdb3888007224b13b425e7c2442b3f462919a52f8785b857c15dd255d5d2 Comparing traces as we speak Has to rebuild the current state (~14 hours) Needs access to an S3 bucket in order not to have to sync from scratch constantly Implemented the withdrawal contract to be able to go through the transition once it gets there

Chain Infra

• Gateway No one present

Dencun

• Devnets
  • Empty system address? We’ll try to send a few wei to the system account on Chiado first
  • Devnets Carlos is preparing configurations for the devnet Should be ready for end of this week or start of the next one Working on the max churn limit

Hive Tests

Started working on the dencun tests As soon as tests are ready, they’ll be shown in the dashboard https://hive-gno.nethermind.io/ Work on upstreaming will be done after dencun tests are implemented Somnath can help for that Will speak with Egor about adding older Hive tests

Research

• AuRa contracts Should be cleaned up Jorge was exploring this topic but gave up Marek will ask him about his opinion
• What about BEACON_ROOTS_ADDRESS? Will we have the same address or a different one? We should be able to create it at the same address Worst case we can just hardcode it in the genesis file (ideally CREATE2 though) Guillaume asked Matt Might rely on a dependency on EF, in which case we can just make the address configurable and deploy it with our own keys
• Are legacy transactions disabled? Might still be enabled in Erigon, but not in Geth and Nethermind? Andrew will check Geth disabled legacy transactions (without chain id) in the transaction pool, but still accepts them in the block

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

September 20, 2023

EL​

• Nethermind: Did some work for the optional fast sync protocol (to fix the eth/66 requirement and enable eth/68) This would be a temporary workaround Might be able to deliver the new layout + snap server before dencun Made contact with Shutter New release planned for next week Will include the RPC fix (Gnosis and Gateway are using custom images right now)

• Erigon: Root node dump for Guillaume? Somnath is on it and syncing Gnosis mainnet Mostly done with Dencun, probably some small things left Somnath has been working on improving the transaction pool for blob transactions

• Geth: Tanishq is going to generate traces from the execution of the reward contract to compare them (Nethermind vs Geth) Can’t sync anymore for some reason and is getting a lot of timeouts Waiting for Somnath’s dumps

Chain Infra

• Gateway No one present

Dencun

• Devnets
  • We’re going to launch a Gnosis Dencun devnet at the same time as devnet 9
    • So we can get the most features included
    • Probably next week or within 10 days
    • Still some issues, but worse case we can launch a bit later
• Empty system address
  • Marek thinks that it’s empty
  • Andrew proposes to remove the system address on the first dencun block if it is empty
    • To be more in line with Ethereum
    • Marek needs to think about this
    • The downside is that it makes the logic more complicated
• System address -> no system address
  • Different AuRa systems that have different properties and the code still needs to be maintained
  • We still have to support it for archive Gnosis
  • Tricky solution: send a few wei to the system account

Hive Tests

No updates Andrew would like to talk about upstreaming to the Ethereum repo We should talk about this when Marcos is back

Research

• Max churn limit
  • We can set the max value right now, which solves most of the issues
• Big block experiment
  • Clickhouse is now working and we have the data
  • Adapted a Jupyter notebook to handle the data
  • https://colab.research.google.com/drive/1XAnZm4fSboSh8WFb4VujNMwqon7fdwqG?authuser=1
• System transaction differences

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

September 6, 2023

EL​

• Nethermind: Met the Shutter guys There were some questions and answers Asked for priorities

  1. Cancun We want EIP-4844 as soon as possible
  2. Shutter

• Erigon: No updates

• Geth: Got some Chiado data but it was missing the code, so waiting for a new dump Requesting more details about the system address being or not being in the state Guillaume will contact Lukasz on Telegram EIP-158 needs to be disabled for system transactions

Chain Infra

• Gateway No updates

Mainnet

Deposit UI somewhat fixed The previous deposits cache is now kept up to date on every build But we’re still getting RPC issues a lot of the times, making deposits quite painful The RPC error wa actually fixed by Nethermind, but not released There was a custom release for Gateway Auto claim Some sample contracts were written but there’s nothing concrete

Dencun

We should start launching devnets There will be some differences for shadow forks as well We need a copy of the deposit / withdrawals contract, so that the shadow forked network doesn’t have all validators deposited We just have to deploy the deposit contract on Gnosis mainnet Will require some changes in Nethermind and Erigon to support deposit contract swaps

Hive Tests

No updates

Research

• Big blocks experiment What size should we allocate to 4844? Just parameter changes
• Cancun: there are definitely some differences
  • 4844 requires eth/68, but we don’t have that because we don’t have snap sync
    • Nethermind has a client for snap sync but doesn’t serve that data
    • Erigon also doesn’t serve the data
    • A clean solution might be to introduce a new way to sync data?
  • System transaction differences between Gnosis and mainnet for 4337
    • Erigon would prefer to do all the system transactions the same way for Gnosis, so use the current concept for Gnosis and a different one for Ethereum
    • There are some downsides
  • Hive tests will have to be different
  • Lion: why not keep the same code as for Ethereum? We already have hacks for AuRa, and we would have to add hacks to the new system transaction way as well

Next week DappCon

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

EL​

• Nethermind: A new state layout is being worked on, which will optimize block processing significantly 80% to 20% of the block processing time Will introduce snapshots with this update or soon after Will allow for snap sync for Gnosis Chain Might ship end of year, but no guarantee Geth has been working on something similar for 2 years and will be shipping soon

• Erigon: No updates

• Geth: Guillaume got pre-shapella and post-shapella states He’s going to implement the hard fork and see if he gets the right post-state after A Nethermind archive node was set up for Guillaume so he can trace and get states

CL​

• Lighthouse Merged the Chiado config

Chain Infra

• Gateway No updates

Mainnet

Auto claim

Hive Tests

Fixed the last issue for Erigon Related to warm coinbase Working on merging dencun tests Will likely break some tests, but that makes sense

Research

• Big block experiment We’re going to deliberately spam the network with big blocks to see if it causes issues Latency Slower slots / propagation Less attestations / falling behind
• History of Gnosis Chain, specifically regarding spamming and attacks Started by implementing AuRa in 2019 and then POSDAO Not much recollection of the different attacks It would be quite cheap to fill blocks, and having somewhat full blocks could cause a lot of nodes to struggle Right now we’re at 5-10% of block capacity It should still be in spec, be it for block producing or propagation Block production should stay under 150ms Might be a bit more expensive for node operators, but that makes sense Lion’s concern: we now have a very decentralized network, where users might not have sufficiently powerful systems or network connections. Could this introduce risks? Of course, some users with under specced systems could be left behind, but that shouldn’t be an issue For the cheap spamming issue, we had talks about increasing the minimum base fee We were quite against this though, as the costs should scale automatically based on needs

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

August 16, 2023

EL​

• Nethermind: Nethermind Started developing the shutter implementation

• Erigon: Finishing bits and pieces of dencun

• Geth: Pretty much at a dead end at this point Still getting the wrong root and started making experiments Confirmed that the initial state root is correct Confirmed that updates to the state are correct Alternatives explored Test on Chiado because the state is way smaller Getting two consecutive states from Chiado, which archive nodes can’t easily do Call with Andrew tomorrow

Chain Infra

• Gateway Missing transactions on Omni Bridge and AMB Seems related to a container that is constantly restarting Working on this today

Mainnet

Automatic withdrawals Not much progress

Hive Tests

Progress with fixing them for different Nethermind versions 1.19.0 to 1.19.3 seem to work, next ones don’t The first post-shapella block can’t sync Failing withdrawal test Merging latest changes from hive upstream

Research

Shutterized Beacon Chain Potential competitor: https://writings.flashbots.net/the-future-of-mev-is-suave/

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

August 9, 2023

EL​

• Nethermind: No updates

• Erigon: No updates Busy with dencun Looking into Geth issues with Guillaume

• Geth: There’s an issue with a specific transaction Fee collection seems to work correctly If the reward contract writes anything to the state it might cause issues The block root is still incorrect Could be linked to incorrect import of all the previous data Something could be missing (from AuRa?) The reward contract could write something to the state, thus changing the root

CL​

• Lighthouse Fixed some testing related issue for Chiado https://github.com/sigp/lighthouse/pull/4530/commits/b35c1dcb71239bb8f074dd0994d6a39191e07133

Chain Infra

• Gateway Launchpad tested on mainnet

Mainnet

Checkpoint sync fixed Fairly janky though Lot of dependencies to update, fork and maintain Automatic claims We’re likely going with an opt-in solution with signature to confirm If spamming become an issue (as anyone can claim for any address) we can update the deposit contract to require a signature

Hive Tests

No updates

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

August 2, 2023

EL​

• Nethermind: No particular updates Looking into the contractAddress issue in receipts Has some ideas about improvements for eth_getLogs

• Erigon: No specific news Busy working on dencun Helping Marcos with the Hive tests

• Geth: No updates Waiting for input from Erigon Traces from Erigon

CL​

• Nimbus Merged Chiado

• Lighthouse Accepted the PR for Chiado

Chain Infra

• Gateway Post mortem for the slashing issue Looking into the checkpoint sync logs Launchpad updated? Checking with DevOps

Mainnet

Shapella went through smoothly Checkpoint sync Investigation needed Forgot to check that web3signer was updated Was discovered less than 24 hours before the fork Had to put out urgent comms to update this on Dappnode and for users who need web3signer in their infra (i.e. StakeWise) Participation dropped from 96 to 90% for a few epochs but went up to 95% very quickly right after Went really well Erigon nodes weren’t synced for the hard fork (forkmon, ethstats, grafana)

Hive Tests

From the 20 tests passing on Nethermind, only 14 tests pass for Erigon right now