Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

March 20, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing in particular
  • Starting to think about the next release
    • Might potentially include half-path and a snap sync server
    • Expected in a few weeks
  • Progressive on paprika as well, which can’t fast sync and would thus require snap sync servers

• Erigon:

  • Going to make a new release with post-dencun fixes today
  • Includes a feature for more aggressive pruning of logs and receipts
    • Will only be enabled for new nodes (i.e. synced from scratch)
    • Existing nodes will have to resync
  • Transaction logs are filling the network quite a lot, at least in the last 1-2m blocks
    • This behavior doesn’t happen in Chiado / Sepolia
    • This causes the bloating issue
    • With the default pruning flags, full nodes should be around ~500 GB and archive nodes should be ~1.3 TB
    • One can also use a more aggressive pruning strategy (i.e. last 1k - 5k blocks), which of course would also result in a smaller node. Default is 90k blocks.

• Geth:

  • Not present

Chain Infra

• Gateway
  • Deployed Erigon + Lighthouse on mainnet before Dencun and went fine

Hive Tests

• No updates

Innovation

• Shutter
  • Managed to register the validators
  • Looking into a couple of issues
    • Race condition
    • Libp2p library on MacOS specifically
  • 2 week ETA should still be realistic

Mainnet

• 95% attestations immediately after the hard fork
  • Some issues with the Gateway RPC
  • Might be fixed, Alex will double check

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

March 06, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing in particular
  • Released an experimental version with snap server enabled
    • We can slowly start deploying nodes that are can serve snap sync
    • Enabling the snap server requires a full resync
    • Working on 3 solutions for the state db
      • Path based
      • Half path
      • Paprika
    • Release schedule not defined yet, and will depend on development progress, because paprika will require a new hard fork again after half-path
    • Considered for the next release, but after Dencun for sure
    • Might be a locked behind a flag rather than being a default to start with
    • When snap sync becomes faster than fast sync this feature will be enabled by default

• Erigon:

  • Nothing in particular
    • Looking into why the database is bloating up still

• Geth:

  • Found the issue after London and started syncing again
  • Was missing a location for the free transactions in London
    • At 300k blocks past the last issue (which was on block 11 after London)
  • Next issue will likely be the merge, and after the merge everything has already been tested to work
  • Will then have to rebase for the latest Dencun changes

Chain Infra

• Gateway
  • The RPC nodes with Lighthouse for both Erigon and Nethermind
    • Will likely deliver validators with Erigon this week

Hive Tests

• No updates

Innovation

• Shutter
  • Made some progress on p2p connections
  • Working on integrating the different parts together
  • Will then setup a test validator
  • The first test validator might be ready in 1-2 weeks

Mainnet

• Issue with blob gas being burned instead of added to the fee collector
  • Going to be postponed

Shadow Fork

• Nodes still aren’t done syncing
  • They’re very close to tip but don’t catch up
  • Only Erigon nodes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

February 28, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing in particular
  • Stress tested Chiado with blobs and everything was ok, which is good news

• Erigon:

  • The prune hack is now mainstream
    • The nodes will now have half the disk footprint than they used to
  • There is an issue currently where the database is bloating
    • Reason not found yet
  • Caplin should be doable on Gnosis
    • Will ask Giulio

• Geth:

  • Still stuck on the same block
  • Obtained traces from Erigon and Nethermind
  • Some contract update is not happening currently, but the reason should be found with the traces now

Chain Infra

• Gateway
  • About to deploy Erigon + Lighthouse on mainnet

Hive Tests

• No updates

Innovation

• Shutter
  • Managed to handle the connection between Nethermind and the keyper network with the new C# libp2p library

Shadow Fork

• Most nodes should be updated today

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

February 21, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing in particular
  • Dencun ready release: V1.25.4

• Erigon:

  • Worked on a memory optimization, will be released for devnet 3 later today
  • Issue with blob inclusion in blocks, doesn’t seem straightforward and might take a few days
  • Dencun ready release: 2.58.0

• Geth:

  • Managed to execute 16 blocks since last time
  • Just found the issue: engine configuration
  • Might have more to announce later today
  • Next step is the merge
  • Hasn’t had the time to start snap syncing from Nethermind, but intend to try that as well

CL​

• Lighthouse: v5.0.0
• Teku: v24.2.0
• Nimbus: v24.2.1
• Lodestar: still missing

Chain Infra

• Gateway
  • No one present

Hive Tests

• Updated with upstream
• Working on some fixes for Nethermind tests

Innovation

• Shutter
  • Things started moving faster
  • Shutter has the keyper network up on Chiado
    • Some things to fix, but should be getting there
    • Will allow Nethermind to test the libp2p stuff

Shadow Fork

• No updates, still syncing the nodes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

February 14, 2024

Client Team Updates

EL​

• Nethermind:

  • New release planned this week for Dencun-ready Gnosis

• Erigon:

  • Memory consumption for the RPC filters in the transaction pool is being worked on
    • Question about how to handle filters for blob transactions, as the current version is consuming a lot of RAM
  • New release this week or Monday that will also be Dencun ready

• Geth:

  • Has successfully synced all the way until London (~19m) but then crashed because of a null pointer related to config
    • Guillaume knows where to look
  • Next challenge is the consensus change from POSDAO to Merge
    • Next parts up until Cancun should already be implemented and working
  • Marek mentions that snap syncing is now available in Nethermind and could help for Geth as well
    • It’s in an experimental release

Chain Infra

• Gateway
  • Migrating data centers

Mainnet

• Slot: 14237696 Epoch: 889856 Timestamp: 1710181820 Date: UTC Mon 11/03/2024, 18:30:20

Chiado

• 100% proposals all the time

Hive Tests

• No updates

Innovation

• Shutter
  • Also done with the cryptography part
  • Ready to integrate network and crypto

Shadow Fork

• Syncing the nodes
  • New deposit contract proxy deployed

    • 0xF110ac616a7BBE7b2a2946432BCF36975DCf0C04

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

February 07, 2024

Client Team Updates

EL​

• Nethermind:

  • No particular update
    • Preparing a new release

• Erigon:

  • Working on changes in the blob transaction pool to limit spamming
  • Working on testing their final release before Dencun
  • No other specific updates

• Geth:

  • Guillaume was able to fix the syncing issue from block 1301 and is now syncing
    • Currently at block 1.7m

Chain Infra

• Gateway
  • Working on client diversity for Chiado and mainnet
    • Will have Nimbus + Erigon and Nimbus + Nethermind pairs

Mainnet

• We’re ready for the hard fork with a window of 1-2 months for releases and time for users to update
  • Nethermind and Erigon are committing to it

Chiado

• Issues during the hard fork
  • Erigon: the new pending transaction filter might leak memory (issue #9356), especially for blobs
• Gnosis nodes were not up to date
• One Gateway Erigon node was not up to date for 500 keys, but was fixed immediately
• We should prepare diversity better for the next hard fork
  • EL: We kind of got there, but still more Nethermind nodes than Erigon
  • CL: No validators on Nimbus
    • Needs to be tested before mainnet

Devnet

• We should spin up a devnet for Aura Contract work

Hive Tests

• No updates

Innovation

• Shutter
  • Received some traffic from the Shutter test system
  • Their documentation is a bit out of date, so working on figuring out some things
  • In general there’s some progress

Shadow Fork

• Deploy same contract on mainnet than Chiado (deposit contract proxy)
  • Philippe will prepare it

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 31, 2024

Client Team Updates

EL​

• Nethermind:

  • No particular update
    • Planning a shadow fork for mainnet after the Chiado hard fork

• Erigon:

  • No particular update

• Geth:

  • Still at block 1301, but starting to understand what’s going wrong
  • EIP-158 is not active (accounts are being deleted when they shouldn’t, easy fix)
  • Some contracts being called with from or to = 0x0 (system calls?)
  • The transition is being detected and happening

Chain Infra

• Gateway
  • No specific news

Dencun

• Dashboards and metrics for Chiado
  • Nethermind have available dashboard for Chiado
  • Gnosis chain devops dashboard

Chiado

• Keys
  • Nethermind is running 1.5k Nethermind + Teku / Lodestar / Lighthouse keys
  • Gnosis is running 1.5k Nethermind + Lighthouse keys
  • Gateway is running 2665 nodes
    • Nethermind and Lighthouse
• Useful links
  • https://forkmon.chiadochain.net/
  • https://forky.chiadochain.net/
  • https://ethstats.chiadochain.net/
  • https://beacon.chiadochain.net/validators
  • https://checkpoint.chiadochain.net/
  • https://gnosis-chiado.blockscout.com
  • http://170.187.154.203:8080/ (Forkmon)
  • http://170.187.154.203:8084/explore?schemaVersion=1&panes=%7B%22vcF%22:%7B%22datasource%22:%22loki_ds_1%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcontainer_name%3D%5C%22execution%5C%22,%20instance%3D%5C%22chiado-nimbus-0%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki_ds_1%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-5m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1 (Grafana)

Devnet

• Same issues as last week
  • Just Erigon + Nimbus issue

Hive Tests

• Fixed withdrawal tests for Erigon

Innovation

• Shutter
  • Working on the libp2p library because of some issues
  • Cryptography is largely working now, details to be figured out

Shadow Fork

• Successfully ran a shadow fork for Chiado

• Fixed the beacon genesis tool

• Was able to go through the Dencun hard fork

  • Blobs work

• Tested with Nethermind and Erigon

• All seems well!

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 24, 2024

Client Team Updates

EL​

• Nethermind:

  • No updates

• Erigon:

  • New release out today(with some fixes)

• Geth:

  • Blocked at block 1301
  • Validator transition at block 1300 issue as transition contract activation takes place on 1301 height
  • Merkle roots is not correct, but reward execution probably correct, figuring out that
  • Dencun brings a lot of changes to Geth, so Guillaume prioritize to not use very hacky way so that he don't have to start over after the deancun

Shadow Fork

• Tested creating shadow fork and had some issue with genesis
  • Were not able to create it properly, Lion will help

Chain Infra

• Gateway
  • No specific news

Dencun

• Dashboards and metrics for Chiado
  • Nethermind have available dashboard for Chiado
  • Gnosis chain devops dashboard

Hive Test

• No updates

Devnet

• Some issues with Erigon
  • Low transactions count issue related to Erigon specifics, but those limit was increased with Somnath help
  • Was undetected bug earlier in a tx pool, PR with fix pushed
  • Have missing slots with erigon+lighthouse and erigon+nimbus
• Nimbus works with Nethermind without any issues
• High memory usage on CL clients
• Erigon new version should be available later today

Innovation

• Shutter
  • Docker image was published

    • Ruben had a look and communicate with shutter

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 17, 2024

Client Team Updates

EL​

• Nethermind:

  • Released a bug fix version that is also Chiado ready (v1.25.1)
    • There might still be another release before Chiado

• Erigon:

  • Philippe still hasn’t tested the pruning image
  • Will do a new release this week for Chiado

• Geth:

  • Not present

Shadow Fork

• Deployed a new proxy to the deposit contract implementation
  • 0x4A9Eaee643070823afC7477bC0910585E5855e19
• Carlos already started some nodes, so we’re moving into the shadow fork slowly but surely
  • Potentially ready this week, but will require a custom Erigon image
  • Would be nice to fix erigon init command so that we don’t require custom images constantly for devnets / shadow forks etc
    • There’s another linked issue to this so not straightforward to solve

Chain Infra

• Gateway
  • No specific news

Dencun

• Chiado
  • Everything should be ready
  • No concerns at this point

Hive Test

• Fixed the actions that were failing, which were due to misconfiguration in the clients
• Erigon is still missing 3 tests that should not be failing
• Building a tool to push hive test logs to Telegram every day

Devnet

• Issues with Nimbus
  • Wasn’t including the KZG ceremony for Gnosis
    • Fixed in 7443a4ac086000b1d2a7d5af6685cc5ef9537b9d
  • Still forks off
• Issues with Erigon
  • Creates empty blocks (or at least with less transactions)
    • There might be a bug in the transaction pool
    • Some recent commits fix different issues related to overflowing in relation to warning logs from the failing nodes
    • Pending PR for a mdbx race condition
    • Somnath will update the image for devnet 3
    • Somnath will also work on another suspected issue
• Issues with Caplin (experimental) are expected
• Erigon also has occasional issues with block production on Ethereum mainnet
  • Their transaction pool likely needs some love
• Carlos will check if clearing the txpool and restarting the node solves the issue for some time
• Goerli’s devnet is probably mostly Geth + Nethermind so maybe the block production bug for Erigon hasn’t been caught there
• Somnath will release a new version later today when the PRs are approved
• Some issue with Erigon + Lighthouse
  • Lighthouse lost its peers, restarted the node and now it works again

Innovation

• Shutter
  • Still no Docker image for testing
    • Promised one for end of this week
    • Probably busy with their DAO launch
    • Required to work on the networking layer, so it’s blocking the Nethermind implementation significantly
    • The block building and cryptography parts can be worked on currently at least

      • Lot of things to implement for cryptography and the Nethermind team is discussing with Shutter to change some cryptography protocols (BLS12-381 vs BN254)

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 10, 2024

Client Team Updates

EL​

• Nethermind:

  • Shutter
    • Working on Shutter’s cryptographic implementation
    • Still no Docker image for the testnet

• Released 1.25.0 with performance improvements, geth compatible tracers and JSON-RPC improvements (was a preview last week)

• Erigon:

  • We need to test the image for aggressive pruning provided by Somnath
  • Checked the logs for Nimbus + Erigon missed slots for devnet 3
    • Could not find a reason why the payloads would be invalid
    • The sequence of requests for payloads might not be optimal on Nimbus’ side (some type of race condition?)
    • There might be a bug in the transaction pool for Erigon during block building
      • Discovered because of a Hive test failure on Ethereum
      • Somnath is looking into it
  • Released a new version

• Geth:

  • Not present
  • There’s still the issue about the synced state not being considered as the canonical one
  • Guillaume is looking into implementing full sync with AuRa

Shadow Fork

• We want to shadow fork Chiado to make sure that everything works
• We need a copy of the deposit contract deployed on Chiado
  • I.e. just a new proxy that points to the same implementation
    • https://gnosis-chiado.blockscout.com/address/0xb97036A26259B7147018913bD58a774cf91acf25

Chain Infra

• Gateway
  • updated a couple of nethermind nodes to nethermindeth/nethermind:release-1.25.0-rc,

Dencun

• Chiado
  • Hard fork scheduled for
    • Epoch: 516608 Slot: 8265728 Timestamp: 1706724940 Date: UTC Wed 31/01/2024, 18:15:40
    • Chosen to align with the historical roots
      • For example era files use this
      • Ethereum does it as well
  • Dependencies
    • 4788: https://gnosis-chiado.blockscout.com/address/0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02

Hive Test

• Fixed the actions that were failing, which were due to misconfiguration in the clients
• Erigon is still missing 3 tests that should not be failing
• Building a tool to push hive test logs to Telegram every day

Devnet

• Running devnet 3 that started this week
  • Overall the network is healthier than devnet 2
• Two issues
  • Nimbus + Erigon pair, Somnath is looking at it currently
    • Other Erigon pairs seem to work fine
  • Blob fuzzing script has issues