Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

February 14, 2024

Client Team Updates

EL​

• Nethermind:

  • New release planned this week for Dencun-ready Gnosis

• Erigon:

  • Memory consumption for the RPC filters in the transaction pool is being worked on
    • Question about how to handle filters for blob transactions, as the current version is consuming a lot of RAM
  • New release this week or Monday that will also be Dencun ready

• Geth:

  • Has successfully synced all the way until London (~19m) but then crashed because of a null pointer related to config
    • Guillaume knows where to look
  • Next challenge is the consensus change from POSDAO to Merge
    • Next parts up until Cancun should already be implemented and working
  • Marek mentions that snap syncing is now available in Nethermind and could help for Geth as well
    • It’s in an experimental release

Chain Infra

• Gateway
  • Migrating data centers

Mainnet

• Slot: 14237696 Epoch: 889856 Timestamp: 1710181820 Date: UTC Mon 11/03/2024, 18:30:20

Chiado

• 100% proposals all the time

Hive Tests

• No updates

Innovation

• Shutter
  • Also done with the cryptography part
  • Ready to integrate network and crypto

Shadow Fork

• Syncing the nodes
  • New deposit contract proxy deployed

    • 0xF110ac616a7BBE7b2a2946432BCF36975DCf0C04

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

February 07, 2024

Client Team Updates

EL​

• Nethermind:

  • No particular update
    • Preparing a new release

• Erigon:

  • Working on changes in the blob transaction pool to limit spamming
  • Working on testing their final release before Dencun
  • No other specific updates

• Geth:

  • Guillaume was able to fix the syncing issue from block 1301 and is now syncing
    • Currently at block 1.7m

Chain Infra

• Gateway
  • Working on client diversity for Chiado and mainnet
    • Will have Nimbus + Erigon and Nimbus + Nethermind pairs

Mainnet

• We’re ready for the hard fork with a window of 1-2 months for releases and time for users to update
  • Nethermind and Erigon are committing to it

Chiado

• Issues during the hard fork
  • Erigon: the new pending transaction filter might leak memory (issue #9356), especially for blobs
• Gnosis nodes were not up to date
• One Gateway Erigon node was not up to date for 500 keys, but was fixed immediately
• We should prepare diversity better for the next hard fork
  • EL: We kind of got there, but still more Nethermind nodes than Erigon
  • CL: No validators on Nimbus
    • Needs to be tested before mainnet

Devnet

• We should spin up a devnet for Aura Contract work

Hive Tests

• No updates

Innovation

• Shutter
  • Received some traffic from the Shutter test system
  • Their documentation is a bit out of date, so working on figuring out some things
  • In general there’s some progress

Shadow Fork

• Deploy same contract on mainnet than Chiado (deposit contract proxy)
  • Philippe will prepare it

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 31, 2024

Client Team Updates

EL​

• Nethermind:

  • No particular update
    • Planning a shadow fork for mainnet after the Chiado hard fork

• Erigon:

  • No particular update

• Geth:

  • Still at block 1301, but starting to understand what’s going wrong
  • EIP-158 is not active (accounts are being deleted when they shouldn’t, easy fix)
  • Some contracts being called with from or to = 0x0 (system calls?)
  • The transition is being detected and happening

Chain Infra

• Gateway
  • No specific news

Dencun

• Dashboards and metrics for Chiado
  • Nethermind have available dashboard for Chiado
  • Gnosis chain devops dashboard

Chiado

• Keys
  • Nethermind is running 1.5k Nethermind + Teku / Lodestar / Lighthouse keys
  • Gnosis is running 1.5k Nethermind + Lighthouse keys
  • Gateway is running 2665 nodes
    • Nethermind and Lighthouse
• Useful links
  • https://forkmon.chiadochain.net/
  • https://forky.chiadochain.net/
  • https://ethstats.chiadochain.net/
  • https://beacon.chiadochain.net/validators
  • https://checkpoint.chiadochain.net/
  • https://gnosis-chiado.blockscout.com
  • http://170.187.154.203:8080/ (Forkmon)
  • http://170.187.154.203:8084/explore?schemaVersion=1&panes=%7B%22vcF%22:%7B%22datasource%22:%22loki_ds_1%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcontainer_name%3D%5C%22execution%5C%22,%20instance%3D%5C%22chiado-nimbus-0%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki_ds_1%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-5m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1 (Grafana)

Devnet

• Same issues as last week
  • Just Erigon + Nimbus issue

Hive Tests

• Fixed withdrawal tests for Erigon

Innovation

• Shutter
  • Working on the libp2p library because of some issues
  • Cryptography is largely working now, details to be figured out

Shadow Fork

• Successfully ran a shadow fork for Chiado

• Fixed the beacon genesis tool

• Was able to go through the Dencun hard fork

  • Blobs work

• Tested with Nethermind and Erigon

• All seems well!

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 24, 2024

Client Team Updates

EL​

• Nethermind:

  • No updates

• Erigon:

  • New release out today(with some fixes)

• Geth:

  • Blocked at block 1301
  • Validator transition at block 1300 issue as transition contract activation takes place on 1301 height
  • Merkle roots is not correct, but reward execution probably correct, figuring out that
  • Dencun brings a lot of changes to Geth, so Guillaume prioritize to not use very hacky way so that he don't have to start over after the deancun

Shadow Fork

• Tested creating shadow fork and had some issue with genesis
  • Were not able to create it properly, Lion will help

Chain Infra

• Gateway
  • No specific news

Dencun

• Dashboards and metrics for Chiado
  • Nethermind have available dashboard for Chiado
  • Gnosis chain devops dashboard

Hive Test

• No updates

Devnet

• Some issues with Erigon
  • Low transactions count issue related to Erigon specifics, but those limit was increased with Somnath help
  • Was undetected bug earlier in a tx pool, PR with fix pushed
  • Have missing slots with erigon+lighthouse and erigon+nimbus
• Nimbus works with Nethermind without any issues
• High memory usage on CL clients
• Erigon new version should be available later today

Innovation

• Shutter
  • Docker image was published

    • Ruben had a look and communicate with shutter

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 17, 2024

Client Team Updates

EL​

• Nethermind:

  • Released a bug fix version that is also Chiado ready (v1.25.1)
    • There might still be another release before Chiado

• Erigon:

  • Philippe still hasn’t tested the pruning image
  • Will do a new release this week for Chiado

• Geth:

  • Not present

Shadow Fork

• Deployed a new proxy to the deposit contract implementation
  • 0x4A9Eaee643070823afC7477bC0910585E5855e19
• Carlos already started some nodes, so we’re moving into the shadow fork slowly but surely
  • Potentially ready this week, but will require a custom Erigon image
  • Would be nice to fix erigon init command so that we don’t require custom images constantly for devnets / shadow forks etc
    • There’s another linked issue to this so not straightforward to solve

Chain Infra

• Gateway
  • No specific news

Dencun

• Chiado
  • Everything should be ready
  • No concerns at this point

Hive Test

• Fixed the actions that were failing, which were due to misconfiguration in the clients
• Erigon is still missing 3 tests that should not be failing
• Building a tool to push hive test logs to Telegram every day

Devnet

• Issues with Nimbus
  • Wasn’t including the KZG ceremony for Gnosis
    • Fixed in 7443a4ac086000b1d2a7d5af6685cc5ef9537b9d
  • Still forks off
• Issues with Erigon
  • Creates empty blocks (or at least with less transactions)
    • There might be a bug in the transaction pool
    • Some recent commits fix different issues related to overflowing in relation to warning logs from the failing nodes
    • Pending PR for a mdbx race condition
    • Somnath will update the image for devnet 3
    • Somnath will also work on another suspected issue
• Issues with Caplin (experimental) are expected
• Erigon also has occasional issues with block production on Ethereum mainnet
  • Their transaction pool likely needs some love
• Carlos will check if clearing the txpool and restarting the node solves the issue for some time
• Goerli’s devnet is probably mostly Geth + Nethermind so maybe the block production bug for Erigon hasn’t been caught there
• Somnath will release a new version later today when the PRs are approved
• Some issue with Erigon + Lighthouse
  • Lighthouse lost its peers, restarted the node and now it works again

Innovation

• Shutter
  • Still no Docker image for testing
    • Promised one for end of this week
    • Probably busy with their DAO launch
    • Required to work on the networking layer, so it’s blocking the Nethermind implementation significantly
    • The block building and cryptography parts can be worked on currently at least

      • Lot of things to implement for cryptography and the Nethermind team is discussing with Shutter to change some cryptography protocols (BLS12-381 vs BN254)

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 10, 2024

Client Team Updates

EL​

• Nethermind:

  • Shutter
    • Working on Shutter’s cryptographic implementation
    • Still no Docker image for the testnet

• Released 1.25.0 with performance improvements, geth compatible tracers and JSON-RPC improvements (was a preview last week)

• Erigon:

  • We need to test the image for aggressive pruning provided by Somnath
  • Checked the logs for Nimbus + Erigon missed slots for devnet 3
    • Could not find a reason why the payloads would be invalid
    • The sequence of requests for payloads might not be optimal on Nimbus’ side (some type of race condition?)
    • There might be a bug in the transaction pool for Erigon during block building
      • Discovered because of a Hive test failure on Ethereum
      • Somnath is looking into it
  • Released a new version

• Geth:

  • Not present
  • There’s still the issue about the synced state not being considered as the canonical one
  • Guillaume is looking into implementing full sync with AuRa

Shadow Fork

• We want to shadow fork Chiado to make sure that everything works
• We need a copy of the deposit contract deployed on Chiado
  • I.e. just a new proxy that points to the same implementation
    • https://gnosis-chiado.blockscout.com/address/0xb97036A26259B7147018913bD58a774cf91acf25

Chain Infra

• Gateway
  • updated a couple of nethermind nodes to nethermindeth/nethermind:release-1.25.0-rc,

Dencun

• Chiado
  • Hard fork scheduled for
    • Epoch: 516608 Slot: 8265728 Timestamp: 1706724940 Date: UTC Wed 31/01/2024, 18:15:40
    • Chosen to align with the historical roots
      • For example era files use this
      • Ethereum does it as well
  • Dependencies
    • 4788: https://gnosis-chiado.blockscout.com/address/0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02

Hive Test

• Fixed the actions that were failing, which were due to misconfiguration in the clients
• Erigon is still missing 3 tests that should not be failing
• Building a tool to push hive test logs to Telegram every day

Devnet

• Running devnet 3 that started this week
  • Overall the network is healthier than devnet 2
• Two issues
  • Nimbus + Erigon pair, Somnath is looking at it currently
    • Other Erigon pairs seem to work fine
  • Blob fuzzing script has issues

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 03, 2024

Client Team Updates

EL​

• Nethermind:

  • No specific updates
  • Preparing a new release with performance improvements
    • RC this week
    • Next week full release
  • One more release with experimental state db changes with large improvements
    • Initial tests were very promising

• Erigon:

  • Testing change to make validator nodes lighter (better pruning, i.e. just keep receipts for the deposit contract)
  • Somnath will work on a release so we can test it on the devel branch

• Geth:

  • Not present

Chain Infra

• Gateway
  • No specific updates
  • Can test new Nethermind releases, just ping Anton on Telegram
    • nethermindeth/nethermind:release-1.25.0

Dencun

• Chiado
  • Tentative date around end of January, ideally Jan 31st (during the core dev call)
  • Around the same time as Sepolia
  • Lion will run the script to find optimal epochs (right time and epoch boundaries)

Hive Test

• Migrated more tests for Cancun, but in the process broke withdrawal tests
  • These are being worked on currently

Devnet

• Solved Nethermind related issues
• Some issues with Erigon-Teku
• We had one node missing slots (Erigon + Teku)
  • Technically one forked, but it’s likely because it was out of sync with the network and then produced a block itself
• We need to add Lighthouse (unstable) and Nimbus (v23.11.0)
• We need to observe the network a bit more
• How are Teku nodes configured currently, as the preset isn’t set up in the code base yet?

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

December 20, 2023

Client Team Updates

EL​

• Nethermind:

  • Communications with the Shutter team
  • New version was released
    • This one isn’t downgradeable
  • New version early next year with the JSON-RPC improvements
    • Much better processing without state redesign as well

• Erigon:

  • No updates
  • Was not able to reproduce the OOM in Docker issue

• Geth:

  • No updates

Chain Infra

• Gateway
  • No specific updates
  • Nethermind should send over a Docker image with the JSON-RPC changes so that it can be deployed by Gateway and be tested
    • Ideally this week or it will be moved to next year

Hive Tests

• Solved issues with multi-clients from last week
• Merged our branch with the upstream one, which caused issues
  • They are being looked into

Devnet

• Monitoring and investigating issues but no specific results
• Restarting fast-synced nodes didn’t work but was fixed

Innovation

• Shutterized Beacon Chain
  • Nethermind had some questions regarding the transaction database (events or in-contract)
    • Seems like both teams are on the same page about drawbacks and consequences

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

December 13, 2023

Client Team Updates

EL​

• Nethermind:

  • Slowly progressing with Shutter for the parts they can do alone
    • Requested a Docker image from Shutter

• Merged JSON-RPC related changes that should result in much better performance

  • There might still be some issues
  • Nethermind wants to test this upgrade on the official Gnosis RPC
    • We’re going to start with one node and look at the metrics
    • Maybe it’s a bit early to upgrade all nodes
    • Sending traffic there in a shadow mode / replicate traffic to it without using the return data would be useful
    • This is currently in the master branch, not officially released
  • Working on one regression for a cancun related issue

• Working on Nethermind 2, a big state redesign that would include performance a lot that would allow for snap servers

  • Probably going to be released after Cancun

• Erigon:

  • No news in particular
  • Busy working on Caplin and Erigon 3

• Geth:

  • Can’t join today

Chain Infra

• Gateway
  • Saw an increase in traffic and had to scale the infrastructure
    • This helped to reveal minor issues that are being worked on

Hive Tests

• Working on multi client tests for receiving blob transactions at the same time

Devnet

• Still having some general Nethermind-specific issues
  • Sometimes out of sync
  • One node with a specific issue
• No issues with the Erigon nodes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

December 06, 2023

Client Team Updates

EL​

• Nethermind:

  • Released a new version: 1.23.0
  • Includes the configuration change for gas limit (17m)
  • Memory issue fixes
  • Continuing Shutter exploration

• Erigon:

  • Looking into how to improve pruning

• Geth:

  • No update, the synced data is not getting saved to the database
  • Guillaume is still digging

Chain Infra

• Gateway
  • No updates

Hive Tests

• Couple of new tests related to Dencun that will be published tomorrow

Devnet

• Doing well
• No Dencun specific issues
• One node is down all the time, but most likely infra issues