Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

January 03, 2024

Client Team Updates

EL​

• Nethermind:

  • No specific updates
  • Preparing a new release with performance improvements
    • RC this week
    • Next week full release
  • One more release with experimental state db changes with large improvements
    • Initial tests were very promising

• Erigon:

  • Testing change to make validator nodes lighter (better pruning, i.e. just keep receipts for the deposit contract)
  • Somnath will work on a release so we can test it on the devel branch

• Geth:

  • Not present

Chain Infra

• Gateway
  • No specific updates
  • Can test new Nethermind releases, just ping Anton on Telegram
    • nethermindeth/nethermind:release-1.25.0

Dencun

• Chiado
  • Tentative date around end of January, ideally Jan 31st (during the core dev call)
  • Around the same time as Sepolia
  • Lion will run the script to find optimal epochs (right time and epoch boundaries)

Hive Test

• Migrated more tests for Cancun, but in the process broke withdrawal tests
  • These are being worked on currently

Devnet

• Solved Nethermind related issues
• Some issues with Erigon-Teku
• We had one node missing slots (Erigon + Teku)
  • Technically one forked, but it’s likely because it was out of sync with the network and then produced a block itself
• We need to add Lighthouse (unstable) and Nimbus (v23.11.0)
• We need to observe the network a bit more
• How are Teku nodes configured currently, as the preset isn’t set up in the code base yet?

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

December 20, 2023

Client Team Updates

EL​

• Nethermind:

  • Communications with the Shutter team
  • New version was released
    • This one isn’t downgradeable
  • New version early next year with the JSON-RPC improvements
    • Much better processing without state redesign as well

• Erigon:

  • No updates
  • Was not able to reproduce the OOM in Docker issue

• Geth:

  • No updates

Chain Infra

• Gateway
  • No specific updates
  • Nethermind should send over a Docker image with the JSON-RPC changes so that it can be deployed by Gateway and be tested
    • Ideally this week or it will be moved to next year

Hive Tests

• Solved issues with multi-clients from last week
• Merged our branch with the upstream one, which caused issues
  • They are being looked into

Devnet

• Monitoring and investigating issues but no specific results
• Restarting fast-synced nodes didn’t work but was fixed

Innovation

• Shutterized Beacon Chain
  • Nethermind had some questions regarding the transaction database (events or in-contract)
    • Seems like both teams are on the same page about drawbacks and consequences

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

December 13, 2023

Client Team Updates

EL​

• Nethermind:

  • Slowly progressing with Shutter for the parts they can do alone
    • Requested a Docker image from Shutter

• Merged JSON-RPC related changes that should result in much better performance

  • There might still be some issues
  • Nethermind wants to test this upgrade on the official Gnosis RPC
    • We’re going to start with one node and look at the metrics
    • Maybe it’s a bit early to upgrade all nodes
    • Sending traffic there in a shadow mode / replicate traffic to it without using the return data would be useful
    • This is currently in the master branch, not officially released
  • Working on one regression for a cancun related issue

• Working on Nethermind 2, a big state redesign that would include performance a lot that would allow for snap servers

  • Probably going to be released after Cancun

• Erigon:

  • No news in particular
  • Busy working on Caplin and Erigon 3

• Geth:

  • Can’t join today

Chain Infra

• Gateway
  • Saw an increase in traffic and had to scale the infrastructure
    • This helped to reveal minor issues that are being worked on

Hive Tests

• Working on multi client tests for receiving blob transactions at the same time

Devnet

• Still having some general Nethermind-specific issues
  • Sometimes out of sync
  • One node with a specific issue
• No issues with the Erigon nodes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

December 06, 2023

Client Team Updates

EL​

• Nethermind:

  • Released a new version: 1.23.0
  • Includes the configuration change for gas limit (17m)
  • Memory issue fixes
  • Continuing Shutter exploration

• Erigon:

  • Looking into how to improve pruning

• Geth:

  • No update, the synced data is not getting saved to the database
  • Guillaume is still digging

Chain Infra

• Gateway
  • No updates

Hive Tests

• Couple of new tests related to Dencun that will be published tomorrow

Devnet

• Doing well
• No Dencun specific issues
• One node is down all the time, but most likely infra issues

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

November 29, 2023

Client Team Updates

EL​

• Nethermind:

  • No updates

• Erigon:

  • No updates

• Geth:

  • Was able to sync the whole mainnet up to the head block
  • However, none of the progress is getting saved to the database, so restarting the node requires a new sync every time. It also restarts after a full sync.
    • Probably related to the difficulty. Geth seems to expect the difficulty to be the TTD rather than an empty array.
    • This is likely the last issue to fix

Chain Infra

• Gateway
  • worked on enabling eth_getProof on our Gnosis RPC by forwarding it to the Nethermind nodes, at least for certain customers
  • genome (.gno domains) infra is ready, we are waiting for them to complete whatever legalities

Hive Tests

• Having some issues with fees regarding blobs

Devnet

• Added 3 Erigon nodes
  • No issues for now
• Started transaction fuzzers
  • Already had blob generators
• Some reorgs are happening, but all nodes fixed themselves (both Erigon and Nethermind)
• Might have a fix ready for the Nethermind syncing issue but are still testing
• eth/68 issue potentially resolved as well

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

November 22, 2023

Client Team Updates

EL​

• Nethermind:

  • New engineer: Marc
    • Going to explore Shutter with Ruben
  • Noticed 2 bad blocks
    • https://gnosischa.in/slot/12320625 Graffiti: FreeAssangeNow!
    • https://gnosischa.in/slot/12330966 Graffiti: Powered by ⚡Allnodes
  • Had some out of memory issues in edge cases, but those were fixed
    • Not on Gnosis Chain, just Ethereum

• Erigon:

  • Discovered parameter that was causing issues
  • Fixed problem related to OOM, problem for several nodes, incl Gnosis chain
  • An Erigon node is now 700GB with --prune=hrtc, which is around the same size as an archive node
    • Maybe there’s some issue with pruning for Gnosis?

• Geth:

  • Second branch was able to start syncing this morning
  • Might need some help from Nethermind regarding receipt encoding

Chain Infra

• Gateway
  • No one present

Hive Tests

• No update

Devnet

• Erigon updated the configs
• We’re not sure if it’s running because Carlos is not present
• There might be some issue with Nethermind and eth/68
  • Syncing issues
• The chain seems to be progressing

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

November 08, 2023

EL​

• Nethermind:

  • No specific updates
  • Trying to get something ready for Shutter before DevConnect
  • Still seeing bad blocks

• Erigon:

  • No specific updates

• Geth:

  • Working on importing receipts from Nethermind
    • New step required in the Shapella branch
    • Issue with the randomness contract
    • Tanishq is going to provide the data from Nethermind

Chain Infra

• Gateway
  • Provisioning the resolver for .gno

Hive Tests

• Added Erigon for deneb tests
• The current dashboard will include those tests later today

Devnet

• We should spin up a new one
• Erigon has a trie mismatch
• Nethermind nodes might be having issues
  • A new devnet is required anyways to get better logging
• New devnet today
  • Lion will review the config (mostly CL)

Dencun

• Finalized all configuration variables
  • https://github.com/gnosischain/configs/pull/18

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 25, 2023

EL​

• Nethermind:

  • New release with the incident fix and removal of deprecated xdai configs after this call
  • Now has more time to work on Shutter

• Erigon:

  • Maybe the pruning is suboptimal because the node storage size is fairly big
  • Low priority because they’re focused on Erigon 3 which will change storage significantly anyways

• Geth:

  • Kept digging into why the new branch isn’t connecting
  • Only works with eth/66 clients but gets rejected by the other side
  • Asked Tanishq and Somnath to sync nodes to get logs / information

Mainnet

• Mainnet missed epochs: what happened?
  • Nethermind proved that it was possible to build bad blocks with integration tests
    • https://github.com/NethermindEth/nethermind/pull/6212
    • 100% sure that they fixed an edge case
    • Those blocks were built on incorrect state roots
    • There’s a high probability that this was the edge case that happened on mainnet
    • A release for this bug fix is coming very soon
      • This will also include the --config xdai(_archive) removal
    • This bad block situation requires some conditions
      • Specific transactions
      • Specific combination of fork choice request
      • Specific order of events
      • A different transaction pool, restarted Nethermind nodes or Erigon produced blocks could all be reasons to break out of this issue
  • Nethermind is going to go through logs a bit more to confirm
• Lion: Thank you to the Nethermind team for the debugging efforts even throughout the weekend
  • Wants to write a postmortem because it was quite significant of an issue
  • What can we do to improve this type of situation?
    • Monitoring. We’ve already set up
    • Forkly
    • Loki logs in Grafana
    • Grafana metrics for all client combinations
    • Big block experiment and associated dashboards (in ClickHouse)
  • Access to partners / logs
  • We could also add additional non-pruning nodes (not necessarily archive)
    • We already run archive nodes on dev machines where we can SSH and run arbitrary code
  • Add alerts to post on Telegram
    • In case we see bad blocks
    • If we’re not finalizing
  • In this event we learned it too late (from Butta from Beaconcha.in rather than our own alerts)
  • Nethermind could implement a bad block metric
• This issue really highlight the need of client diversity
  • Could moving a large portion of the network to Erigon cause more issues?

Chain Infra

• Gateway
  • Looked at metrics and noticed the issue as well because the RPC had issues
  • Will soon be able to quickly provision rollups on Chiado

Hive Tests

• No updates

Devnet

• New devnet (devnet-1) was expected to be spun up today, but this is going to be postponed, most likely to tomorrow
  • Does this work with the mainnet KZG?
    • Nethermind updated it as well, so it should
  • Somnath is going to work on the configs for Erigon

Research

• Automatic sync tests
  • Lion will check

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 18, 2023

EL​

• Nethermind:

  • Finished some Gnosis-specific features
    • Configurable EIP-4844 configurations
  • Cancun merged to main branch
  • Kickstarted an initiative to rework the eth_getLogs system to optimize it

• Erigon:

  • Sync issues (OOM on 32 GB nodes also running CL)
    • Exclusively happens in Docker, GOMAXPROC should be a workaround (4 instead of 24 for example)
  • High disk usage (400 GB with --prune=hrtc vs 170 GB on Nethermind for the standard gnosis preset)
  • Fairly unreliable (I have to restart it every few days at this point, maybe too many attached validators? I miss quite a few epochs on mainnet too but don’t need to restart at least.)
  • Configurable EIP-4844 configurations are merged
  • Working on Erigon 3

• Geth:

  • Tried the static peers and it seems to work better
  • Still not getting a connection
    • The old branch systematically uses eth/66 and that works
    • The new branch gets disconnected when trying to connect to eth/66

Dencun

• Config
  • Max churn limit of 2
  • Target and max blobs per block
  • Security first: 1/2
    • Thoughts of EL devs?
  • https://github.com/gnosischain/specs/pull/24
  • BLOB_GASPRICE_UPDATE_FRACTION=1112826

Chain Infra

• Gateway
  • Discovered a bug in the load balancer that resulted in 0.5 to 1% of errors at the current load
    • Sudden increase in eth_getLogs, which are typically quite slow on the nodes, which were subsequently falling behind
    • The solution was to account for this slowness

Gnosis

• The public RPC was down during the weekend, but that was because of Gnosis.

Hive Tests

• Solved the configuration issues for Dencun
• Working on an error when sending blobs

Devnet

• Has Erigon joined?
  • thorax/erigon:docker_gno_dencun_devnet_0
  • --chain=gno_dencun_0
  • Issues with the internal CL
  • Not tested yet

Research

• Automatic sync tests

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams such as:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 11, 2023

EL​

Nethermind:

• Working on internal dencun specific things for Gnosis
• Right now 4844 variables are hardcoded in Nethermind and they’re working on making this configurable
• A new release candidate is underway, majorly integrating cancun features along with some fixes.

Erigon: No one present

Geth:

• Not much progress since last week, mostly investigating
• 2 branches
  • One that extracts the data from the Nethermind DB and creates the DB to use by the next branch
    • Figured out why it was running out of memory: memory leak in the transaction pool, but didn’t spend too much debugging because of the new branch
  • One that is more modern and supports withdrawals, but doesn’t currently sync.
    • Has issues connecting to other nodes in the network.

Chain Infra

Gateway No one present

Hive Tests

• Fixed the withdrawal issues after merge.
• Working on the dencun tests.

Devnet

• The devnet nodes have been stable and the network is finalizing
• Lighthouse requires the preset to be built in?
• The blob transaction spammer has some issues, it would be better to try to internal Nethermind tool
• Running with the min gas per blob preset since Friday
• Two more parameters need to be updated, and we need to spin up a new devnet with Erigon from the get go