Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

December 06, 2023

Client Team Updates

EL​

• Nethermind:

  • Released a new version: 1.23.0
  • Includes the configuration change for gas limit (17m)
  • Memory issue fixes
  • Continuing Shutter exploration

• Erigon:

  • Looking into how to improve pruning

• Geth:

  • No update, the synced data is not getting saved to the database
  • Guillaume is still digging

Chain Infra

• Gateway
  • No updates

Hive Tests

• Couple of new tests related to Dencun that will be published tomorrow

Devnet

• Doing well
• No Dencun specific issues
• One node is down all the time, but most likely infra issues

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

November 29, 2023

Client Team Updates

EL​

• Nethermind:

  • No updates

• Erigon:

  • No updates

• Geth:

  • Was able to sync the whole mainnet up to the head block
  • However, none of the progress is getting saved to the database, so restarting the node requires a new sync every time. It also restarts after a full sync.
    • Probably related to the difficulty. Geth seems to expect the difficulty to be the TTD rather than an empty array.
    • This is likely the last issue to fix

Chain Infra

• Gateway
  • worked on enabling eth_getProof on our Gnosis RPC by forwarding it to the Nethermind nodes, at least for certain customers
  • genome (.gno domains) infra is ready, we are waiting for them to complete whatever legalities

Hive Tests

• Having some issues with fees regarding blobs

Devnet

• Added 3 Erigon nodes
  • No issues for now
• Started transaction fuzzers
  • Already had blob generators
• Some reorgs are happening, but all nodes fixed themselves (both Erigon and Nethermind)
• Might have a fix ready for the Nethermind syncing issue but are still testing
• eth/68 issue potentially resolved as well

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

November 22, 2023

Client Team Updates

EL​

• Nethermind:

  • New engineer: Marc
    • Going to explore Shutter with Ruben
  • Noticed 2 bad blocks
    • https://gnosischa.in/slot/12320625 Graffiti: FreeAssangeNow!
    • https://gnosischa.in/slot/12330966 Graffiti: Powered by ⚡Allnodes
  • Had some out of memory issues in edge cases, but those were fixed
    • Not on Gnosis Chain, just Ethereum

• Erigon:

  • Discovered parameter that was causing issues
  • Fixed problem related to OOM, problem for several nodes, incl Gnosis chain
  • An Erigon node is now 700GB with --prune=hrtc, which is around the same size as an archive node
    • Maybe there’s some issue with pruning for Gnosis?

• Geth:

  • Second branch was able to start syncing this morning
  • Might need some help from Nethermind regarding receipt encoding

Chain Infra

• Gateway
  • No one present

Hive Tests

• No update

Devnet

• Erigon updated the configs
• We’re not sure if it’s running because Carlos is not present
• There might be some issue with Nethermind and eth/68
  • Syncing issues
• The chain seems to be progressing

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

November 08, 2023

EL​

• Nethermind:

  • No specific updates
  • Trying to get something ready for Shutter before DevConnect
  • Still seeing bad blocks

• Erigon:

  • No specific updates

• Geth:

  • Working on importing receipts from Nethermind
    • New step required in the Shapella branch
    • Issue with the randomness contract
    • Tanishq is going to provide the data from Nethermind

Chain Infra

• Gateway
  • Provisioning the resolver for .gno

Hive Tests

• Added Erigon for deneb tests
• The current dashboard will include those tests later today

Devnet

• We should spin up a new one
• Erigon has a trie mismatch
• Nethermind nodes might be having issues
  • A new devnet is required anyways to get better logging
• New devnet today
  • Lion will review the config (mostly CL)

Dencun

• Finalized all configuration variables
  • https://github.com/gnosischain/configs/pull/18

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 25, 2023

EL​

• Nethermind:

  • New release with the incident fix and removal of deprecated xdai configs after this call
  • Now has more time to work on Shutter

• Erigon:

  • Maybe the pruning is suboptimal because the node storage size is fairly big
  • Low priority because they’re focused on Erigon 3 which will change storage significantly anyways

• Geth:

  • Kept digging into why the new branch isn’t connecting
  • Only works with eth/66 clients but gets rejected by the other side
  • Asked Tanishq and Somnath to sync nodes to get logs / information

Mainnet

• Mainnet missed epochs: what happened?
  • Nethermind proved that it was possible to build bad blocks with integration tests
    • https://github.com/NethermindEth/nethermind/pull/6212
    • 100% sure that they fixed an edge case
    • Those blocks were built on incorrect state roots
    • There’s a high probability that this was the edge case that happened on mainnet
    • A release for this bug fix is coming very soon
      • This will also include the --config xdai(_archive) removal
    • This bad block situation requires some conditions
      • Specific transactions
      • Specific combination of fork choice request
      • Specific order of events
      • A different transaction pool, restarted Nethermind nodes or Erigon produced blocks could all be reasons to break out of this issue
  • Nethermind is going to go through logs a bit more to confirm
• Lion: Thank you to the Nethermind team for the debugging efforts even throughout the weekend
  • Wants to write a postmortem because it was quite significant of an issue
  • What can we do to improve this type of situation?
    • Monitoring. We’ve already set up
    • Forkly
    • Loki logs in Grafana
    • Grafana metrics for all client combinations
    • Big block experiment and associated dashboards (in ClickHouse)
  • Access to partners / logs
  • We could also add additional non-pruning nodes (not necessarily archive)
    • We already run archive nodes on dev machines where we can SSH and run arbitrary code
  • Add alerts to post on Telegram
    • In case we see bad blocks
    • If we’re not finalizing
  • In this event we learned it too late (from Butta from Beaconcha.in rather than our own alerts)
  • Nethermind could implement a bad block metric
• This issue really highlight the need of client diversity
  • Could moving a large portion of the network to Erigon cause more issues?

Chain Infra

• Gateway
  • Looked at metrics and noticed the issue as well because the RPC had issues
  • Will soon be able to quickly provision rollups on Chiado

Hive Tests

• No updates

Devnet

• New devnet (devnet-1) was expected to be spun up today, but this is going to be postponed, most likely to tomorrow
  • Does this work with the mainnet KZG?
    • Nethermind updated it as well, so it should
  • Somnath is going to work on the configs for Erigon

Research

• Automatic sync tests
  • Lion will check

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 18, 2023

EL​

• Nethermind:

  • Finished some Gnosis-specific features
    • Configurable EIP-4844 configurations
  • Cancun merged to main branch
  • Kickstarted an initiative to rework the eth_getLogs system to optimize it

• Erigon:

  • Sync issues (OOM on 32 GB nodes also running CL)
    • Exclusively happens in Docker, GOMAXPROC should be a workaround (4 instead of 24 for example)
  • High disk usage (400 GB with --prune=hrtc vs 170 GB on Nethermind for the standard gnosis preset)
  • Fairly unreliable (I have to restart it every few days at this point, maybe too many attached validators? I miss quite a few epochs on mainnet too but don’t need to restart at least.)
  • Configurable EIP-4844 configurations are merged
  • Working on Erigon 3

• Geth:

  • Tried the static peers and it seems to work better
  • Still not getting a connection
    • The old branch systematically uses eth/66 and that works
    • The new branch gets disconnected when trying to connect to eth/66

Dencun

• Config
  • Max churn limit of 2
  • Target and max blobs per block
  • Security first: 1/2
    • Thoughts of EL devs?
  • https://github.com/gnosischain/specs/pull/24
  • BLOB_GASPRICE_UPDATE_FRACTION=1112826

Chain Infra

• Gateway
  • Discovered a bug in the load balancer that resulted in 0.5 to 1% of errors at the current load
    • Sudden increase in eth_getLogs, which are typically quite slow on the nodes, which were subsequently falling behind
    • The solution was to account for this slowness

Gnosis

• The public RPC was down during the weekend, but that was because of Gnosis.

Hive Tests

• Solved the configuration issues for Dencun
• Working on an error when sending blobs

Devnet

• Has Erigon joined?
  • thorax/erigon:docker_gno_dencun_devnet_0
  • --chain=gno_dencun_0
  • Issues with the internal CL
  • Not tested yet

Research

• Automatic sync tests

Gnosis Core Devs Call Notes

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams such as:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on YouTube.

October 11, 2023

EL​

Nethermind:

• Working on internal dencun specific things for Gnosis
• Right now 4844 variables are hardcoded in Nethermind and they’re working on making this configurable
• A new release candidate is underway, majorly integrating cancun features along with some fixes.

Erigon: No one present

Geth:

• Not much progress since last week, mostly investigating
• 2 branches
  • One that extracts the data from the Nethermind DB and creates the DB to use by the next branch
    • Figured out why it was running out of memory: memory leak in the transaction pool, but didn’t spend too much debugging because of the new branch
  • One that is more modern and supports withdrawals, but doesn’t currently sync.
    • Has issues connecting to other nodes in the network.

Chain Infra

Gateway No one present

Hive Tests

• Fixed the withdrawal issues after merge.
• Working on the dencun tests.

Devnet

• The devnet nodes have been stable and the network is finalizing
• Lighthouse requires the preset to be built in?
• The blob transaction spammer has some issues, it would be better to try to internal Nethermind tool
• Running with the min gas per blob preset since Friday
• Two more parameters need to be updated, and we need to spin up a new devnet with Erigon from the get go

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

October 4, 2023

EL​

• Nethermind: Working on Shutter No more xdai(_archive) configs in the new Nethermind release (1.21.0) Potential issues in Avado

• Erigon: The header download functionality is acting up a bit from time to time Somnath is working on it Transaction pool should work on Gnosis but hasn’t gone through extensive testing Same for validating

• Geth: Was able to sync 1.5m blocks Geth is now running out of memory on the old branch Did a rebase and is cleaning up some code, but that new branch isn’t syncing

Chain Infra

• Gateway Is running 100% of the RPC traffic of the Gnosis RPC

Dencun

• EIP-4788
  • We believe that the HISTORY_BUFFER_LENGTH variable doesn’t matter that must
    • Having 1.14 days or 0.47 days give roughly the same UX
• We’re going to keep that number and thus deploy the exact same contracts
  • Gnosis: https://gnosis.blockscout.com/address/0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02
  • Chiado: https://gnosis-chiado.blockscout.com/address/0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02

EIP-4844

• Big block experiment
  • The network suffers from big blocks (we spammed it super cheaply for a 10 - 20 minutes)
  • The head participation went down quite a bit (close to 20%)
• Min gas price will be 1 gwei to be on the same scale as normal transactions (1 gwei minimal priority fee)
• Nethermind wrote a tool to spam blobs and it broke a lot of clients
  • These might just have been early implementation bugs
• EIP-7514
  • https://hackmd.io/jWVHqjmrQ1aXtpt6iTNTdw?view
  • An attacker could get a 67% share in the network in 2.5 days because the entry queue scaled way too quickly

Hive Tests

• Merged the dencun tests from upstream
• Will work with Egor to catch up as fast as possible

Devnet

• We have a Devnet with Nethermind, Teku and Lodestar
  • Lighthouse doesn’t work for custom configs anymore, Carlos is in communication with them
• There are some issues, probably related to peering
• Also deployed the blob scanner and spammer

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

September 27, 2023

EL​

• Nethermind: Exploring Shutter and waiting for some answers Probably going to release a new version of Nethermind end of this week

• Erigon: No updates

• Geth: Found the previous problem thanks to Somnath The contracts weren’t properly imported, but the logic works correctly Was able to sync 145k blocks After this, one transaction has an invalid gas usage (more expensive on geth) https://gnosisscan.io/tx/0x05a1fdb3888007224b13b425e7c2442b3f462919a52f8785b857c15dd255d5d2 Comparing traces as we speak Has to rebuild the current state (~14 hours) Needs access to an S3 bucket in order not to have to sync from scratch constantly Implemented the withdrawal contract to be able to go through the transition once it gets there

Chain Infra

• Gateway No one present

Dencun

• Devnets
  • Empty system address? We’ll try to send a few wei to the system account on Chiado first
  • Devnets Carlos is preparing configurations for the devnet Should be ready for end of this week or start of the next one Working on the max churn limit

Hive Tests

Started working on the dencun tests As soon as tests are ready, they’ll be shown in the dashboard https://hive-gno.nethermind.io/ Work on upstreaming will be done after dencun tests are implemented Somnath can help for that Will speak with Egor about adding older Hive tests

Research

• AuRa contracts Should be cleaned up Jorge was exploring this topic but gave up Marek will ask him about his opinion
• What about BEACON_ROOTS_ADDRESS? Will we have the same address or a different one? We should be able to create it at the same address Worst case we can just hardcode it in the genesis file (ideally CREATE2 though) Guillaume asked Matt Might rely on a dependency on EF, in which case we can just make the address configurable and deploy it with our own keys
• Are legacy transactions disabled? Might still be enabled in Erigon, but not in Geth and Nethermind? Andrew will check Geth disabled legacy transactions (without chain id) in the transaction pool, but still accepts them in the block

Gnosis Core Devs Call Notes

Welcome to the weekly Gnosis Core Devs Call, where members of the Gnosis team and contributors gather to discuss the latest developments and updates related to the Gnosis ecosystem. This meeting is held every Wednesday and serves as a platform for participants to collaborate and exchange ideas.

The participants of this call include representatives from various teams such as Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis DevRels, Gnosis Comms team, and community contributors. The presence of diverse team members ensures that discussions are comprehensive and informative, with insights from multiple perspectives.

For those who missed the live meeting, a full recording is available on YouTube.

The Gnosis Core Devs Call is a crucial component of the Gnosis ecosystem. It provides an opportunity for participants to stay informed, share ideas and collaborate, thus driving innovation and progress within the Gnosis community.

September 20, 2023

EL​

• Nethermind: Did some work for the optional fast sync protocol (to fix the eth/66 requirement and enable eth/68) This would be a temporary workaround Might be able to deliver the new layout + snap server before dencun Made contact with Shutter New release planned for next week Will include the RPC fix (Gnosis and Gateway are using custom images right now)

• Erigon: Root node dump for Guillaume? Somnath is on it and syncing Gnosis mainnet Mostly done with Dencun, probably some small things left Somnath has been working on improving the transaction pool for blob transactions

• Geth: Tanishq is going to generate traces from the execution of the reward contract to compare them (Nethermind vs Geth) Can’t sync anymore for some reason and is getting a lot of timeouts Waiting for Somnath’s dumps

Chain Infra

• Gateway No one present

Dencun

• Devnets
  • We’re going to launch a Gnosis Dencun devnet at the same time as devnet 9
    • So we can get the most features included
    • Probably next week or within 10 days
    • Still some issues, but worse case we can launch a bit later
• Empty system address
  • Marek thinks that it’s empty
  • Andrew proposes to remove the system address on the first dencun block if it is empty
    • To be more in line with Ethereum
    • Marek needs to think about this
    • The downside is that it makes the logic more complicated
• System address -> no system address
  • Different AuRa systems that have different properties and the code still needs to be maintained
  • We still have to support it for archive Gnosis
  • Tricky solution: send a few wei to the system account

Hive Tests

No updates Andrew would like to talk about upstreaming to the Ethereum repo We should talk about this when Marcos is back

Research

• Max churn limit
  • We can set the max value right now, which solves most of the issues
• Big block experiment
  • Clickhouse is now working and we have the data
  • Adapted a Jupyter notebook to handle the data
  • https://colab.research.google.com/drive/1XAnZm4fSboSh8WFb4VujNMwqon7fdwqG?authuser=1
• System transaction differences