Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Nov 27, 2024

Infrastructure

Gateway​

• Everything’s working well
• Progressing with a Nimbus CL endpoint

Innovation

Shutter​

• Still working on the peer discovery issue

Testing

Hive​

• No updates

Ethereum/tests​

• Working on solving the issues with Nethermind t8n

Client updates

EL​

Nethermind​

• Fixed the most blocking issue for the next release
• Still working on an issue that happens after restarts
  • Difficult to reproduce, fairly rare
  • Seems to happen on very fast machines
• Next release
  • Performance improvements
  • Bug fixes
  • Pectras PRs
  • Shutter in the main branch

Erigon​

• Supports Gnosis Pectra Devnet 1
  • Fix for importing genesis
• Pectra Devnet 4 is live on 2.61.0-beta1
• A v3 version should also work on Gnosis Devnet 1

Geth​

• Finished a full sync on Geth
• Rebased on the latest official release of Geth: 1.14.12
  • Working on a clean PR
  • Able to follow the chain and produce blocks on Chiado
  • Will do a full resync on this latest rebase
• Implemented tracing for system calls
• Cleaned code and removed temporary stuff
• Not sure what Pectra needs to be done on Gnosis
  • https://github.com/gnosischain/specs/blob/master/network-upgrades/pectra.md
• Helped Debjit for Reth
• Missing for a public release
  • The rebase on the right branch
  • Validate everything on the latest rebase with a full sync

Reth​

• Got the post-merge state dump from Geth
• Was able to sync ~70 blocks after importing the state
  • Debugging issues with block execution now, invalid block roots

Research

• Native rollups

  • A debate between Martin and Justin Drake:

• Misc

  • https://eips.ethereum.org/EIPS/eip-7639
    • Generally the consensus is to drop pre-merge history in 6 months
      • Because in 6 months, 2 TB will no longer be sufficient to run a full node
      • Will require support for ERA files, but keep the Portal network optional
        • Nethermind has a PR for ERA files, should be merged soon
          • https://github.com/NethermindEth/nethermind/pull/6547
          • Compatible with both Ethereum and Gnosis
  • We could do the same for Gnosis Chain as a first step
  • Might be problematic for clients like Reth that need to sync the full chain
  • After this, to sync an archive node you’ll need to download and import ERA files manually and the rest would work like it does currently

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Nov 20, 2024

Infrastructure

Gateway​

• Worked on a public RPC for the consensus layer
• Previous version of container deployed by mistake
• Resulted in max 100 batch requests instead of 500
  • Should not happen again

Innovation

Shutter​

• Most of the team was at DevCon
• Working on stability improvements between validators and keepers
• Looking into discoverability algorithms

Testing

Hive​

• Adapted consensus spec tests to Gnosis

Ethereum/tests​

• Discovered the issue on t8ntool but isn’t fixed yet
• Dan and Spencer from the EF joined
  • Want to use EELS/EEST for Gnosis
  • https://notes.ethereum.org/@danceratopz/eest-gnosis
  • This would replace ethereum/tests
  • We would only focus on post-merge tests, as AuRa is too different

Client updates

EL​

Nethermind​

• Working on the next release (delayed because of DevCon)
  • Found critical issues
• Discussing internally about how to do Shutter testing

Erigon​

• No particular updates
• Shutter implementation is going to take 1-2 months

Geth​

• Finished sync up to merge
• 7m blocks to go
• Working on a dump for Debjit on the merge block

Reth​

• Raised some PRs, waiting for review

CL​

Pectra​

Research

• Feedback about Martin’s native rollups
  • Marek thinks that it’s very interesting and would be interested in joining this working group
  • Lukasz has reservations about this idea
    • Mostly social, not technical
    • Ethereum core devs have limited throughput, so decentralizing it through L2s would allow for a faster timeline

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 23, 2024

Infrastructure

Gateway​

• No issues the past week
• Implemented rate limits for batch requests
• In the process of releasing CL endpoints
  • ETA this or next week

Innovation

Shutter​

• Nethermind
  • Engine room release
  • Working on refinements and improvements of the Shutter explorer
  • Testing the latest version on Chiado
    • The stability is looking very good
      • 24 failures out of 8k+
  • Planning for the mainnet release
  • Implementation started by Milen from Erigon
    • Frederik will set a call up

Testing

Hive​

• The execution API suite will get moved
• Some suites rely on t8n

Ethereum/tests​

• Integrating t8n by Nethermind

Client updates

EL​

Nethermind​

• 1.29.1 solves the block creation problem
  • Official release and pushed to Dappnodes etc
  • No missing slots anymore
• Worked on a new Shutter release for 1.29.1, which was released as 1.29.2
• Release 1.30 planned in 1-2 weeks
• Worked with Twinstake
• Should be able to focus more on 44s
• Implemented an endpoint that should allow Nethermind to be used as a block builder

Erigon​

• No particular updates
• Working on devnet 4 and devnet 5
• Will start working on joining Gnosis Devnet 1 as soon as it’s published

Geth​

• This the rebase to the latest 1.14 version (1.14.11)
  • Which should support live tracer
  • Should be ready for Gnosis Devnet 1
    • Missing blob fee collector change
• Snap sync works
• Block building works on mainnet (Gnosis is running 52 validators)
• Still in the process of doing a full sync, but will last at least 2-3 weeks
  • Still pre-merge currently, everything works until there at least

Reth​

• Working on importing the post-merge state from geth or any other client
  • Geth seems to have a command to export the state
    • Guillaume thinks that it might not work
• Waiting for t8ntool

CL​

Pectra​

• Gnosis Devnet 0
  • Based on devnet 3
  • Should probably move to devnet 4 (for Gnosis Devnet 1)
    • Both Nethermind and Erigon agrees, especially because 3 is not compatible with 4 at all

Research

• No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 09, 2024

Infrastructure

Gateway​

• RPC working fine with latest EL versions *Implementing batch rate limits
• Will start working on consensus layer proxy

Innovation

Shutter​

• Nethermind
  • Main Shutter PR has been merged
  • Still working on a few p2p improvements
  • Floodsub has been implemented
  • Peer exchange is being implemented as well
  • Some keys are still not received in time
• Erigon
  • Milen is going to start working on Shutter support

Testing

Hive​

• Focused on CI improvements

Ethereum/tests​

• No updates

Client updates

EL​

Nethermind​

• Testing new release
• Pushing for Pectra
• Talks about gas limit and EIP-4444
• Work on era files for Gnosis will be implemented after 7702

Erigon​

• No updates
• Erigon 2 will support the first fork of Pectra in the end
• Erigon 3 is not yet rock solid stable and the increased RAM requirements might be unfair, especially on Gnosis Chain

Geth​

• Found a bug in the rebase, now syncing the chain
• Will then test block production
• Sync should complete in a few hours

Reth​

• State root mismatch issues after genesis
• Starting on a chain with Shapella but without Dencun
• Only a reward contract call is being made
• Try with 0x0 and system accounts have > 0 balance

CL​

Pectra​

• Potentially looking at Q1 2025

Research

• No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 02, 2024

Client Team Updates

EL​

• Nethermind:

  • Testing a new release:
    • performance improvements, fixes, engine_getBlobs ->
    • might be useful for gnosis validators a little bit who are struggling with bandwidth,
  • A lot of work on Pectra
    • Generally a lot of progress in the performance area

• Erigon:

  • Busy with Erigon 3
    • Small forks for Pectra
    • Forcing to users from Erigon 2 to Erigon 3
    • User have to resync
    • Sync time will significantly reduce
  • General availability Erigon 3
    • Changing UX
    • Config and storing everything will change
    • Trade-off: optimizing RAM
    • Dealing with hive issues
      • will fix it soon

• Geth:

  • Still debugging the rebase

• Reth:

  • Welcome Debjit!
  • Fixed the differences between production and verification

Pectra

• Which path should we follow for Gnosis
  • See Ethereum ACD
  • https://github.com/gnosischain/specs/pull/51
• When should we start testing?
  • Gnosis specific devnet
  • Nethermind: we are starting slowly

Chain Infra

• Gateway
  • Gateway: Created new extended feature batch limit.
    • allowing 500 requests
  • RPC works well with the latest version.

Innovation

• Shutter
  • Finalized engine room. Scheduled next week.
  • Explorer.shutter.network is ready
  • Working on next iteration. Tomorrow will be live
    • New deployment on Chiado
  • Nethermind:
    • peer discovery testing is underway
  • Erigon:
    • Team working on for implementation
    • New team member!

Testing

• Hive
  • Updated new test
• Ethereum/tests:
  • Fix some operational issues
  • Still working on it

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Aug 28, 2024

Client Team Updates

EL​

• Nethermind:

  • Working on the next release

    • Improvement on performance

    • Probably mid next month

• Erigon:

  • Nothing new

• Geth:

  • Started rebasing from 1.13 to 1.14 (which actually changed many things in the miner)

    • 1.14 can no longer sync pre-merge chains

    • So one has to sync with 1.13 and then update to 1.14

• Reth:

  • No update

Chain Infra

• Gateway

  • No major updates

  • Everything is working well except for one quick restart

Innovation

• Shutter

  • Still working on the last 10% in missed transactions

    • New keyper version with floodsub discovery mechanism

  • Also working on the Shutter explorer

Testing

• Hive

  • Good progress

  • Fixed all Nethermind tests

  • 2 new tests suits (13 new tests) for Nethermind

  • Fixing in progress for Erigon

  • New PR with new engine suite

• Ethereum/tests

  • Started to integrate the Nethermind implementation

  • Might require a new repo

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Aug 21, 2024

Client Team Updates

EL​

• Nethermind:

  • Released v1.28.0 (snap sync client enabled by default)
  • Working on several performance improvements for the next version
  • Still focused on Pectra

• Erigon:

  • Failing a lot of tests (even on Erigon) because of a performance hack
  • Working on the next alpha version of Erigon v3 and Pectra

• Geth:

  • Was able to produce blocks on Chiado, but they’re all empty
    • Investigating as to why this might be happening
    • By default, Geth produces an empty block in case there’s not enough time to actually build one
    • Right now it’s sending that empty block despite there being transactions in the mempool
    • Might be related to a blob processing issue

• Reth:

  • Got stuck with a state root mismatch on a devnet
  • Working with Reth to get a dump and Nethermind to get a diff
  • In general, this comes down to a lack of tooling again

Chain Infra

• Gateway
  • No major updates
  • Everything is working well except for one quick restart

Innovation

• Shutter
  • No major updates
  • Continuing work on dashboard and general stability of the network
  • Updated the specs to align with what’s actually deployed
    • Will open a PR soon

Testing

• Hive
  • Can now add new clients faster
  • Geth is being added right now
  • Fixed almost all Nethermind tests
• Ethereum/tests
  • Need someone with good Gnosis understanding to craft the test cases
  • Need a repo for those test cases
  • Clients will have to implement the interface those tests because currently clients are bound to the Ethereum protocol
  • Might work differently internally, we’ll have to see on a per-client basis

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Aug 7, 2024

Client Team Updates

EL​

• Nethermind:

  • RC 1.28.0 https://github.com/NethermindEth/nethermind/releases/tag/1.28.0-rc
  • Would enable snap sync client by default
  • Full release early next week
  • Includes bug fixes and eth_simulate
    • Potentially some block processing improvement
  • Depending on the node: 2-3% or 10-15%
  • Focus on stability
  • Next version should be more performance oriented

• Erigon:

  • Second alpha of Erigon v3
    • Introducing OtterSync, which makes it extremely fast to sync minimal, full and archive nodes
      • Most of the time is spent downloading data
      • A minimal node is like a full node without history (EIP-4444)
        • Should be able to validate
        • Should consume less than 100 GB of SSD
  • Started working on Shutter
  • Automated QA testing for tracking the tip on Erigon v2 and Erigon v3
    • Erigon v3 is likely failing Hive tests on Gnosis
    • There’s an ongoing issue with snapshot availability for Chiado
      • There might not be enough Erigon Chiado nodes

• Geth:

  • Was able to sync Chiado
    • It can produce blocks but they are incorrect
    • Wanted to sync Erigon for Chiado but that’s quite slow and runs out of memory
    • Needs to be able to debug one specific block
  • How can one attach a debugger when a specific block is being executed?

• Reth:

  • Testing post merge implementation with self generated blocks from Nethermind using engine api and feeding them to reth

• Currently stuck with wrong state root on block

• Different state root that might be related to AuRa

  • Is it possible to export state diffs about everything in Erigon? If not, Nethermind is willing to implement some tools to that effect.
    • Erigon has state diffs for transactions and system transactions, but not sure if there’s a tool to consume that data
    • This is really necessary and slowing everything down for Guillaume and Reth on GC as well
    • In Erigon v3 there’s state diffs for virtual transactions at the beginning and end of block

Chain Infra

• Gateway
  • The RPC was offline for a few seconds during the day because of the provider
  • Recently updated one of the nodes

Innovation

• Shutter
  • Working on the dashboard
  • Finished the data collection part
  • Start a Grafana dashboard and web dashboard
  • Debugging remaining issues with a long-running test
    • 90% of the transactions got included immediately
    • Looking into why the remaining 10% are not being included
      • Potentially latency issues
  • Ongoing discussion about p2p discovery because the current one used by the keypers is a bit difficult to implement in the C# library
  • Nethermind is checking if they can implement a simpler version of Kademlia DHT

Testing

• Hive
  • Setting up proper debugging
• Ethereum/tests
  • https://github.com/NethermindEth/nethermind/pull/6833
  • This would allow Nethermind to run the Ethereum tests
  • Looking into how to make this work for Gnosis

Research

• Gas Market
  • No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

July 31, 2024

Client Team Updates

EL​

• Nethermind:

  • Working on new release: v1.28.0 - Next week
    • Will enable snap sync by default for Gnosis Chain
    • Should make it possible to sync Nethermind in ~30 minutes
    • Includes the fix for Chiado, like v1.27.1
  • Not much progress on EIP-4444 yet

• Erigon:

  • Busy working on Pectra
    • Announced an alpha of Erigon v3
    • Still takes around two days to sync on a 32 GB of RAM machine
    • Because the most recent snapshot is one year old
      • Should be updated next week

• Geth:

  • Still debugging what’s happening with Chiado

• Reth:

  • No updates

Chain Infra

• Gateway
  • The mainnet and Chiado configs were not identical
  • Filter queries were disabled on Chiado when they were properly configured in mainnet
  • Now the config is identical

Innovation

• Shutter
  • Working on general stability improvements
    • Was related to latency
    • A new keyper release helped a lot on the p2p side
    • Some slots are still missed
    • Yesterday night, all keypers stopped sending keys, which was fixed this morning
      • By restarting one of the bootnodes, which seemingly unstuck the keypers
    • On the dashboard side
      • No work on the frontend yet
      • Lot of progress on the backend (network observer, collecting data

Testing

• Hive
  • Worked on tools to make debugging of hive tests easier

Research

• No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

July 24, 2024

Client Team Updates

EL​

• Nethermind:

  • Released hotfix version v1.27.1

    • Hot fix for post Aura

    • Enabled pre-warming blocks on Aura and OP chains

    • Could also run v1.26 or lower fine

• Erigon:

  • Released the first alpha version for Erigon 3!
  • All in one EVM node
  • Sync archive node in a matter of hours not days
  • Also the granularity of state history is reduced from the block level to the transaction level
  • Highly recommend that people start playing with it and experimenting

• Geth:

  • 2 issues
    • Block produced are invalid
    • block production is working simply it produces invalid blocks
    • triggered an issue and uncovered a bug in Nethermind
      • Still debugging
  • other problem this one is a bit more difficult to debug
    • Contract creation is fail in everytime
    • Possibility of tracing and system contracts

• Reth:

  • PR to Reth
    • They want to support it
    • current strategy is not to fork the client
    • build on it its extensibility importing
    • Genesis blocks on devnet worked

Chain Infra

• Gateway
  • No updates

Innovation

• Shutter
  • Researching latency
  • Building a dashboard to observe the status of the general system
    • Investigating the performance and stability
    • Way to for users to inspect the transaction lifecycle of their individual transactions
    • Additional information about decryption keys/slot

Testing

• Hive
  • Developed a technical plan for the hive tests
    • final improvements will be divided into five parts and we have already merged part one which includes all the latest upstream
    • working on the second part which will be include genesis refactoring to use the same approach as substream
    • adding new scripts and tools to speed up test development
  • Started working on the ethereum test to migration Gnosis
    • Working on progress

Research

• Nethermind v1.26 issue
  • A bug in Nethermind led to invalid block production; a fix has been implemented.
  • Shutter version is also updated so shutter validators can go back to the v1.27.1 if needed