Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 23, 2024

Infrastructure

Gateway​

• No issues the past week
• Implemented rate limits for batch requests
• In the process of releasing CL endpoints
  • ETA this or next week

Innovation

Shutter​

• Nethermind
  • Engine room release
  • Working on refinements and improvements of the Shutter explorer
  • Testing the latest version on Chiado
    • The stability is looking very good
      • 24 failures out of 8k+
  • Planning for the mainnet release
  • Implementation started by Milen from Erigon
    • Frederik will set a call up

Testing

Hive​

• The execution API suite will get moved
• Some suites rely on t8n

Ethereum/tests​

• Integrating t8n by Nethermind

Client updates

EL​

Nethermind​

• 1.29.1 solves the block creation problem
  • Official release and pushed to Dappnodes etc
  • No missing slots anymore
• Worked on a new Shutter release for 1.29.1, which was released as 1.29.2
• Release 1.30 planned in 1-2 weeks
• Worked with Twinstake
• Should be able to focus more on 44s
• Implemented an endpoint that should allow Nethermind to be used as a block builder

Erigon​

• No particular updates
• Working on devnet 4 and devnet 5
• Will start working on joining Gnosis Devnet 1 as soon as it’s published

Geth​

• This the rebase to the latest 1.14 version (1.14.11)
  • Which should support live tracer
  • Should be ready for Gnosis Devnet 1
    • Missing blob fee collector change
• Snap sync works
• Block building works on mainnet (Gnosis is running 52 validators)
• Still in the process of doing a full sync, but will last at least 2-3 weeks
  • Still pre-merge currently, everything works until there at least

Reth​

• Working on importing the post-merge state from geth or any other client
  • Geth seems to have a command to export the state
    • Guillaume thinks that it might not work
• Waiting for t8ntool

CL​

Pectra​

• Gnosis Devnet 0
  • Based on devnet 3
  • Should probably move to devnet 4 (for Gnosis Devnet 1)
    • Both Nethermind and Erigon agrees, especially because 3 is not compatible with 4 at all

Research

• No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 09, 2024

Infrastructure

Gateway​

• RPC working fine with latest EL versions *Implementing batch rate limits
• Will start working on consensus layer proxy

Innovation

Shutter​

• Nethermind
  • Main Shutter PR has been merged
  • Still working on a few p2p improvements
  • Floodsub has been implemented
  • Peer exchange is being implemented as well
  • Some keys are still not received in time
• Erigon
  • Milen is going to start working on Shutter support

Testing

Hive​

• Focused on CI improvements

Ethereum/tests​

• No updates

Client updates

EL​

Nethermind​

• Testing new release
• Pushing for Pectra
• Talks about gas limit and EIP-4444
• Work on era files for Gnosis will be implemented after 7702

Erigon​

• No updates
• Erigon 2 will support the first fork of Pectra in the end
• Erigon 3 is not yet rock solid stable and the increased RAM requirements might be unfair, especially on Gnosis Chain

Geth​

• Found a bug in the rebase, now syncing the chain
• Will then test block production
• Sync should complete in a few hours

Reth​

• State root mismatch issues after genesis
• Starting on a chain with Shapella but without Dencun
• Only a reward contract call is being made
• Try with 0x0 and system accounts have > 0 balance

CL​

Pectra​

• Potentially looking at Q1 2025

Research

• No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 02, 2024

Client Team Updates

EL​

• Nethermind:

  • Testing a new release:
    • performance improvements, fixes, engine_getBlobs ->
    • might be useful for gnosis validators a little bit who are struggling with bandwidth,
  • A lot of work on Pectra
    • Generally a lot of progress in the performance area

• Erigon:

  • Busy with Erigon 3
    • Small forks for Pectra
    • Forcing to users from Erigon 2 to Erigon 3
    • User have to resync
    • Sync time will significantly reduce
  • General availability Erigon 3
    • Changing UX
    • Config and storing everything will change
    • Trade-off: optimizing RAM
    • Dealing with hive issues
      • will fix it soon

• Geth:

  • Still debugging the rebase

• Reth:

  • Welcome Debjit!
  • Fixed the differences between production and verification

Pectra

• Which path should we follow for Gnosis
  • See Ethereum ACD
  • https://github.com/gnosischain/specs/pull/51
• When should we start testing?
  • Gnosis specific devnet
  • Nethermind: we are starting slowly

Chain Infra

• Gateway
  • Gateway: Created new extended feature batch limit.
    • allowing 500 requests
  • RPC works well with the latest version.

Innovation

• Shutter
  • Finalized engine room. Scheduled next week.
  • Explorer.shutter.network is ready
  • Working on next iteration. Tomorrow will be live
    • New deployment on Chiado
  • Nethermind:
    • peer discovery testing is underway
  • Erigon:
    • Team working on for implementation
    • New team member!

Testing

• Hive
  • Updated new test
• Ethereum/tests:
  • Fix some operational issues
  • Still working on it

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Aug 28, 2024

Client Team Updates

EL​

• Nethermind:

  • Working on the next release

    • Improvement on performance

    • Probably mid next month

• Erigon:

  • Nothing new

• Geth:

  • Started rebasing from 1.13 to 1.14 (which actually changed many things in the miner)

    • 1.14 can no longer sync pre-merge chains

    • So one has to sync with 1.13 and then update to 1.14

• Reth:

  • No update

Chain Infra

• Gateway

  • No major updates

  • Everything is working well except for one quick restart

Innovation

• Shutter

  • Still working on the last 10% in missed transactions

    • New keyper version with floodsub discovery mechanism

  • Also working on the Shutter explorer

Testing

• Hive

  • Good progress

  • Fixed all Nethermind tests

  • 2 new tests suits (13 new tests) for Nethermind

  • Fixing in progress for Erigon

  • New PR with new engine suite

• Ethereum/tests

  • Started to integrate the Nethermind implementation

  • Might require a new repo

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Aug 21, 2024

Client Team Updates

EL​

• Nethermind:

  • Released v1.28.0 (snap sync client enabled by default)
  • Working on several performance improvements for the next version
  • Still focused on Pectra

• Erigon:

  • Failing a lot of tests (even on Erigon) because of a performance hack
  • Working on the next alpha version of Erigon v3 and Pectra

• Geth:

  • Was able to produce blocks on Chiado, but they’re all empty
    • Investigating as to why this might be happening
    • By default, Geth produces an empty block in case there’s not enough time to actually build one
    • Right now it’s sending that empty block despite there being transactions in the mempool
    • Might be related to a blob processing issue

• Reth:

  • Got stuck with a state root mismatch on a devnet
  • Working with Reth to get a dump and Nethermind to get a diff
  • In general, this comes down to a lack of tooling again

Chain Infra

• Gateway
  • No major updates
  • Everything is working well except for one quick restart

Innovation

• Shutter
  • No major updates
  • Continuing work on dashboard and general stability of the network
  • Updated the specs to align with what’s actually deployed
    • Will open a PR soon

Testing

• Hive
  • Can now add new clients faster
  • Geth is being added right now
  • Fixed almost all Nethermind tests
• Ethereum/tests
  • Need someone with good Gnosis understanding to craft the test cases
  • Need a repo for those test cases
  • Clients will have to implement the interface those tests because currently clients are bound to the Ethereum protocol
  • Might work differently internally, we’ll have to see on a per-client basis

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Aug 7, 2024

Client Team Updates

EL​

• Nethermind:

  • RC 1.28.0 https://github.com/NethermindEth/nethermind/releases/tag/1.28.0-rc
  • Would enable snap sync client by default
  • Full release early next week
  • Includes bug fixes and eth_simulate
    • Potentially some block processing improvement
  • Depending on the node: 2-3% or 10-15%
  • Focus on stability
  • Next version should be more performance oriented

• Erigon:

  • Second alpha of Erigon v3
    • Introducing OtterSync, which makes it extremely fast to sync minimal, full and archive nodes
      • Most of the time is spent downloading data
      • A minimal node is like a full node without history (EIP-4444)
        • Should be able to validate
        • Should consume less than 100 GB of SSD
  • Started working on Shutter
  • Automated QA testing for tracking the tip on Erigon v2 and Erigon v3
    • Erigon v3 is likely failing Hive tests on Gnosis
    • There’s an ongoing issue with snapshot availability for Chiado
      • There might not be enough Erigon Chiado nodes

• Geth:

  • Was able to sync Chiado
    • It can produce blocks but they are incorrect
    • Wanted to sync Erigon for Chiado but that’s quite slow and runs out of memory
    • Needs to be able to debug one specific block
  • How can one attach a debugger when a specific block is being executed?

• Reth:

  • Testing post merge implementation with self generated blocks from Nethermind using engine api and feeding them to reth

• Currently stuck with wrong state root on block

• Different state root that might be related to AuRa

  • Is it possible to export state diffs about everything in Erigon? If not, Nethermind is willing to implement some tools to that effect.
    • Erigon has state diffs for transactions and system transactions, but not sure if there’s a tool to consume that data
    • This is really necessary and slowing everything down for Guillaume and Reth on GC as well
    • In Erigon v3 there’s state diffs for virtual transactions at the beginning and end of block

Chain Infra

• Gateway
  • The RPC was offline for a few seconds during the day because of the provider
  • Recently updated one of the nodes

Innovation

• Shutter
  • Working on the dashboard
  • Finished the data collection part
  • Start a Grafana dashboard and web dashboard
  • Debugging remaining issues with a long-running test
    • 90% of the transactions got included immediately
    • Looking into why the remaining 10% are not being included
      • Potentially latency issues
  • Ongoing discussion about p2p discovery because the current one used by the keypers is a bit difficult to implement in the C# library
  • Nethermind is checking if they can implement a simpler version of Kademlia DHT

Testing

• Hive
  • Setting up proper debugging
• Ethereum/tests
  • https://github.com/NethermindEth/nethermind/pull/6833
  • This would allow Nethermind to run the Ethereum tests
  • Looking into how to make this work for Gnosis

Research

• Gas Market
  • No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

July 31, 2024

Client Team Updates

EL​

• Nethermind:

  • Working on new release: v1.28.0 - Next week
    • Will enable snap sync by default for Gnosis Chain
    • Should make it possible to sync Nethermind in ~30 minutes
    • Includes the fix for Chiado, like v1.27.1
  • Not much progress on EIP-4444 yet

• Erigon:

  • Busy working on Pectra
    • Announced an alpha of Erigon v3
    • Still takes around two days to sync on a 32 GB of RAM machine
    • Because the most recent snapshot is one year old
      • Should be updated next week

• Geth:

  • Still debugging what’s happening with Chiado

• Reth:

  • No updates

Chain Infra

• Gateway
  • The mainnet and Chiado configs were not identical
  • Filter queries were disabled on Chiado when they were properly configured in mainnet
  • Now the config is identical

Innovation

• Shutter
  • Working on general stability improvements
    • Was related to latency
    • A new keyper release helped a lot on the p2p side
    • Some slots are still missed
    • Yesterday night, all keypers stopped sending keys, which was fixed this morning
      • By restarting one of the bootnodes, which seemingly unstuck the keypers
    • On the dashboard side
      • No work on the frontend yet
      • Lot of progress on the backend (network observer, collecting data

Testing

• Hive
  • Worked on tools to make debugging of hive tests easier

Research

• No updates

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

July 24, 2024

Client Team Updates

EL​

• Nethermind:

  • Released hotfix version v1.27.1

    • Hot fix for post Aura

    • Enabled pre-warming blocks on Aura and OP chains

    • Could also run v1.26 or lower fine

• Erigon:

  • Released the first alpha version for Erigon 3!
  • All in one EVM node
  • Sync archive node in a matter of hours not days
  • Also the granularity of state history is reduced from the block level to the transaction level
  • Highly recommend that people start playing with it and experimenting

• Geth:

  • 2 issues
    • Block produced are invalid
    • block production is working simply it produces invalid blocks
    • triggered an issue and uncovered a bug in Nethermind
      • Still debugging
  • other problem this one is a bit more difficult to debug
    • Contract creation is fail in everytime
    • Possibility of tracing and system contracts

• Reth:

  • PR to Reth
    • They want to support it
    • current strategy is not to fork the client
    • build on it its extensibility importing
    • Genesis blocks on devnet worked

Chain Infra

• Gateway
  • No updates

Innovation

• Shutter
  • Researching latency
  • Building a dashboard to observe the status of the general system
    • Investigating the performance and stability
    • Way to for users to inspect the transaction lifecycle of their individual transactions
    • Additional information about decryption keys/slot

Testing

• Hive
  • Developed a technical plan for the hive tests
    • final improvements will be divided into five parts and we have already merged part one which includes all the latest upstream
    • working on the second part which will be include genesis refactoring to use the same approach as substream
    • adding new scripts and tools to speed up test development
  • Started working on the ethereum test to migration Gnosis
    • Working on progress

Research

• Nethermind v1.26 issue
  • A bug in Nethermind led to invalid block production; a fix has been implemented.
  • Shutter version is also updated so shutter validators can go back to the v1.27.1 if needed

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

July 17, 2024

Client Team Updates

EL​

• Nethermind:

  • Focused on Shutter and Pectra

• Erigon:

  • Agreed to add Shutter support to Erigon v3

• Geth:

  • The difficulty code had to be rewritten, but doesn’t completely work yet
  • Can still connect to Nethermind nodes and sync blocks (using snap sync)

• Reth:

  • Stuck on testing post-merge on a devnet
  • Customizability for consensus / custom properties in blocks is still fairly lacking in Reth
    • Cannot add the step and signature fields
    • For genesis this is only an issue on devnets with TTD = 0
  • Upgrading from 1.0.0 to 1.0.1 produced a huge diff on our side, highlighting some potential long-term maintainability issues
  • Not quite clear to Lion what needs to be implemented (we just need verification, not capability to produce blocks etc)

Chain Infra

• Gateway
  • No updates

Innovation

• Shutter
  • Launched last week
  • ~5% of validators producing blocks (one every ~3 minutes)
  • There’s a lot of latency until the validator sees the key, which can result in missed blocks
    • If the key is too late, a normal block is produced instead
  • Want to onboard more validators and work on client diversity too
  • How to do block builders for Shutter?
    • Currently keys are public anyways so it’s fine, but there were talks about encrypting them for specific validators

Testing

• Hive
  • Up to date with upstream
  • Improved the repo to make it easier to add new tests and new clients
• Ethereum/tests
  • No updates

Research

• Gas markets
  • 3 open fronts
    • Research: deciding on the primary sell mechanism
      • Potentially the most difficult part
        • Sell vs strike price, difficult to calculate on-chain
    • “Gas Markets” group on Telegram
  • Prove that it’s safe, as we’re messing with EIP-1559
    • Instead of mining, we’ll refund the costs to the users
    • Miners could buy the futures, and then miners get reimbursed, which may break things (and negate the base fee burning introduced by EIP-1559)
  • Execution: how to consume this
    • Buy the option
    • Send specific transactions in a way that will get you a refund
      • Currently doesn’t use pay masters
      • We need to work on good dev experience for this

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

June 26, 2024

Client Team Updates

EL​

• Nethermind:

  • No specific updates
  • Working on Pectra testnet
  • Helping Guillaume (tracing the genesis block and the likes)

• Erigon:

  • Made a patch release (v2.60.2)
    • An issue in eth_estimateGas was introduced in the previous patch release and this was reverted

• Geth:

  • Tracing the genesis block / extract storage slots from genesis constructors
    • Not needed anymore, Guillaume figured out how to build the right block with constructors rather than code + storage
  • Managed to build the genesis block
  • Now testing if he can snap sync from a Nethermind node for Chiado

• Reth:

  • Looking into pre-merge AuRa

Chain Infra

• Gateway
  • The migration was successfully completed
  • Ingresses got new IPs, and rpc.gnosischain.com had IPs cached to the old datacenter, so had to be refreshed for the new datacenter

Innovation

• Shutter
  • From Shutter
    • Launch on Jul 11, 2024
    • Deployed mainnet contracts yesterday
    • First keyper set has been set up
      • Further decentralization in progress (or at least more nodes in more zones)
    • Opt-in for validators, and we’re intentionally going to keep the number of enabled validators pretty low to be safe for the network
• On Nethermind
  • Fixed most known issues
  • Some issues with event decoding (could be linked to the validator or the contract)
    • After fixing this, should be ready for testing
    • If it’s in the contracts, some are easy to deploy, some would require a new keyper set deployment
    • Results in not being able to include Shutterized transactions
• Questions
  • Should we release only internally?
    • I.e. Gnosis and partners
      • Erigon, Gateway, Nethermind, Shutter
    • => Yes
    • Would still be available for everyone, but we’re not going to be very public about running it on socials
  • Is there any e2e testing to make sure Nethermind doesn’t break Shutter support for the next release, and to test other client implementations?
    • There’s nothing Shutter-specific right now
    • Can this be implemented in Hive?
    • Shutter should be able to allocate some resources towards testing after the launch
  • Survey for Gnosis validators
    • https://docs.google.com/forms/d/e/1FAIpQLSe_rKQdf1-uITqAEmV6giMuv51d3ml0BoYbfkNaZJ-vdl0RtQ/viewform

Testing

• Hive
  • Figured out the problem with the Erigon Cancun suit
  • Potentially figured out the issue for withdrawals on Erigon and Nethermind
    • Will contact Marcos to merge those changes and update the dashboard
  • Those were both test issues and not client issues
• Ethereum/tests
  • No updates
  • Once geth syncs, we should be able to use Geth to generate ethereum/tests, but this already works on Nethermind too

Research

• Block validity verification
  • Huge effort, both on EL and CL, DevOps, infra, …
  • Latency penalty may be more harmful, especially for GC
  • First client generates the witness and pass it to other clients to validate the block / witness
    • However, adding the witness adds latency (+ serializing)
    • Re-executing the block adds latency
  • Lukasz thinks that this effort might be somewhat wasted, because the witnesses will be included in the protocol with verkle trees, so there might not be a need to implement this now
    • It’s also completely incompatible with and would delay verkle
  • Guillaume also thinks that this is wasted time
  • Marek says that if there’s a slow client, it would slow the entire stack down
    • https://gist.github.com/karalabe/e106ac58afc1d611641e543312cf41e3?permalink_comment_id=4765606#gistcomment-4765606
  • Lighthouse is implementing something similar / there are two different ways of passing witnesses, at least this should be standardized
  • If there are very large proofs (because some blocks are created in a malicious way), in the worst case it could add too much latency to be functional, especially on GC
  • Pectra is already huge, and spending time on this might not be great
  • It’s a whole different issue for GC because Nethermind is supposedly dominating
    • On mainnet Geth should be under 60% at this point (based on a self-reported survey), and potentially going in the right direction
    • Maybe Reth’s launch will change this a bit more