Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Feb 05, 2025

Infrastructure

Gateway​

• Gateway
  • Added an Erigon v3 node for testing

Innovation

Shutter​

• Waiting for the next validator release for more tests
• Asking for a license on a Shutter related repo from the dApp dev team

Testing

Hive​

• No updates

Ethereum/tests​

• Generated all of the Pectra test cases
• https://github.com/yerke26/gnosis-fixtures/tree/main
• Fixing some issues on the Nethermind side
• Not integrated with any other client yet
  • Will be added once all issues with test generation are resolved

Client updates

EL​

Nethermind​

• Will release an RC today
• Main release delayed to likely next week
• Will start discussing around ERA files this week with Yaroslav
  • Found a couple of new issues

Erigon​

• Busy with Pectra Devnet 6
  • Had an issue on Devnet 5 on a certain block
    • There’s a protocol difference with Nethermind unrelated to Pectra
    • Is the blob fee collector implemented in Erigon? Might be an issue
    • It’s a create transaction
      • Tried to access an account that was not in the access list, and Erigon charges a full cold load whereas Nethermind doesn't. The transaction has a different cost on both clients.
    • Could be an issue on Chiado and mainnet too
  • Devnet 6: erigontech/erigon:pectra6
    • Candidate branch that is still missing some PRs

Geth​

• New security release (1.14.13) for geth, not threatening to Gnosis Chain
  • Not 1.15.0, which will contain almost everything for Pectra
  • Planned second release 1.15.1 that should be ready for the new Devnet
• Rebased the Gnosis Chain node
• Will need someone from DevOps to set up CI again
• Current state for Gnosis Chain Pectra
  • The spec is up to date
  • Nothing has been done in Geth yet, and the rebase on 1.15.0 might be complicated

Reth​

• Upgraded to a new version that has access to the configurable blob schedule
• There’s an issue in the block after one where 2 blobs are included
  • After fixing this, Reth should be ready for Cancun except for the networking issues
• Peers reject connections when requesting any data

CL​

Nimbus​

• Need the blob schedule to be changed in the preset
• Lion will go through all CLs

Pectra​

• Devnet 5 was decommissioned, so we lost the transaction that caused the fork between Erigon and Nethermind
  • Transaction to debug
• Devnet 6
  • Already set up, but the explorers have yet to be fully configured
  • Make sure every core dev has access to devnet traces
  • Try to replay the buggy transaction from Devnet 5
• Mainnet
  • The consolidation contract has new addresses

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Jan 29, 2025

Infrastructure

Gateway​

• Gateway
  • Deployed the update for the 502 errors

Innovation

Shutter​

• Meeting to discuss libp2p slowness
• Opened access to the p2p metrics
• Ran a test for transaction inclusion, only 3.6% failed transactions

Testing

Hive​

• Nothing new

Ethereum/tests​

• Ran all the tests and fixed some small issues in t8n
• Started generating Prague tests
• Adding Prague logic in t8n to generate Gnosis tests
• Nethermind test runner for GC is ready

Client updates

EL​

Nethermind​

• Focusing on the next release
  • Everything looks healthy
  • Found a few smaller issues
  • Release planned for Monday depending on Pectra plans
    • Might include configs for testnets, depending on tomorrow’s ACD
    • Might thus be delayed
  • Mostly stability issues and focused around Pectra
  • Fixed issues related to receipts on archive nodes
  • Performance improvements
  • 44s ERA import + export

Erigon​

• Focusing on the next releases as well
• Both Erigon v2 and v3 will support Pectra
• The next hardfork (Fusaka) will not be supported by Erigon v2

Geth​

• Not present

Reth​

• 2 issues
  • State root mismatch with Nethermind
    • Seems to work when the deposit contract is deployed before going through Cancun (in isolated tests for blob production)
    • Same issue with the transition between Shapella and Dencun
  • Reth has the wrong constants for Blobs, but they’re hardcoded so will require some upstream changes or a custom TxPool
    • Might work with the Pectra blob configs (EIP-7840)
      • Worth looking into, but requires a rebase to a more recent version of Reth

CL​

Pectra​

• Devnet
  • Managed to set up the deposited validator on Devnet
  • Seems to have issues with Dora explorer (cannot display the deposited validator, results in timeout)
    • Marek says that Dora is for sure ready for Pectra, might be an old version though
  • Might be worth waiting for Pectra Devnet 6 to start from scratch
    • The date will likely be decided tomorrow
  • Dmytro / Carlos will check the status of Erigon and tell Somnath offline
    • Most Erigon validators seem to be down
  • We have to deploy the contracts for consolidation requests
    • This might change tomorrow though, so let’s wait before deploying

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Jan 22, 2025

Infrastructure

Gateway​

• Gateway
  • No specific updates
  • Finished fixing the 502 error
    • Now returning proper errors, like 404 for example
  • Also fixed services rebooting

Innovation

Shutter​

• Focused on improving libp2p stability

Testing

Hive​

• Synced the Gnosis fork with upstream and fixed all merge conflicts

Ethereum/tests​

• Started working on generating ethereum-spec-tests
• Tweaking the code to be able to run it on Gnosis Chain

Client updates

EL​

Nethermind​

• Nothing specific

Erigon​

• Released beta 1 of Erigon v3
• Mostly stability differences from the latest alpha

Geth​

• Nothing specific
• Waiting for the new upstream release
  • 1.15.0 and not 1.14.13, so bigger than expected
  • Big update before Pectra
• The issue one person had might be linked to a dying drive
• ETA for Pectra readiness?
  • Shooting for Monday
  • Also needs some changes specific to Gnosis Chain

Reth​

• Restarted a sync from post-merge on the VM instead of uploading it from the laptop
• Debugging an issue specific to blobs
  • Getting some state root mismatches
• Incoming requests on the VM

CL​

Pectra​

• Mainnet shadow fork maybe by end of this month
• Then a hard fork every 2 weeks
• Mainnet could happen as early as the beginning of March

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Jan 15, 2025

Infrastructure

Gateway​

• Added beacon API RPC endpoints for Ethereum based on Lighthouse
  • More information on Telegram
  • Next step: add Nimbus

Innovation

Shutter​

• Call with Marc to debug libp2p issues

• Working on devtooling to make testing easier

• Not present next week

Testing

Hive​

• No major update

Ethereum/tests​

• Working on bug fixes for Pectra

Client updates

EL​

Nethermind​

• Got 2 reports about missing receipts on archive nodes
  • Potentially found the issue
  • A recovery method could be possible
  • Quite a big edge case

Erigon​

• Shutter
  • Basic p2p connection working for Shutter
  • Will join the Shutter libp2p call
  • Working on reading data from the Shutter smart contract needed for sequencing and decrypting
  • Potentially something could work mid-February
• Pectra Devnet 5 changes were almost all merged into the main branch

Geth​

• Waiting for the person who had a stack trace to try to reproduce it
• New release process hasn’t been tested yet because there hasn’t been a new upstream release
• There’s a potential issue with the bloom filter code
  • Caused a missing block, but should be an edge case
  • Could also happen in mainline

Reth​

• Crossed Shanghai and everything works fine!
• Can now connect to the p2p network
  • Currently fetching / executing ~2.8m blocks in debug mode directly from peers instead of manually
  • Outgoing connections work fine but incoming are getting blocked
• Syncing until ~600 blocks before Cancun now, will then sync manually during the hard fork
• Not tested with CL yet

CL​

Pectra​

• Are we increasing the blob count on Gnosis Chain?
  • No, keeping at 1/2
• Are we adapting pricing as on Ethereum?
  • Yes, do the same as Ethereum

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Jan 08, 2025

Infrastructure

Gateway​

• About to finish CLs for Ethereum mainnet
• Solved the issue with the 500 requests / batch
  • Should not happen again

Innovation

Shutter​

• Still some issues with the p2p communication between validators and keypers
  • Not optimally connected
  • Looked into by Shutter and Nethermind

Testing

Hive​

• No major update

Ethereum/tests​

• Generating some Shanghai tests with Deb

Client updates

EL​

Nethermind​

• Got 2 reports about missing receipts on archive nodes
  • Potentially found the issue
  • A recovery method could be possible
  • Quite a big edge case

Erigon​

• Shutter
  • Working on libp2p to get the decryption keys

Geth​

• Worked on updating the documentation
• Reported a stack trace in production
  • Seems to happen from time to time during block building
  • In the AuRa code for some reason
• What’s the default gas limit on Geth right now?
  • 30m, should maybe be changed down to 17m
    • For both mainnet and Chiado
• Will likely go to 36m in the next release Releases should work and build Docker image
  • Will test with the new release, or can be tried with an rc

Reth​

• Fixed 2 issues
  • Related to the native bridge contract
• Synced around 2.5m blocks and reached Shanghai
  • Will manually go through Shanghai because some issue happened in Geth during that fork, so might as well play it safe
    • The issue on Geth was linked to the smart contract bytecode size, around 5 hours after Shanghai activation
    • Linked to EIP-170

CL​

Pectra​

• Waiting for EF to release Devnet 5
• Dmytro is testing Pectra for Gnosis Chain in the meantime
  • The deposit scripts and UI don’t work with the devnet
  • Tools are not ready to work with consolidation
    • You can’t deposit more than 1 GNO

Research​

• EIP-7843
  • The SLOT precompile could be used by Shutter to cancel a transaction if it’s not executed in the correct slot
  • This can already be done by using the block timestamp, but that would break if the slot length changes
• EIP-7793
  • The TXINDEX precompile can be used to get the index of the currently running transaction in the block
  • Could cancel a transaction if it’s not executed at the top of the block
    • Or introduce slashing potentially
• Guillaume says that EL devs generally don’t like this type of EIP because this can’t be patched out once people start deploying contracts with them. Introspection makes a lot of things more complicated in the EVM.

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Dec 18, 2024

Infrastructure

Gateway​

• Private access to CL RPCs is in progress
• Public access to Ethereum RPCs will be deployed this week

Innovation

Shutter​

• Dappnode has a package to run a Shutter keyper on Dappnode

  • Might not be published yet, but should be shortly

• Working on improving metrics

• Dappnode also already supports the latest Nethermind version with Shutter

Testing

Hive​

• Mostly focused on consensus spec tests

Ethereum/tests​

• Worked on fixing the test cases
• Everything works with retesteth
• The output of the test cases is going to be tested by Debjit
  • Basically testing Nethermind test cases with Reth

Client updates

EL​

Nethermind​

• Released 1.30.1
  • Small patch release
  • There was an issue where Nethermind didn’t start when set up with systemd
  • There might still be an issue with database corruption

Erigon​

• Nothing specific

Geth​

• The two full syncs on the latest branch have successfully concluded Working with the DevOps team to build releases

Reth​

• Still syncing blocks about the merge
• Manually creating RLPs of 50k - 100k blocks
  • If anything breaks, rebuilding smaller chunks to debug
• Peering doesn’t work because the block hash at genesis is different (pre-merge / AuRa isn’t implemented)
  • The genesis hash could still be hardcoded
  • Peers reject connection because of wrong genesis hash and fork id

CL​

Pectra​

• Devnet 5 should be ready last week of this year for Ethereum according to ethPandaOps
  • We’re targeting first week of January for Gnosis Chain
• Things might move fairly quickly on Ethereum testnets for Pectra

Research​

• EIP-4444
  • How does importing ERA1 files actually work? Is it automatic?
  • Import and export are manual processes Through CLI or RPC
  • The import can be triggered whenever you want for full sync
    • For archive you start by importing every ERA1 file and then sync
  • Sebastian: what are the trust assumptions?
    • The import will work exactly the same way as when you import from p2p, so everything is re-executed and verified
    • You could sync a wrong chain, but at some point you can check with the beacon chain roots or pivot blocks hardcoded in Nethermind that the full chain is correct. At that point you would have to resync again.
  • ERA1 is a first step with Portal potentially coming next
  • The Portal Network will be able to serve arbitrary blocks without having to reimport the full history
  • The Portal Network will also work with light clients
    • Can access headers, receipts and potentially even the state
• ERA1 files are just defined for pre-merge but not for post-merge
• Lion: how would file discovery work
  • On Erigon the torrent hashes are saved on a GitHub repo, so it’s centralized Might be an issue with firewalls for example (e.g. China)
  • There might be a fix for this
• Importing and exporting works with ERA1 files for Gnosis on Nethermind currently
  • The full flow works, i.e. snap sync + full sync from ERA1
  • Next steps: finalize testing and review PR
  • ETA Q1
  • Geth doesn’t have ERA1 import yet or pre-merge history drop, so if Nethermind drops the history then Geth would never be able to sync
  • Even with snap sync
• We need to confirm support from every client
  • Erigon doesn’t rely on this (they use their own snapshots)
  • They’re still going to serve all the blocks by default
  • Nethermind will support it
  • Geth doesn’t yet
  • Reth doesn’t matter yet I’d say, it’s not sufficiently advanced yet
• CL ERA files (from Nimbus)
• https://gbc-era.gnosiscoredevs.io/

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Dec 11, 2024

Infrastructure

Gateway​

• No updates

Innovation

Shutter​

• Some issues with the libp2p bootnodes
• Will soon add more keypers
• Plan to make the internal observability Grafana dashboard publicly accessible at some point
  • Still need to hide IPs
• Dashboard
  • https://explorer.shutter.network

Testing

Hive​

• Fixed all failing tests
• Added Reth support
  • Most tests are still failing
  • Allow Deb and others to test their implementation with Hive

Ethereum/tests​

• Depends on Retesteth, which is currently being fixed

Retesteth​

• Fixed some scripts because clients were sending data with extraneous fields
• The Dockerfile had some stable / conflicting dependencies
• Will work on fixing ethereum/tests error messages to be able to fill tests

Client updates

EL​

Nethermind​

• Released 1.30.0
• Almost done with 44s
  • Still fixing issues about exporting and importing ERA files
  • ERA files will be hosted somewhere public (Swarm, IPFS, S3)

Erigon​

• Released erigon3-alpha6
  • Now by default a full node instead of archive node
  • Focused on performance for validating
    • https://erigon.tech/announcing-erigon-v3-alpha-6-focus-on-staking-and-full-node-performance/
• Beta 1 should be released in a few weeks
  • Stability is improving

Geth​

• The full sync on the latest rebase is working fine for now
• Some tester had a database corruption issue on Geth
• Geth is ready to be publicly released

Reth​

• Had issues with post-block processing, the state wasn’t getting committed to the database
  • This was fixed and ~1000 blocks were synced successfully after the merge
• Working on a Reth and Hive integration

Pectra

• All the scripts are ready to deploy a version of Devnet 5 for GC

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Nov 27, 2024

Infrastructure

Gateway​

• Everything’s working well
• Progressing with a Nimbus CL endpoint

Innovation

Shutter​

• Still working on the peer discovery issue

Testing

Hive​

• No updates

Ethereum/tests​

• Working on solving the issues with Nethermind t8n

Client updates

EL​

Nethermind​

• Fixed the most blocking issue for the next release
• Still working on an issue that happens after restarts
  • Difficult to reproduce, fairly rare
  • Seems to happen on very fast machines
• Next release
  • Performance improvements
  • Bug fixes
  • Pectras PRs
  • Shutter in the main branch

Erigon​

• Supports Gnosis Pectra Devnet 1
  • Fix for importing genesis
• Pectra Devnet 4 is live on 2.61.0-beta1
• A v3 version should also work on Gnosis Devnet 1

Geth​

• Finished a full sync on Geth
• Rebased on the latest official release of Geth: 1.14.12
  • Working on a clean PR
  • Able to follow the chain and produce blocks on Chiado
  • Will do a full resync on this latest rebase
• Implemented tracing for system calls
• Cleaned code and removed temporary stuff
• Not sure what Pectra needs to be done on Gnosis
  • https://github.com/gnosischain/specs/blob/master/network-upgrades/pectra.md
• Helped Debjit for Reth
• Missing for a public release
  • The rebase on the right branch
  • Validate everything on the latest rebase with a full sync

Reth​

• Got the post-merge state dump from Geth
• Was able to sync ~70 blocks after importing the state
  • Debugging issues with block execution now, invalid block roots

Research

• Native rollups

  • A debate between Martin and Justin Drake:

• Misc

  • https://eips.ethereum.org/EIPS/eip-7639
    • Generally the consensus is to drop pre-merge history in 6 months
      • Because in 6 months, 2 TB will no longer be sufficient to run a full node
      • Will require support for ERA files, but keep the Portal network optional
        • Nethermind has a PR for ERA files, should be merged soon
          • https://github.com/NethermindEth/nethermind/pull/6547
          • Compatible with both Ethereum and Gnosis
  • We could do the same for Gnosis Chain as a first step
  • Might be problematic for clients like Reth that need to sync the full chain
  • After this, to sync an archive node you’ll need to download and import ERA files manually and the rest would work like it does currently

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Nov 20, 2024

Infrastructure

Gateway​

• Worked on a public RPC for the consensus layer
• Previous version of container deployed by mistake
• Resulted in max 100 batch requests instead of 500
  • Should not happen again

Innovation

Shutter​

• Most of the team was at DevCon
• Working on stability improvements between validators and keepers
• Looking into discoverability algorithms

Testing

Hive​

• Adapted consensus spec tests to Gnosis

Ethereum/tests​

• Discovered the issue on t8ntool but isn’t fixed yet
• Dan and Spencer from the EF joined
  • Want to use EELS/EEST for Gnosis
  • https://notes.ethereum.org/@danceratopz/eest-gnosis
  • This would replace ethereum/tests
  • We would only focus on post-merge tests, as AuRa is too different

Client updates

EL​

Nethermind​

• Working on the next release (delayed because of DevCon)
  • Found critical issues
• Discussing internally about how to do Shutter testing

Erigon​

• No particular updates
• Shutter implementation is going to take 1-2 months

Geth​

• Finished sync up to merge
• 7m blocks to go
• Working on a dump for Debjit on the merge block

Reth​

• Raised some PRs, waiting for review

CL​

Pectra​

Research

• Feedback about Martin’s native rollups
  • Marek thinks that it’s very interesting and would be interested in joining this working group
  • Lukasz has reservations about this idea
    • Mostly social, not technical
    • Ethereum core devs have limited throughput, so decentralizing it through L2s would allow for a faster timeline

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Oct 23, 2024

Infrastructure

Gateway​

• No issues the past week
• Implemented rate limits for batch requests
• In the process of releasing CL endpoints
  • ETA this or next week

Innovation

Shutter​

• Nethermind
  • Engine room release
  • Working on refinements and improvements of the Shutter explorer
  • Testing the latest version on Chiado
    • The stability is looking very good
      • 24 failures out of 8k+
  • Planning for the mainnet release
  • Implementation started by Milen from Erigon
    • Frederik will set a call up

Testing

Hive​

• The execution API suite will get moved
• Some suites rely on t8n

Ethereum/tests​

• Integrating t8n by Nethermind

Client updates

EL​

Nethermind​

• 1.29.1 solves the block creation problem
  • Official release and pushed to Dappnodes etc
  • No missing slots anymore
• Worked on a new Shutter release for 1.29.1, which was released as 1.29.2
• Release 1.30 planned in 1-2 weeks
• Worked with Twinstake
• Should be able to focus more on 44s
• Implemented an endpoint that should allow Nethermind to be used as a block builder

Erigon​

• No particular updates
• Working on devnet 4 and devnet 5
• Will start working on joining Gnosis Devnet 1 as soon as it’s published

Geth​

• This the rebase to the latest 1.14 version (1.14.11)
  • Which should support live tracer
  • Should be ready for Gnosis Devnet 1
    • Missing blob fee collector change
• Snap sync works
• Block building works on mainnet (Gnosis is running 52 validators)
• Still in the process of doing a full sync, but will last at least 2-3 weeks
  • Still pre-merge currently, everything works until there at least

Reth​

• Working on importing the post-merge state from geth or any other client
  • Geth seems to have a command to export the state
    • Guillaume thinks that it might not work
• Waiting for t8ntool

CL​

Pectra​

• Gnosis Devnet 0
  • Based on devnet 3
  • Should probably move to devnet 4 (for Gnosis Devnet 1)
    • Both Nethermind and Erigon agrees, especially because 3 is not compatible with 4 at all

Research

• No updates