Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

March 19, 2025

Infrastructure

Gateway​

• Allocated more resources to the CL endpoints

Innovation

Shutter​

• Working on the libp2p updates
• Following up on validator registration issues on Chiado

Testing

Hive​

• Working on EEST integration for Hive

• Had a meeting with EF Testing Team and created a plan for EEST implementation on Gnosis

• Hive as test runner, EEST is implementation of tests

Client updates

EL​

Nethermind​

• Monitoring some small issues for Pectra

  • Small issue on old CPUs, fixed in new release
  • Small issue in trace transactions, related to Hoodi

• New release today for Hoodi support

• Would be good for Chiado validators to upgrade to the latest version, which will also be ran on mainnet

Erigon​

• Release for Hoodi should happen on Monday

• Will implement new Hive and RPC tests

• Testing the Shutter integration on Shutter

  • There seems to be an issue with registering validators, but the Shutter team is looking into it
  • The Erion validators aren’t getting any decryption keys
  • There might be a version mismatch for registration messages

Geth​

• Not present

Reth​

• Upgrading the upstream dependency
  • The diff is huge so it will take some time

CL​

Pectra​

• What would the risks be to still do Pectra on April 1st

  • Some teams might not have had enough time to do all e2e tests yet, for example Lido

• Easter Holidays are April 18th - April 21st

• We’ll reconvene on March 28 to talk about setting a date

  • To potentially set a date for April 14-15
  • And cut releases on April 1

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

No record available

March 12, 2025

Infrastructure

Gateway​

• After Chiado hard fork all good

Innovation

Shutter​

• Small issue in Chiado fork, updating to new libp2p version. Unrelated to Gnosis impl, should be ok by this week.
  • Testing on-going

Testing

Hive​

• Tried to run generated tests, but Hive doesn’t support Gnosis. Need to adjust to make it compatible. Probably a Genesis issue

Ethereum/tests​

• Ethereum/tests
  • No updates, not yet merged

Client updates

EL​

Nethermind​

• Spent time on Holesky recovery, Chiado all good, pending testing results. One issue, which may not affect Chiado. Asked input to Gnosis devops about CPU info.

• Reviewing Pectra post audit, but nothing important found yet

• Working on history expiry

Erigon​

• New release 3.0.0-rc.3 small fix for syncing from scratch post-pectra. One small issue in RPC for estimateGas.

Geth​

• No updates, not ready yet for Pectra.

Reth​

• Made compatible with Chiado, synced up until Cancun stopping before Pectra. Stuck on a post-execution issue.

• Second mainnet sync on-going

• Attempt to be ready by Pectra

CL​

• PR to baked in Chiado, didn’t make it on time for the fork. But for the mainnet release they will be released

Pectra​

• Fork date on Ethereum is unclear at the moment.

• Geth network share appears negligible at the moment.

• We commit to schedule Gnosis date whenever Ethereum schedules, choosing a date that’s a few days earlier. Temptative date of April 1st is dropped.

• DAppNode packages ownership: we should claim ownership of those

  • Nethermind: In their CI already
  • Erigon: Gnosis should CI it
  • Geth: Gnosis should CI it
  • Reth: Gnosis should CI it
  • Lighthouse: Gnosis should CI it
  • Teku: ?, maybe Gnosis should CI it
  • Nimbus: ?, wrapping the release anyway. Gnosis should CI it
  • Lodestar: In their CI, Nico owns it

Research​

• History expiry in Chiado: activation date?
  • Geth may not be ready, but we will keep at least one node able to serve history.
  • Reth already starts from merge state, need to look into support

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

March 05, 2025

Infrastructure

Gateway​

• Updated all the clients to the latest releases
• Still need to look into potential web3signer changes

Innovation

Shutter​

• New test release that includes an updated libp2p
  • Probably released this afternoon in the Chiado release
  • Will test better connectivity with this release

Testing

Hive​

• Python client for Hive to run EEST tests on CI

Ethereum/tests​

• Ethereum/tests
  • Rebasing the EEST repo with upstream

Client updates

EL​

Nethermind​

• Nothing specific except related to Pectra on Holesky, Sepolia and Chiado

• Prepared some monitoring stack for the Chiado hard fork

• New release for the Sepolia / Chiado issue: v1.31.4

Erigon​

• Prepared releases for the Sepolia issue 3.0.0-rc2, and 2.61.3
  • Same release for Chiado

Geth​

• Not present

Reth​

• Pushed Pectra related changes for Chiado

CL​

Pectra​

• Sepolia issue
  • Reproduced on devnet 2 and would appear on Chiado as well
  • The chain would still be progressing but with empty blocks as long as the buggy transaction stays in the mempool
  • We need to upgrade all the Chiado nodes

Research​

• Testing

  • We should test important contract upgrades (namely the deposit contract) in a shadow fork. Had we not caught the issue on Sepolia, we would have instantly broken the chain with a very basic setImplementation call on the deposit contract.

• Marc’s encrypted mempool research

  • https://ethresear.ch/t/smart-account-encrypted-mempools/21834

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Feb 26, 2025

Infrastructure

Gateway​

• No updates

Innovation

Shutter​

• No updates

• Focusing on the Shutter API release

Testing

Hive​

• Finished Pectra testing on QA

• Updated Gnosis Hive repo with upstream

Ethereum/tests​

• Ethereum/tests
  • Almost finished generating Prague tests for Gnosis (95%

Client updates

EL​

Nethermind​

• Prepared a release for the fix regarding Ethereum testnets (Holesky fail)

• Will release Pectra configs for Chiado by end of this week

Erigon​

• Going to release the first release candidate for Erigon v3
  • Should include the Chiado configs

Geth​

• Not present

Reth​

• Second post-merge full-sync is running

  • 25% processed since Monday

• The new features were implemented for Pectra

  • Deb needs a reference to compare his implementation with and is having some issues with Nethermind for this. Will debug on Gnosis QA

CL​

Pectra​

• Check the timestamps for Chiado and Mainnet

  • Nethermind’s automated tests checked the epoch boundaries and that test passed
    • Will still check manually

• Erigon will check

  • The timestamp has to match the slot number
  • The epoch boundary ideally should be checked as well

• Max blob count

  • https://github.com/gnosischain/specs/issues/32
  • On the devnet, Milen had to update some configs for Caplin to work
    • Might be related to the blob count
    • Milen will check what was actually changed, will discuss offline
  • Lion is very confident that going from 6 to 2 max blob is fine

• Blob base fee update fraction

  • Remains unchanged
  • Already documented in the PR on the spec repo

• Holesky failure

  • What if this happens on mainnet?
    • We canonicalize the wrong chain
      • Some deposits will not get counted
      • We have to implement custom rules
    • We slash the wrong chain
  • In any case we need better tools to potentially recover a broken chain if this happens on mainnet
  • Could a shadowfork catch this?
    • In this case not
  • Is it possible to add tests for this type of issue?
    • At the moment there’s nothing universal
    • Client specific

Research​

• Era import / export
  • We’ll work on it after Pectra
  • There’s a tentative consensus around dropping pre-merge history after May 1st

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Feb 19, 2025

Infrastructure

Gateway​

• Upgraded Ethereum nodes for Pectra

Innovation

Shutter​

• Close to a new release for Chiado
• Working with the QA team to set up a Shutter-enabled validator on the Devnet
• Nethermind is testing the fixes but would like Shutter to update their Go libp2p library

Testing

Hive​

• No updates

Ethereum/tests​

• Ethereum/tests

  • Generated 4/8 Pectra test suites for Gnosis Chain

  • There are new tests by the EF we could use, Kamil is looking into it

    • Ethpandaops Assertoor

Client updates

EL​

Nethermind​

• Released 1.31, including Pectra changes for testnets

Erigon​

• Released v3.0.0-beta2, which includes changes for testnets

• Still looking into the eth_estimateGas issue for Coswap

• Code complete for Shutter regarding main functionality

  • Will spend the next 2 weeks testing this on Chiado

Geth​

• Not present

Reth​

• Syncing mainnet, Cancun ready
• Started working on Pectra features
  • Issue with pending non-blob transactions
  • eth_sendRawTransaction RESPONSE {"jsonrpc":"2.0","error":{"code":-32010,"message":"Syncing"},"id":1}

CL​

Pectra​

• Devnet

  • [SOLVED] consolidation doesn't work (high priority)

  • [SOLVED] blob fee collector works

  • [SOLVED] EL exists - no access to GNO (high priority)

  • [SOLVED] Erigon issue with execution requests (high priority)

  • [SOLVED] Erigon runs Gnosis Pectra tests

  • [Deployed for Gnosis Mainnet, not sure about Chiado] deploy 7002/7251/6110 contracts

  • [SOLVED] devnet-5 issue

    • Most likely a mismatch in the versions (Erigon was using Devnet 6 configs, Nethermind Devnet 5)
    • Anders from Nethermind is still looking into it
    • Not reproducible on Devnet 6, but was reproduced on Devnet 5

  • [DISCUSSED with @filoozom] review deposit contract (high priority)

  • Fork on Erigon with Lodestar

    • Niko from Lodestar is looking into it
    • Didn't manage to reproduce it with either Lodestar or Erigon
    • Might not be Gnosis-specific

  • [Low priority] Dora issues

  • [Low priority, can be ignored if consolidation works] The tooling for deposits doesn't work for Ethereum-Devnet, but on Gnosis-Devnet we managed to do deposits

  • Deploy contracts

    • Done on mainnet
    • Need a node with unsafe EIP-155 support
      • Does anyone have one?

  • Run Pectra tests on Erigon

    • Done

  • Chiado schedule

    • Slot 15171584 - Thu 06/03/2025, 09:43:40 UTC

  • Mainnet schedule

    • Slot 20905984 - Tue 01/04/2025, 16:01:00 UTC

  • We should push for consolidations after mainnet to make the network more efficient

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Feb 12, 2025

Infrastructure

Gateway​

• Gateway
  • No Updates

Innovation

Shutter​

• Working on libp2p
• Nethermind pointed out that there’s a new libp2p release for Go
• Found some libp2p bugs that could be responsible for the issues

Testing

Hive​

• No updates

Ethereum/tests​

• Improvements for EIP-6610

Client updates

EL​

Nethermind​

• Released RC
  • Narrowed down 2 small PRs that still need to be merged
• Full version today or tomorrow for the Ethereum deadline
  • Still investigating the issue for Gnosis Devnet 5 before cutting a new release

Erigon​

• Will have a release ready by tomorrow
• Carlos reported an issue about a request having an incorrect prefix
  • Already fixed and merged, to be tested and confirmed by Carlos before release
• Call later today to decide whether the release will be tomorrow or later

Geth​

• Released 1.15.0, not the final release for devnets, should still be able to join
  • Current release might not be able to create blocks with blobs
• Gnosis Chain Geth
  • Currently rebasing on 1.15.0
  • Blob base fee collector still to be done

Reth​

• Bugs with 2 blobs
  • Was using the blob parameters in some locations but not others (and defaulting to Ethereum’s values)
  • Has now been made consistent everywhere, so it now syncs Cancun correctly
• Synced to ~80 days ago
• Wants to join the Pectra devnet
  • Will need the genesis etc to join
• What are execution speeds for other clients?
  • Reth currently syncs at 40-70 mgas/s
  • Geth is around 6 mgas/s

CL​

Pectra​

• Devnet
  • Consolidations don’t work
    • Linked to churn limit
    • Will spin up a new devnet today
  • Can’t access the GNO after EL exits
    • What happened here?
    • Trying to reproduce, not 100% sure that it’s an issue
    • The exit worked, but the GNO might not be accessible
      • Let’s keep the current devnet 6 as the exit is already done to confirm this
  • Erigon issue with execution requests
    • There is already a new Erigon version that should fix this
      • The last version failed because the CLs didn’t accept the payload
      • Linked to the incorrect prefixes
  • Review deposit contract
    • For EL exits?
    • For batchDeposit
  • Spam with blobs
    • Already running
  • Gnosis Pectra tests on Erigon
    • Pectra tests are ran against Nethermind and work, would be good for Erigon to run those as well
      • Waiting for Yerke to be back from holidays
  • Deploy 7002/7251/6110 contracts
    • Will be done by Philippe today
    • 7002 (0x00000961Ef480Eb55e80D19ad83579A64c007002)
    • 7251 (0x0000BBdDc7CE488642fb579F8B00f3a590007251)
    • 2935 (0x0000F90827F1C53a10cb7A02335B175320002935)
  • Devnet 5 issue
    • Should be close to figuring out what’s happening
    • Pretty much narrowed it down to one PR, building a new image now for testing
    • Might be related to running the wrong spec on Devnet 5
  • Dora issues (low priority)
    • Not a blocker, but might be helpful to fix
  • Tooling issues on Gnosis vs Ethereum (low priority)
    • https://github.com/eth-educators/ethstaker-deposit-cli doesn’t work?
    • Might also be linked to the churn limit

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Feb 05, 2025

Infrastructure

Gateway​

• Gateway
  • Added an Erigon v3 node for testing

Innovation

Shutter​

• Waiting for the next validator release for more tests
• Asking for a license on a Shutter related repo from the dApp dev team

Testing

Hive​

• No updates

Ethereum/tests​

• Generated all of the Pectra test cases
• https://github.com/yerke26/gnosis-fixtures/tree/main
• Fixing some issues on the Nethermind side
• Not integrated with any other client yet
  • Will be added once all issues with test generation are resolved

Client updates

EL​

Nethermind​

• Will release an RC today
• Main release delayed to likely next week
• Will start discussing around ERA files this week with Yaroslav
  • Found a couple of new issues

Erigon​

• Busy with Pectra Devnet 6
  • Had an issue on Devnet 5 on a certain block
    • There’s a protocol difference with Nethermind unrelated to Pectra
    • Is the blob fee collector implemented in Erigon? Might be an issue
    • It’s a create transaction
      • Tried to access an account that was not in the access list, and Erigon charges a full cold load whereas Nethermind doesn't. The transaction has a different cost on both clients.
    • Could be an issue on Chiado and mainnet too
  • Devnet 6: erigontech/erigon:pectra6
    • Candidate branch that is still missing some PRs

Geth​

• New security release (1.14.13) for geth, not threatening to Gnosis Chain
  • Not 1.15.0, which will contain almost everything for Pectra
  • Planned second release 1.15.1 that should be ready for the new Devnet
• Rebased the Gnosis Chain node
• Will need someone from DevOps to set up CI again
• Current state for Gnosis Chain Pectra
  • The spec is up to date
  • Nothing has been done in Geth yet, and the rebase on 1.15.0 might be complicated

Reth​

• Upgraded to a new version that has access to the configurable blob schedule
• There’s an issue in the block after one where 2 blobs are included
  • After fixing this, Reth should be ready for Cancun except for the networking issues
• Peers reject connections when requesting any data

CL​

Nimbus​

• Need the blob schedule to be changed in the preset
• Lion will go through all CLs

Pectra​

• Devnet 5 was decommissioned, so we lost the transaction that caused the fork between Erigon and Nethermind
  • Transaction to debug
• Devnet 6
  • Already set up, but the explorers have yet to be fully configured
  • Make sure every core dev has access to devnet traces
  • Try to replay the buggy transaction from Devnet 5
• Mainnet
  • The consolidation contract has new addresses

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Jan 29, 2025

Infrastructure

Gateway​

• Gateway
  • Deployed the update for the 502 errors

Innovation

Shutter​

• Meeting to discuss libp2p slowness
• Opened access to the p2p metrics
• Ran a test for transaction inclusion, only 3.6% failed transactions

Testing

Hive​

• Nothing new

Ethereum/tests​

• Ran all the tests and fixed some small issues in t8n
• Started generating Prague tests
• Adding Prague logic in t8n to generate Gnosis tests
• Nethermind test runner for GC is ready

Client updates

EL​

Nethermind​

• Focusing on the next release
  • Everything looks healthy
  • Found a few smaller issues
  • Release planned for Monday depending on Pectra plans
    • Might include configs for testnets, depending on tomorrow’s ACD
    • Might thus be delayed
  • Mostly stability issues and focused around Pectra
  • Fixed issues related to receipts on archive nodes
  • Performance improvements
  • 44s ERA import + export

Erigon​

• Focusing on the next releases as well
• Both Erigon v2 and v3 will support Pectra
• The next hardfork (Fusaka) will not be supported by Erigon v2

Geth​

• Not present

Reth​

• 2 issues
  • State root mismatch with Nethermind
    • Seems to work when the deposit contract is deployed before going through Cancun (in isolated tests for blob production)
    • Same issue with the transition between Shapella and Dencun
  • Reth has the wrong constants for Blobs, but they’re hardcoded so will require some upstream changes or a custom TxPool
    • Might work with the Pectra blob configs (EIP-7840)
      • Worth looking into, but requires a rebase to a more recent version of Reth

CL​

Pectra​

• Devnet
  • Managed to set up the deposited validator on Devnet
  • Seems to have issues with Dora explorer (cannot display the deposited validator, results in timeout)
    • Marek says that Dora is for sure ready for Pectra, might be an old version though
  • Might be worth waiting for Pectra Devnet 6 to start from scratch
    • The date will likely be decided tomorrow
  • Dmytro / Carlos will check the status of Erigon and tell Somnath offline
    • Most Erigon validators seem to be down
  • We have to deploy the contracts for consolidation requests
    • This might change tomorrow though, so let’s wait before deploying

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members of the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With diverse voices present, our discussions are rich and multifaceted and aim to foster innovation within the community.

Watch the record here

Jan 22, 2025

Infrastructure

Gateway​

• Gateway
  • No specific updates
  • Finished fixing the 502 error
    • Now returning proper errors, like 404 for example
  • Also fixed services rebooting

Innovation

Shutter​

• Focused on improving libp2p stability

Testing

Hive​

• Synced the Gnosis fork with upstream and fixed all merge conflicts

Ethereum/tests​

• Started working on generating ethereum-spec-tests
• Tweaking the code to be able to run it on Gnosis Chain

Client updates

EL​

Nethermind​

• Nothing specific

Erigon​

• Released beta 1 of Erigon v3
• Mostly stability differences from the latest alpha

Geth​

• Nothing specific
• Waiting for the new upstream release
  • 1.15.0 and not 1.14.13, so bigger than expected
  • Big update before Pectra
• The issue one person had might be linked to a dying drive
• ETA for Pectra readiness?
  • Shooting for Monday
  • Also needs some changes specific to Gnosis Chain

Reth​

• Restarted a sync from post-merge on the VM instead of uploading it from the laptop
• Debugging an issue specific to blobs
  • Getting some state root mismatches
• Incoming requests on the VM

CL​

Pectra​

• Mainnet shadow fork maybe by end of this month
• Then a hard fork every 2 weeks
• Mainnet could happen as early as the beginning of March

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

Jan 15, 2025

Infrastructure

Gateway​

• Added beacon API RPC endpoints for Ethereum based on Lighthouse
  • More information on Telegram
  • Next step: add Nimbus

Innovation

Shutter​

• Call with Marc to debug libp2p issues
• Working on devtooling to make testing easier
• Not present next week

Testing

Hive​

• No major update

Ethereum/tests​

• Working on bug fixes for Pectra

Client updates

EL​

Nethermind​

• Got 2 reports about missing receipts on archive nodes
  • Potentially found the issue
  • A recovery method could be possible
  • Quite a big edge case

Erigon​

• Shutter
  • Basic p2p connection working for Shutter
  • Will join the Shutter libp2p call
  • Working on reading data from the Shutter smart contract needed for sequencing and decrypting
  • Potentially something could work mid-February
• Pectra Devnet 5 changes were almost all merged into the main branch

Geth​

• Waiting for the person who had a stack trace to try to reproduce it
• New release process hasn’t been tested yet because there hasn’t been a new upstream release
• There’s a potential issue with the bloom filter code
  • Caused a missing block, but should be an edge case
  • Could also happen in mainline

Reth​

• Crossed Shanghai and everything works fine!
• Can now connect to the p2p network
  • Currently fetching / executing ~2.8m blocks in debug mode directly from peers instead of manually
  • Outgoing connections work fine but incoming are getting blocked
• Syncing until ~600 blocks before Cancun now, will then sync manually during the hard fork
• Not tested with CL yet

CL​

Pectra​

• Are we increasing the blob count on Gnosis Chain?
  • No, keeping at 1/2
• Are we adapting pricing as on Ethereum?
  • Yes, do the same as Ethereum