Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

June 19, 2024

Client Team Updates

EL​

• Nethermind:

  • Fixing some issues on the release with better processing times
  • Dmytro joined as a QA engineer and will work on Hive testing and testing in general with Egor
  • Exploring EIP-4444

• Erigon:

  • Nothing Special

• Geth:

  • Not much progress
  • Still working on Chiado to get the proper genesis block
    • Geth doesn’t support the constructor config for genesis
    • The 4 first contracts in the file work, but the 5th reverts
    • Maybe we can just fetch the code for those contracts at genesis and then fetch the storage slots so we don’t need to execute the constructors
      • Could be done with Erigon (check with Somnath)
  • Questions
    • Is it possible to trace the genesis block on any of the clients?
    • What was the block building issue on mainnet?
      • Not quite clear, will try to run some validators again

• Reth:

  • Most of the Gnosis specific logic is implemented
  • Lion now wants to run a devnet with Nethermind + Reth
  • Importing state and restarting post-merge can be tough
    • Testing always takes 45-60 minutes to import state
    • In the end, implementing AuRa in Geth took 2 weeks, which was must faster than testing state import

Chain Infra

• Gateway
  • Preparing the migration, should happen this week

Research

• EIP-7002 (EL withdrawals)
  • Still ongoing

Innovation

• Shutter
  • Testing transactions from the dApp
    • Call with Marc?
      • Wasn’t able to join
  • Marc applied a fix and is trying to run this code on Chiado

Testing

• Was https://hive-gno.nethermind.io/ fixed?
  • Yes, but only for Nethermind
    • Some test cases are failing, Marcos will look at them this week
  • There’s an issue on the config for Erigon
• Were Hive tests fixed
  • For Nethermind
  • For Erigon

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

June 12, 2024

Client Team Updates

EL​

• Nethermind:

  • Released an RC with block processing improvements
  • Hired a new QA engineer
    • Talked with Lion about state testing GC (because Hive tests don’t cover that)
    • Could help to prepare those tests

• Erigon:

  • Was able to run Erigon v3 on Gnosis up to T-50k blocks
  • Running fine, but there might be some issues with block downloading towards the end (might also be a networking issue though)

• Geth:

  • Guillaume is sick

• Reth:

  • We want to bring reth to Gnosis (there’s also demand from CoW and other projects)
  • Reth is close to being considered stable
  • Reth is quite extensible, so it might be possible to implement support for Gnosis without having to maintain a fork
  • Will require a state snapshot from the merge, so we don’t have to implement AuRa, as this would be significantly more difficult
  • Reth doesn’t support snap sync yet

Chain Infra

• Gateway
  • Updated Erigon to the latest version

Testing

• Planning on forking https://github.com/ethereum/tests
  • Try to fork it with Nethermind as a generator without modifications
    • Should mostly be implemented by Nethermind but wasn’t battle tested
    • Maybe we should try generating tests for mainnet and compare them
  • Likely some tests will fail, and we can adapt them afterwards
• Is Erigon running POSDAO tests?
  • Geth for sure isn’t
• We shouldn’t necessarily assume that passing mainnet tests means that we also pass Gnosis tests, but it might not be reasonable to fork all tests either
  • We should adapt some tests at least
  • The question is where to draw the line

Research

• EIP-7002 (EL withdrawals)
  • On Ethereum the fee is burned, but we don’t want that on Gnosis Chain
  • Either we keep the same contract and burn the xDAI, or we do a different contract with a different address (which might be worse UX)
  • Burning it shouldn’t be too bad, but it’s not only a “chain revenue” decision, it’s also the fact that the xDAI burned will never be recoverable, which doesn’t really make sense

Innovation

• Shutter
  • All the signatures are now being verified
    • Keypers haven’t swapped over to blst yet, but they have test vectors that are working
  • Nethermind is feature complete
  • Debugging an issue where some transactions are not being included when there’s too many
  • Debugging high CPU usage
  • Code review is finished

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

May 29, 2024

Client Team Updates

EL​

• Nethermind:

  • Big push for optimizing the head processing
  • Soon to be released
  • Big focus on Pectra of course
  • Fix for snap sync server (it was incompatible with geth)

• Erigon:

  • No particular updates
  • Andrew wasn’t able to reproduce the issue
    • Philippe will open a fresh issue
    • Will publish a new version if the specific issue can be reproduced
  • Erigon v3
    • Works on Gnosis
    • Might not be integrated with Caplin yet
    • Will have to test internally to see if it works and how it performs
    • 2-3x higher requirements (mostly RAM)
      • V2 required 16 GB of RAM
      • V3 requires 64 GB of RAM
        • Working on optimizations
    • ETA ~6 months based on stability
      • Contingency for Pectra if it takes too long?
        • V2 can backport Pectra changes in the worst case

• Geth:

  • Working on adding Chiado to Geth to test block proposals at all
    • Problematic because there’s custom contracts and state in the genesis

Chain Infra

• Gateway
  • No update

Research

• Stable Fees With Gas Futures
  • Gnosis Chain can fairly easily do gas futures using the fee collector
  • Checking with economists if this is safe and makes sense
    • It might break EIP-1559
  • Private auctions might be complicated to select sell parameters

Innovation

• Shutter
  • No updates
  • Launch on Chiado went well
  • Worked on some documentation about setting up validators
  • Has stayed connector on both machines since launch
  • There’s still the BLST hack because of incompatibilities between Shutter and Nethermind’s libraries

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

May 08, 2024

Client Team Updates

EL​

• Nethermind:

  • Snap sync and half-path published last week
  • Preparing for Pectra

• Erigon:

  • Switched internal development to Erigon 3
  • Release candidate of Erigon v2.60.0
    • Should be the last significant v2 release
    • There might still be patch releases
    • Includes the eth_getLogs regression introduced in v2.59
      • Log pruning issue fix described in the release notes:
        • https://github.com/ledgerwatch/erigon/releases/tag/v2.60.0-rc1
          • Required for people who ran v2.59 with pruning
  • Erigon v3
    • Syncs way faster (doesn’t have to execute all blocks since genesis)
    • Granularity of state history is much finer
    • Should be cheaper to run because less data has to be on fast drives and colder history can be on cheaper disks

• Geth:

  • No update

Chain Infra

• Gateway
  • No update

Innovation

• Shutter
  • Nethermind now integrates with the keypers, nothing is hardcoded anymore
  • Working on a few stability improvements
    • Syncing / missed slots: fixed
    • Gossipsub disconnection issue still ongoing
    • Not receiving keys on time
    • Discovered an issue on the keyper side for key signing

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

April 24, 2024

Client Team Updates

EL​

• Nethermind:

  • Working on Pectra, close to be ready for devnet0 (expected next week but will be announced at ACD)
  • Testing snap sync
  • Release planned for tomorrow
    • One potential bug remaining
    • Includes half-path and snap sync server for newly synced nodes by default
    • Snap sync in ~45 minutes with one peer
      • Nethermind ↔ Nethermind snap sync tested and working
      • Nethermind from Geth also worked

• Erigon:

  • Working on Pectra

• Geth:

  • Fully syncs on GC!
  • Serving snap sync, was able to snap sync a few nodes from it
  • 3 official Geth nodes were deployed an synced, can be used for snap syncing
    • Enodes to be sent in the Telegram group
  • Working on a rebase from a more recent Geth version
  • Block building seems to be broken
  • There’s a bug for full sync that needs to be addressed
  • Very positive in general, just some housekeeping left

Chain Infra

• Gateway
  • Fixed the checkpointz endpoint
  • Observed OOM issues with 1.25.4
    • The node was killed and couldn’t resync after restarting
    • Will keep Nethermind posted with logs and information

Innovation

• Shutter
  • Pushed a new update with the new curves in keypers
    • Working on integrating with that
  • Working on the libp2p stability issue (probably linked to GossipSub implementation)

Research

• EIP-3074 + EIP-5003

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

April 18, 2024

Innovation​

• Account abstraction
  • https://eips.ethereum.org/EIPS/eip-3074
  • https://eips.ethereum.org/EIPS/eip-5003
    • https://github.com/ethereum/pm/issues/1016#issuecomment-2056927690
  • 5003 would allow to fully transition from EOA to contract based accounts
    • If it’s not done in the next hard fork, the wait for the next one would be quite long and capabilities would not be “complete” for 3074
  • Would be interesting to hear about security considerations for 5003
  • Would be great to have 5003 in addition to 3074 to have the full account abstraction picture
  • Issues
    • Signing messages would still be possible
      • For permit transactions for example
        • Could be fixed by changing the ecrecover precompile for example, but then that would create a dependency on the state, which is a first for precompiles
    • Would break composability
      • The key would work on some chains as an EOA but wouldn’t work at all on others
    • This should be discussed in ACD calls
  • Bound historical data
    • https://eips.ethereum.org/EIPS/eip-4444
    • First step being considered by EF
      • https://github.com/ethereum/EIPs/pull/8266
      • Basically drop all blocks before the merge
    • Data
      • Portal
        • Dedicated team at the EF
        • It’s a parallel network of DHTs
        • You have to custody data based on your node ID
          • Issues
            • Can be spammed because you can mine IDs
            • Network needs sufficient replication in order not to lose data, which introduces more bandwidth and storage requirements
        • Not really proven to work yet
      • Bittorrent
        • Currently employed by Erigon (but not for EIP-4444)
          • They don’t have a state snapshot yet, meaning that you couldn’t sync without doing the whole current process
          • Will be possible with Erigon v3
            • Would only need to store historical data for RPCs or if you need to serve historical data
          • Each node creates checkpoints and seed that data
            • Hashes are added to clients to be able to find them
        • Should be sufficient, because the data can be verified anyways, and is unlikely to disappear because it’s easy to seed and back up
        • ERA1 format
          • https://github.com/ethereum/go-ethereum/pull/26621
          • Potentially not compatible with AuRa
          • Only compatible with pre-merge blocks yet
      • IPFS
      • Swarm
      • Something else in the meantime?
  • Issues
    • Would create two different types of nodes, one of which would be parasitic
      • If everyone stops serving historical data, older nodes would no longer be able to sync
      • https://eips.ethereum.org/EIPS/eip-7542
      • Potential solution
    • Can every CL sync deposits?
      • Lodestar not yet but every team supports this so they’ll implement it soon too
    • Syncing from a state rather than history is not possible on Erigon v2 and will require v3, which is yet to be released
      • Even in v3, historical data would be done in their proprietary format for now, and the client would require changes to support ERA1
  • Questions
  • What’s good enough for Gnosis Chain?
    • In terms of decentralization
  • Why do we care about historical data?
    • Tax reasons, archiving reasons, …
  • Requirements
    • Data doesn’t get lost
    • Newcomers are able to access that data eventually (not necessarily quickly)

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

April 10, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing new
    • Working on Pectra and half-path / snap sync server

• Erigon:

  • Working on Pectra and Erigon 3

• Geth:

  • Issue when the first blob was included in a block
  • Currently less than 2 weeks behind head
  • Then will test snap sync with a new geth node

Chain Infra

• Gateway
  • No updates

Innovation

• Shutter

  • The first Shutter transaction was included!
  • Transaction submitted to sequencer contract and included in a block
  • However, keypers are mocked for now, so the keys are hard coded
    • Because the keypers are still using the BN curve rather than BLS
    • Will require some changes from Shutter and then some testing
    • No ETA yet
  • The libp2p issue hasn’t been resolved yet, so the connection breaks after around half a hour

• Verkle Tree Integration

  • Gnosis Chain has less time per block to do the transition
  • What’s the upper bound for time we can take to do the transition?
    • One issue is that snap sync is not entirely figured out for the transition yet, so syncing could be a bit more difficult (but at most nodes would have to sync 2 months worth of blocks, which on Gnosis Chain is not that huge of an issue, and it likely doesn’t change anything for Erigon at least)
  • Ethereum is targeting 2 weeks, on Gnosis Chain with identical parameters it would take 4x longer, so 2 months (mental math, to be checked)

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

March 27, 2024

Client Team Updates

EL​

• Nethermind:

  • Planning on releasing half-path
    • Still a few weeks away
    • Will have around half the speed of geth for serving snap sync

• Erigon:

  • Small release today with fixes for Caplin (2.59.1)

• Geth:

  • Was able to go all the way to Dencun
  • Will have to rebase now for Dencun support

Chain Infra

• Gateway
  • The eth_getLogs service unavailable error happens on the public RPC when the servers are overloaded
    • With an API key that doesn’t happen

Hive Tests

• No updates

Innovation

• Shutter
  • Managed to fix the threading issue
  • The validator is able to run but can’t produce blocks yet because there’s an issue with passing the decryption keys
    • Might be the protobuf implementation
    • Once this is done, block production should work

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

March 20, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing in particular
  • Starting to think about the next release
    • Might potentially include half-path and a snap sync server
    • Expected in a few weeks
  • Progressive on paprika as well, which can’t fast sync and would thus require snap sync servers

• Erigon:

  • Going to make a new release with post-dencun fixes today
  • Includes a feature for more aggressive pruning of logs and receipts
    • Will only be enabled for new nodes (i.e. synced from scratch)
    • Existing nodes will have to resync
  • Transaction logs are filling the network quite a lot, at least in the last 1-2m blocks
    • This behavior doesn’t happen in Chiado / Sepolia
    • This causes the bloating issue
    • With the default pruning flags, full nodes should be around ~500 GB and archive nodes should be ~1.3 TB
    • One can also use a more aggressive pruning strategy (i.e. last 1k - 5k blocks), which of course would also result in a smaller node. Default is 90k blocks.

• Geth:

  • Not present

Chain Infra

• Gateway
  • Deployed Erigon + Lighthouse on mainnet before Dencun and went fine

Hive Tests

• No updates

Innovation

• Shutter
  • Managed to register the validators
  • Looking into a couple of issues
    • Race condition
    • Libp2p library on MacOS specifically
  • 2 week ETA should still be realistic

Mainnet

• 95% attestations immediately after the hard fork
  • Some issues with the Gateway RPC
  • Might be fixed, Alex will double check

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

March 06, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing in particular
  • Released an experimental version with snap server enabled
    • We can slowly start deploying nodes that are can serve snap sync
    • Enabling the snap server requires a full resync
    • Working on 3 solutions for the state db
      • Path based
      • Half path
      • Paprika
    • Release schedule not defined yet, and will depend on development progress, because paprika will require a new hard fork again after half-path
    • Considered for the next release, but after Dencun for sure
    • Might be a locked behind a flag rather than being a default to start with
    • When snap sync becomes faster than fast sync this feature will be enabled by default

• Erigon:

  • Nothing in particular
    • Looking into why the database is bloating up still

• Geth:

  • Found the issue after London and started syncing again
  • Was missing a location for the free transactions in London
    • At 300k blocks past the last issue (which was on block 11 after London)
  • Next issue will likely be the merge, and after the merge everything has already been tested to work
  • Will then have to rebase for the latest Dencun changes

Chain Infra

• Gateway
  • The RPC nodes with Lighthouse for both Erigon and Nethermind
    • Will likely deliver validators with Erigon this week

Hive Tests

• No updates

Innovation

• Shutter
  • Made some progress on p2p connections
  • Working on integrating the different parts together
  • Will then setup a test validator
  • The first test validator might be ready in 1-2 weeks

Mainnet

• Issue with blob gas being burned instead of added to the fee collector
  • Going to be postponed

Shadow Fork

• Nodes still aren’t done syncing
  • They’re very close to tip but don’t catch up
  • Only Erigon nodes