Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

July 17, 2024

Client Team Updates

EL​

• Nethermind:

  • Focused on Shutter and Pectra

• Erigon:

  • Agreed to add Shutter support to Erigon v3

• Geth:

  • The difficulty code had to be rewritten, but doesn’t completely work yet
  • Can still connect to Nethermind nodes and sync blocks (using snap sync)

• Reth:

  • Stuck on testing post-merge on a devnet
  • Customizability for consensus / custom properties in blocks is still fairly lacking in Reth
    • Cannot add the step and signature fields
    • For genesis this is only an issue on devnets with TTD = 0
  • Upgrading from 1.0.0 to 1.0.1 produced a huge diff on our side, highlighting some potential long-term maintainability issues
  • Not quite clear to Lion what needs to be implemented (we just need verification, not capability to produce blocks etc)

Chain Infra

• Gateway
  • No updates

Innovation

• Shutter
  • Launched last week
  • ~5% of validators producing blocks (one every ~3 minutes)
  • There’s a lot of latency until the validator sees the key, which can result in missed blocks
    • If the key is too late, a normal block is produced instead
  • Want to onboard more validators and work on client diversity too
  • How to do block builders for Shutter?
    • Currently keys are public anyways so it’s fine, but there were talks about encrypting them for specific validators

Testing

• Hive
  • Up to date with upstream
  • Improved the repo to make it easier to add new tests and new clients
• Ethereum/tests
  • No updates

Research

• Gas markets
  • 3 open fronts
    • Research: deciding on the primary sell mechanism
      • Potentially the most difficult part
        • Sell vs strike price, difficult to calculate on-chain
    • “Gas Markets” group on Telegram
  • Prove that it’s safe, as we’re messing with EIP-1559
    • Instead of mining, we’ll refund the costs to the users
    • Miners could buy the futures, and then miners get reimbursed, which may break things (and negate the base fee burning introduced by EIP-1559)
  • Execution: how to consume this
    • Buy the option
    • Send specific transactions in a way that will get you a refund
      • Currently doesn’t use pay masters
      • We need to work on good dev experience for this

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

June 26, 2024

Client Team Updates

EL​

• Nethermind:

  • No specific updates
  • Working on Pectra testnet
  • Helping Guillaume (tracing the genesis block and the likes)

• Erigon:

  • Made a patch release (v2.60.2)
    • An issue in eth_estimateGas was introduced in the previous patch release and this was reverted

• Geth:

  • Tracing the genesis block / extract storage slots from genesis constructors
    • Not needed anymore, Guillaume figured out how to build the right block with constructors rather than code + storage
  • Managed to build the genesis block
  • Now testing if he can snap sync from a Nethermind node for Chiado

• Reth:

  • Looking into pre-merge AuRa

Chain Infra

• Gateway
  • The migration was successfully completed
  • Ingresses got new IPs, and rpc.gnosischain.com had IPs cached to the old datacenter, so had to be refreshed for the new datacenter

Innovation

• Shutter
  • From Shutter
    • Launch on Jul 11, 2024
    • Deployed mainnet contracts yesterday
    • First keyper set has been set up
      • Further decentralization in progress (or at least more nodes in more zones)
    • Opt-in for validators, and we’re intentionally going to keep the number of enabled validators pretty low to be safe for the network
• On Nethermind
  • Fixed most known issues
  • Some issues with event decoding (could be linked to the validator or the contract)
    • After fixing this, should be ready for testing
    • If it’s in the contracts, some are easy to deploy, some would require a new keyper set deployment
    • Results in not being able to include Shutterized transactions
• Questions
  • Should we release only internally?
    • I.e. Gnosis and partners
      • Erigon, Gateway, Nethermind, Shutter
    • => Yes
    • Would still be available for everyone, but we’re not going to be very public about running it on socials
  • Is there any e2e testing to make sure Nethermind doesn’t break Shutter support for the next release, and to test other client implementations?
    • There’s nothing Shutter-specific right now
    • Can this be implemented in Hive?
    • Shutter should be able to allocate some resources towards testing after the launch
  • Survey for Gnosis validators
    • https://docs.google.com/forms/d/e/1FAIpQLSe_rKQdf1-uITqAEmV6giMuv51d3ml0BoYbfkNaZJ-vdl0RtQ/viewform

Testing

• Hive
  • Figured out the problem with the Erigon Cancun suit
  • Potentially figured out the issue for withdrawals on Erigon and Nethermind
    • Will contact Marcos to merge those changes and update the dashboard
  • Those were both test issues and not client issues
• Ethereum/tests
  • No updates
  • Once geth syncs, we should be able to use Geth to generate ethereum/tests, but this already works on Nethermind too

Research

• Block validity verification
  • Huge effort, both on EL and CL, DevOps, infra, …
  • Latency penalty may be more harmful, especially for GC
  • First client generates the witness and pass it to other clients to validate the block / witness
    • However, adding the witness adds latency (+ serializing)
    • Re-executing the block adds latency
  • Lukasz thinks that this effort might be somewhat wasted, because the witnesses will be included in the protocol with verkle trees, so there might not be a need to implement this now
    • It’s also completely incompatible with and would delay verkle
  • Guillaume also thinks that this is wasted time
  • Marek says that if there’s a slow client, it would slow the entire stack down
    • https://gist.github.com/karalabe/e106ac58afc1d611641e543312cf41e3?permalink_comment_id=4765606#gistcomment-4765606
  • Lighthouse is implementing something similar / there are two different ways of passing witnesses, at least this should be standardized
  • If there are very large proofs (because some blocks are created in a malicious way), in the worst case it could add too much latency to be functional, especially on GC
  • Pectra is already huge, and spending time on this might not be great
  • It’s a whole different issue for GC because Nethermind is supposedly dominating
    • On mainnet Geth should be under 60% at this point (based on a self-reported survey), and potentially going in the right direction
    • Maybe Reth’s launch will change this a bit more

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

June 19, 2024

Client Team Updates

EL​

• Nethermind:

  • Fixing some issues on the release with better processing times
  • Dmytro joined as a QA engineer and will work on Hive testing and testing in general with Egor
  • Exploring EIP-4444

• Erigon:

  • Nothing Special

• Geth:

  • Not much progress
  • Still working on Chiado to get the proper genesis block
    • Geth doesn’t support the constructor config for genesis
    • The 4 first contracts in the file work, but the 5th reverts
    • Maybe we can just fetch the code for those contracts at genesis and then fetch the storage slots so we don’t need to execute the constructors
      • Could be done with Erigon (check with Somnath)
  • Questions
    • Is it possible to trace the genesis block on any of the clients?
    • What was the block building issue on mainnet?
      • Not quite clear, will try to run some validators again

• Reth:

  • Most of the Gnosis specific logic is implemented
  • Lion now wants to run a devnet with Nethermind + Reth
  • Importing state and restarting post-merge can be tough
    • Testing always takes 45-60 minutes to import state
    • In the end, implementing AuRa in Geth took 2 weeks, which was must faster than testing state import

Chain Infra

• Gateway
  • Preparing the migration, should happen this week

Research

• EIP-7002 (EL withdrawals)
  • Still ongoing

Innovation

• Shutter
  • Testing transactions from the dApp
    • Call with Marc?
      • Wasn’t able to join
  • Marc applied a fix and is trying to run this code on Chiado

Testing

• Was https://hive-gno.nethermind.io/ fixed?
  • Yes, but only for Nethermind
    • Some test cases are failing, Marcos will look at them this week
  • There’s an issue on the config for Erigon
• Were Hive tests fixed
  • For Nethermind
  • For Erigon

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Watch the record here

June 12, 2024

Client Team Updates

EL​

• Nethermind:

  • Released an RC with block processing improvements
  • Hired a new QA engineer
    • Talked with Lion about state testing GC (because Hive tests don’t cover that)
    • Could help to prepare those tests

• Erigon:

  • Was able to run Erigon v3 on Gnosis up to T-50k blocks
  • Running fine, but there might be some issues with block downloading towards the end (might also be a networking issue though)

• Geth:

  • Guillaume is sick

• Reth:

  • We want to bring reth to Gnosis (there’s also demand from CoW and other projects)
  • Reth is close to being considered stable
  • Reth is quite extensible, so it might be possible to implement support for Gnosis without having to maintain a fork
  • Will require a state snapshot from the merge, so we don’t have to implement AuRa, as this would be significantly more difficult
  • Reth doesn’t support snap sync yet

Chain Infra

• Gateway
  • Updated Erigon to the latest version

Testing

• Planning on forking https://github.com/ethereum/tests
  • Try to fork it with Nethermind as a generator without modifications
    • Should mostly be implemented by Nethermind but wasn’t battle tested
    • Maybe we should try generating tests for mainnet and compare them
  • Likely some tests will fail, and we can adapt them afterwards
• Is Erigon running POSDAO tests?
  • Geth for sure isn’t
• We shouldn’t necessarily assume that passing mainnet tests means that we also pass Gnosis tests, but it might not be reasonable to fork all tests either
  • We should adapt some tests at least
  • The question is where to draw the line

Research

• EIP-7002 (EL withdrawals)
  • On Ethereum the fee is burned, but we don’t want that on Gnosis Chain
  • Either we keep the same contract and burn the xDAI, or we do a different contract with a different address (which might be worse UX)
  • Burning it shouldn’t be too bad, but it’s not only a “chain revenue” decision, it’s also the fact that the xDAI burned will never be recoverable, which doesn’t really make sense

Innovation

• Shutter
  • All the signatures are now being verified
    • Keypers haven’t swapped over to blst yet, but they have test vectors that are working
  • Nethermind is feature complete
  • Debugging an issue where some transactions are not being included when there’s too many
  • Debugging high CPU usage
  • Code review is finished

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

May 29, 2024

Client Team Updates

EL​

• Nethermind:

  • Big push for optimizing the head processing
  • Soon to be released
  • Big focus on Pectra of course
  • Fix for snap sync server (it was incompatible with geth)

• Erigon:

  • No particular updates
  • Andrew wasn’t able to reproduce the issue
    • Philippe will open a fresh issue
    • Will publish a new version if the specific issue can be reproduced
  • Erigon v3
    • Works on Gnosis
    • Might not be integrated with Caplin yet
    • Will have to test internally to see if it works and how it performs
    • 2-3x higher requirements (mostly RAM)
      • V2 required 16 GB of RAM
      • V3 requires 64 GB of RAM
        • Working on optimizations
    • ETA ~6 months based on stability
      • Contingency for Pectra if it takes too long?
        • V2 can backport Pectra changes in the worst case

• Geth:

  • Working on adding Chiado to Geth to test block proposals at all
    • Problematic because there’s custom contracts and state in the genesis

Chain Infra

• Gateway
  • No update

Research

• Stable Fees With Gas Futures
  • Gnosis Chain can fairly easily do gas futures using the fee collector
  • Checking with economists if this is safe and makes sense
    • It might break EIP-1559
  • Private auctions might be complicated to select sell parameters

Innovation

• Shutter
  • No updates
  • Launch on Chiado went well
  • Worked on some documentation about setting up validators
  • Has stayed connector on both machines since launch
  • There’s still the BLST hack because of incompatibilities between Shutter and Nethermind’s libraries

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

May 08, 2024

Client Team Updates

EL​

• Nethermind:

  • Snap sync and half-path published last week
  • Preparing for Pectra

• Erigon:

  • Switched internal development to Erigon 3
  • Release candidate of Erigon v2.60.0
    • Should be the last significant v2 release
    • There might still be patch releases
    • Includes the eth_getLogs regression introduced in v2.59
      • Log pruning issue fix described in the release notes:
        • https://github.com/ledgerwatch/erigon/releases/tag/v2.60.0-rc1
          • Required for people who ran v2.59 with pruning
  • Erigon v3
    • Syncs way faster (doesn’t have to execute all blocks since genesis)
    • Granularity of state history is much finer
    • Should be cheaper to run because less data has to be on fast drives and colder history can be on cheaper disks

• Geth:

  • No update

Chain Infra

• Gateway
  • No update

Innovation

• Shutter
  • Nethermind now integrates with the keypers, nothing is hardcoded anymore
  • Working on a few stability improvements
    • Syncing / missed slots: fixed
    • Gossipsub disconnection issue still ongoing
    • Not receiving keys on time
    • Discovered an issue on the keyper side for key signing

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

April 24, 2024

Client Team Updates

EL​

• Nethermind:

  • Working on Pectra, close to be ready for devnet0 (expected next week but will be announced at ACD)
  • Testing snap sync
  • Release planned for tomorrow
    • One potential bug remaining
    • Includes half-path and snap sync server for newly synced nodes by default
    • Snap sync in ~45 minutes with one peer
      • Nethermind ↔ Nethermind snap sync tested and working
      • Nethermind from Geth also worked

• Erigon:

  • Working on Pectra

• Geth:

  • Fully syncs on GC!
  • Serving snap sync, was able to snap sync a few nodes from it
  • 3 official Geth nodes were deployed an synced, can be used for snap syncing
    • Enodes to be sent in the Telegram group
  • Working on a rebase from a more recent Geth version
  • Block building seems to be broken
  • There’s a bug for full sync that needs to be addressed
  • Very positive in general, just some housekeeping left

Chain Infra

• Gateway
  • Fixed the checkpointz endpoint
  • Observed OOM issues with 1.25.4
    • The node was killed and couldn’t resync after restarting
    • Will keep Nethermind posted with logs and information

Innovation

• Shutter
  • Pushed a new update with the new curves in keypers
    • Working on integrating with that
  • Working on the libp2p stability issue (probably linked to GossipSub implementation)

Research

• EIP-3074 + EIP-5003

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

April 18, 2024

Innovation​

• Account abstraction
  • https://eips.ethereum.org/EIPS/eip-3074
  • https://eips.ethereum.org/EIPS/eip-5003
    • https://github.com/ethereum/pm/issues/1016#issuecomment-2056927690
  • 5003 would allow to fully transition from EOA to contract based accounts
    • If it’s not done in the next hard fork, the wait for the next one would be quite long and capabilities would not be “complete” for 3074
  • Would be interesting to hear about security considerations for 5003
  • Would be great to have 5003 in addition to 3074 to have the full account abstraction picture
  • Issues
    • Signing messages would still be possible
      • For permit transactions for example
        • Could be fixed by changing the ecrecover precompile for example, but then that would create a dependency on the state, which is a first for precompiles
    • Would break composability
      • The key would work on some chains as an EOA but wouldn’t work at all on others
    • This should be discussed in ACD calls
  • Bound historical data
    • https://eips.ethereum.org/EIPS/eip-4444
    • First step being considered by EF
      • https://github.com/ethereum/EIPs/pull/8266
      • Basically drop all blocks before the merge
    • Data
      • Portal
        • Dedicated team at the EF
        • It’s a parallel network of DHTs
        • You have to custody data based on your node ID
          • Issues
            • Can be spammed because you can mine IDs
            • Network needs sufficient replication in order not to lose data, which introduces more bandwidth and storage requirements
        • Not really proven to work yet
      • Bittorrent
        • Currently employed by Erigon (but not for EIP-4444)
          • They don’t have a state snapshot yet, meaning that you couldn’t sync without doing the whole current process
          • Will be possible with Erigon v3
            • Would only need to store historical data for RPCs or if you need to serve historical data
          • Each node creates checkpoints and seed that data
            • Hashes are added to clients to be able to find them
        • Should be sufficient, because the data can be verified anyways, and is unlikely to disappear because it’s easy to seed and back up
        • ERA1 format
          • https://github.com/ethereum/go-ethereum/pull/26621
          • Potentially not compatible with AuRa
          • Only compatible with pre-merge blocks yet
      • IPFS
      • Swarm
      • Something else in the meantime?
  • Issues
    • Would create two different types of nodes, one of which would be parasitic
      • If everyone stops serving historical data, older nodes would no longer be able to sync
      • https://eips.ethereum.org/EIPS/eip-7542
      • Potential solution
    • Can every CL sync deposits?
      • Lodestar not yet but every team supports this so they’ll implement it soon too
    • Syncing from a state rather than history is not possible on Erigon v2 and will require v3, which is yet to be released
      • Even in v3, historical data would be done in their proprietary format for now, and the client would require changes to support ERA1
  • Questions
  • What’s good enough for Gnosis Chain?
    • In terms of decentralization
  • Why do we care about historical data?
    • Tax reasons, archiving reasons, …
  • Requirements
    • Data doesn’t get lost
    • Newcomers are able to access that data eventually (not necessarily quickly)

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

April 10, 2024

Client Team Updates

EL​

• Nethermind:

  • Nothing new
    • Working on Pectra and half-path / snap sync server

• Erigon:

  • Working on Pectra and Erigon 3

• Geth:

  • Issue when the first blob was included in a block
  • Currently less than 2 weeks behind head
  • Then will test snap sync with a new geth node

Chain Infra

• Gateway
  • No updates

Innovation

• Shutter

  • The first Shutter transaction was included!
  • Transaction submitted to sequencer contract and included in a block
  • However, keypers are mocked for now, so the keys are hard coded
    • Because the keypers are still using the BN curve rather than BLS
    • Will require some changes from Shutter and then some testing
    • No ETA yet
  • The libp2p issue hasn’t been resolved yet, so the connection breaks after around half a hour

• Verkle Tree Integration

  • Gnosis Chain has less time per block to do the transition
  • What’s the upper bound for time we can take to do the transition?
    • One issue is that snap sync is not entirely figured out for the transition yet, so syncing could be a bit more difficult (but at most nodes would have to sync 2 months worth of blocks, which on Gnosis Chain is not that huge of an issue, and it likely doesn’t change anything for Erigon at least)
  • Ethereum is targeting 2 weeks, on Gnosis Chain with identical parameters it would take 4x longer, so 2 months (mental math, to be checked)

Welcome to the Gnosis Core Devs weekly gathering. Every Wednesday, key members from the Gnosis team, contributors, and various team representatives convene to discuss, collaborate, and update one another on the Gnosis ecosystem's progression.

Participants represent teams:

Erigon, Gateway, Nethermind, Geth, Gnosis DevOps, Gnosis Core Devs, Gnosis Comms team.

With a diverse set of voices present, our discussions are rich, multifaceted, and aim to foster innovation within the community.

Missed the meeting? Catch the full recording on Gnosis Chain YouTube channel.

March 27, 2024

Client Team Updates

EL​

• Nethermind:

  • Planning on releasing half-path
    • Still a few weeks away
    • Will have around half the speed of geth for serving snap sync

• Erigon:

  • Small release today with fixes for Caplin (2.59.1)

• Geth:

  • Was able to go all the way to Dencun
  • Will have to rebase now for Dencun support

Chain Infra

• Gateway
  • The eth_getLogs service unavailable error happens on the public RPC when the servers are overloaded
    • With an API key that doesn’t happen

Hive Tests

• No updates

Innovation

• Shutter
  • Managed to fix the threading issue
  • The validator is able to run but can’t produce blocks yet because there’s an issue with passing the decryption keys
    • Might be the protobuf implementation
    • Once this is done, block production should work