Eth 2.0 Dev Update #56 — “Getting ready for Spadina”

Eth 2.0 Dev Update #56 — “Getting ready for Spadina”

Testnet Updates and Hard Fork Capabilities

Spadina testnet release, genesis event rehearsal:

Spadina testnet has been set to genesis on September 29 at 12pm UTC. (Genesis time: 1601380800) This is a dress rehearsal Testnet for the community to practice sending deposits and launching beacon nodes and validator clients starting from genesis. We all know practice makes perfect, so this will help us become familiar with the process as depositing and genesis are the more difficult and risky parts. This is a small scale testnet with 1024 validators to genesis where the main net spec requires 16384 validators to launch. The Testnet is meant to be short lived (~3 days) and after whoever wants to maintain can continuously maintain it.

Some useful links:

Oh! and there’s still time to send your deposits, so don’t miss out on the fun!

Weak subjectivity spec and implementation

Weak subjectivity check is necessary to defend against a problem known as a long range attack. Similar to a genesis block that is finalized and irreversible by definition, a weak subjectivity checkpoint serves the same purpose. A beacon node which uses the weak subjectivity checkpoint will have to ensure that checkpoint is always part of the canonical chain. If a node sees a block conflicting with a weak subjectivity checkpoint, the node will immediately reject the block and exit itself out of the running process.

There has been great analysis done by Aditya Asgaonkar on how long a safe weak subjectivity period should be. For detailed analysis I recommend reading his post. The official eth2 spec repo has a new section on weak subjectivity which will be implemented by the client teams.

In Prysm, we have started our implementation in the following PR. Node operators will be able to try out adding weak subjectivity checkpoint via command line soon. For more background on weak subjectivity, I recommend the following post written by Vitalik

thank you weak subjectivity check for preventing me going on an attacker’s chain! ❤

Upgrades & hard forks in ETH2

Ethereum 2.0 is a phased rollout starting with Phase 0. Each phase requires a critical system upgrade, sometimes referred to as a “hard fork”, in which all participants in the system will switch to a different set of rules or parameters at a specific time. The current specification in phase 0 is set up to support critical system upgrades but does not outline exactly how such an upgrade would be performed. This week, our teammate Preston Van Loon investigates a potential implementation for applying upgrades in phase 0. Check out the one page design document here. In a collaborative effort, Danny Ryan responded in another document here.

Merged Code, Pull Requests, and Issues

Process logs streaming via websockets

We now have the ability to subscribe to logs for our beacon node and validator client via a websocket connection, making it easy to display those logs in dashboards or stream them to other processes! Beacon node logs can be accessed at ws://localhost:8080/logs by default and the validator logs at ws://localhost:8081/logs.

Eth2 Standard API definitions complete + implementation started

Our team member Ivan has been working on integrating the standard API for Eth2 into Prysm in order for our API to align with the official standard. Once all clients align with these standards, tooling could be built that is compatible with every client and integrations become much simpler. This can open the door for more block explorers, community tools, and overall more seamless integrations that use multiple clients. We have finished the definitions and are moving forward with the integration.

Voluntary exits implementation ready

After several weeks of work we are happy to announce that voluntary exits are finally complete. Most of the work concentrated on initiating the exit on the validator’s side and sending it over to the beacon node via a gRPC call. This week the last PR has been merged, the documentation has been released and the first validator has exited (big shout-out to our Discord user @Raphael for sacrificing his validator to help us test the feature on Medalla).

👋 good bye to validator 39048

As usual, we are encouraging everyone to try out the functionality yourself. This will help us improve user experience and find possible implementation bugs. And it will give you the warm fuzzy feeling that your hard-earned GöETH is finally safe and sound.

Test coverage improvements

In a continuous effort to improve the test coverage of critical components we have landed a number of important commits. The most significant update is related to full coverage of initial synchronization state transition functions (see #7320, #7286, and #7285). The updated test suite handles all state transitions that might occur during initial synchronization, which solidifies the codebase even further, before the mainnet. Another important area that has received more tests is related to attestation proposing mechanisms (see #7267 and #7206).

We will keep up extending our quality assurance infrastructure, making sure that all of the critical paths are fully covered.

Prysm web UI beta testing very soon

We know the community really wants a user interface for their eth2 nodes and validators, and we are planning our first round of beta testing the first week of October! Our web UI has been shaping up nicely and includes a lot of useful functionality for those stakers that are more comfortable on the web than interacting with terminal commands. We designed and built the entire interface ourselves, putting our heart into creating something useful for our stakers.

We believe this interface will add a lot more usability to our client, making it a more appealing choice going into mainnet. Some of the features included at launch are:

  • Ability to create a new HD wallet or import keystores
  • Ability to see your validators’ recent performance, wallet details, and more.
  • Ability to monitor your beacon node and validator logs
  • Ability to look at your beacon node’s sync status
  • Ability to see the validator queue and your validators’ places on the queue
  • Ability to monitor global validator participation
  • Back up keys, filter keys, and add more keys to your wallet

Data migration for slashing protection between clients

As part of eth2.0 multi client approach it is important to support easy and safe validator migrations between clients. Making a safe transition of validators between clients consists of both moving public keys safely and moving slashing protection db data between clients. As part of the effort to make the transition safe @sproul from sigma prime team compiled a doc with standardization of the local slashing protection data interchange format. In order to support the full standard we are making changes in our local protection db and adding the import and export features to our codebase. We hope to have the new protection db format running on most nodes in the coming weeks.

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Angular and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Eth 2.0 Dev Update #56 — “Getting ready for Spadina” was originally published in Prysmatic Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Eth 2.0 Dev Update #55 —Medalla Testnet!

Medalla Testnet Launched Successfully

Summarizing the Launch

The entire Ethereum community has been waiting for a stable, “official” multiclient testnet for eth2 for quite some time, and we have finally delivered. On August 4th, the Medalla testnet for phase 0 of eth2 went live, with 5 separate implementations to choose from. At launch, the Ethereum also created the official eth2 launchpad as a great initiative put together by the Ethereum Foundation to help anyone participate in the Medalla multiclient testnet. The launch was broadcast live on Youtube

With high stakes at launch, and client teams rushing to push last minute updates to users, many validators were not perfectly set up at genesis, leading to subpar starting participation. The chain requires at least 66% of validators to be actively validating for the chain to finalize, and we were just short of 57% for the first half hour of the launch. Soon after, more people started coming online, updating their nodes, and participation suddenly jumped to around 80–85%, leading to greater stability in the chain. Although this is not much of a concern during the testnet, the mainnet launch needs to be far more telegraphed, with clients having a stable version a long time before the release.

Overall, Medalla has been an excellent experience, igniting hundreds of people’s excitement for eth2 and getting more people involved. A lot of validators decided to use our project, Prysm, to participate in the testnet and we have been thrilled about the community it now has. We now have over 5000 users in our Discord! If you want to explore the chain’s data, we highly recommend checking the popular block explorer by Bitfly,, as they have put a lot of work into providing great tools to monitor your validators and their performance.

How to Join the Multiclient Testnet With Prysm

At launch of the testnet, there were 5 separate implementations stakers could choose from, including our project, Prysm. We have put a lot of work into properly documenting how to join the testnet with dedicated instructions here. In summary, during the launchpad process, you can generate your validator keys and submit your deposit, and then import these keys into Prysm to run your validator. Since launch of the testnet, we are now seeing around 65% of the nodes running Prysm (note: this does not count nodes that may be running behind a NAT, only those which have an advertised, public IP address).

Aside from our instructions on how to join Medalla, we also prepared detailed information on all there is to know about wallet and accounts in Prysm, everything about running a validator, and information about how to check if your system is running correctly. If you still haven’t joined the testnet and need some help, join our Discord and our team will be glad to assist you.

New Teammate

Prysmatic Labs has always been a grassroots, community effort. Most us started with a curiosity for eth2, began contributing and asking questions to better understand this multi-year effort. One of our consistent contributors, Radosław Kapka, also known as Radek (, impressed us with how quickly he’s learned the internals of our code base, submitting non-trivial optimizations to the Prysm client. Radek has many years of experience in software architecture and brings a detail-oriented perspective into building code for eth2. Welcome Radek!

Merged Code, Pull Requests, and Issues

Prysm is up to date with eth2 wallet standards

In the days leading up to the Medalla public testnet launch, we were hard at work on ensuring we align to eth2 wallet and accounts standards for managing validator keys. The testnet launch was a hallmark event for client interoperability, and we needed users to easily transition from the eth2 launchpad into running Prysm. We have completely revamped how Prysm handles validator keys, the wallets we support, and the different customization options available for users. Prysm, by default, supports 3 different kinds of wallets:

  1. Remote signing wallet An advanced kind of wallet in which validator keys and signing requests are processed by a remote server via gRPC
  2. non-HD wallet: A simple wallet in which accounts are password protected and imported from an external source. This is the recommended approach if you want to import an account from the Medalla testnet launchpad and you read dedicated instructions here.
  3. HD wallet: A common type of blockchain wallet which is generated from a english mnemonic, able to create new accounts deterministically.

As a validator, you’re expected to be consistently online to produce blocks and vote on others’ blocks, as this is how you get rewarded for participating in eth2. To do this, your software needs to have instant access to your validating key, also referred to often as a “hot key” or access to a “hot wallet”. Keeping your withdrawal key, or wallet mnemonic far away from your validator client is what will give you optimal security in eth2. If someone were to steal your validating keys, they wouldn’t be able to withdraw your validator’s staked ETH.

The ideal security for an average user participating as a validator is as follows:

  • Create a wallet using the official eth2.0-deposit-cli and keep your mnemonic stored offline, safely
  • Import only the validating keys you need into your validator client, such as by following the instructions here

For best security in production cloud deployments, it’s best you use a remote signer, as that offers absolute separation of your secret keys and your validator client software. Read more about remote signers here.

Resolving missed attestations reports from users

Users have reported seeing attestation status “missing” while viewing validator attestations in the beacon block explorer. This could happen when there’s a lot of skip blocks in an epoch or this could happen when the validator is offline. Upon further investigations, we did root cause an edge case where Prysm validator unintentionally loses signed committee attestations. Let us explain how that could happen 👇

When the beacon node first starts up, its first job is to find relevant peers in the same committee subnet to gossip committee attestations with. In abnormal cases, this peer discovery process may take more than a few seconds. While the beacon node is still discovering peers, if the validator client requests beacon node to public attestation in the committee subnet where there’s no peer, the beacon node would just broadcast the attestation “into the void”, aka broadcast to no one. Such an issue was cross-checked with the metric `p2p_attestation_subnet_recovered_broadcasts`. We restarted the beacon node and saw `p2p_attestation_subnet_recovered_broadcasts` immediately jumped to 17 while discovering committee subnet peers, that was 17 attestations unanswered! The fix for this is in #6893. With this fix. We are confident that there will no longer be any missed attestation unless the validator client went offline or there’s a heavy chain congestion.

In addition, we have added a RPC debug endpoint to query attestation inclusion slot of a given validator ID and the validator’s assigned committee slot. This is really useful to debug missing attestation and isolate protocol and network failure without suspecting block chain explorer may be displaying data wrong. See #6895 for more info.

Roughtime bug resolved for clock skew

Some of the common errors that users have reported since Medalla genesis are deadline exceeded and the following.

level=error msg=”Could not request attestation to sign at slot” error=”rpc error: code = InvalidArgument desc = invalid request: attestation slot 3558 not within attestation propagation range of 2326 to 2358 (current slot)” prefix=validator pubKey=0x881525ad8f18 slot=3558″

In prysm, we use the roughtime package as a NTP (network time protocol) backup. Why do we use roughtime? NTP responses are insecure and may be subject to MITM attacks. This is especially risky if the host machine only synchronizes its clock every few hours. Roughtime is an awesome project that aims to provide secure time synchronisation. It can resolve clock time within <10s acceptance. While using the roughtime implementation, we suspected there may be a clock skew caused by roughtime in the validator client or beacon node to cause attestation requests that far in advance. We later confirmed our suspicions by the histogram. The clock offset went way up for 1 hour!

To mitigate this issue, the node will now check roughtime more frequently when there’s a large clock offset. At the moment, we only check the clock calibration every hour. #6879 fixes to check every minute instead of every hour. In parallel, #6898 in happening in parallel to make use of roughtime an opt in feature while we evaluate potential issues stemming from erroneous roughtime response.

Fix big issues in local slashing protection and more

A few users have reported local slashing protection cause validator client to panic in #6468. This issue was immediately fixed in #6877, the fix is to prevent a panic in case the history data is nil. We also added better logging in the event where slasher is not up in #6907 also better redial mechanism in #6889 and a race fix in #6871. The local slashing is enabled by default but we also recommend users to run a remote slasher to a more enhanced prevention.

Upcoming Work

Validator API to build a web application for managing your validator client

Modern services running on cloud deployments typically expose a web application for administering them effectively and easily. A highly requested feature for eth2 validator clients has been having a graphical user interface, and we are planning to deliver on this front very soon. Our goal is to have a password-protected local interface you can use to manage your validator accounts, view your validator’s performance, and look at process analytics as needed. We are starting work on the backend and frontend concurrently, and you can take a look at our API design so far here.

Peer scoring and improvements to chain sync

The first phase of a creation of a more sophisticated peer scoring routine has been completed. During that phase we have:

  • Updated our peer allow/ban list functionality to be a part of the newly implemented scoring package. Eventually, this scoring information will be used by GossipSub as well, so that evil peers will be excluded from the mesh.
  • Created a better protection against slow peers, so that peers returning low number of blocks on average, are used probabilistically less frequently.
  • Created a better protection against weak peers, so that peers returning less useful blocks (blocks that cannot be processed), are descored and used probabilistically less frequently.

The next phase will include working with GossipSub protocol by injecting our peer scoring function, and re-arranging the mesh based on relative peers performance. All this will allow for a more smooth synchronization experience in adversary environments. For more information, please, see #6622.

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Eth 2.0 Dev Update #55 —Medalla Testnet! was originally published in Prysmatic Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Eth 2.0 Dev Update #54 — So Close to Official Multi-client Testnet!

Eth 2.0 Dev Update #54 — So Close to Official Multi-client Testnet!

Onyx Testnet Updates

Bad blocks propagated in Onyx

Last week, the Onyx testnet experienced a few hours of volatile peer to peer messaging. Users were reporting high error rates in processing new blocks and it seemed to affect the whole network. This issue caused a period where the beacon chain was unable to finalize. We identified the root cause to be a Prysm client which was out of sync and producing blocks with a parent older than the last finalized block. In the Ethereum 2.0 specification, blocks cannot build upon old finalized blocks. However, the specification allowed for blocks which violated this condition to be “valid” for propagation. The result was that clients would receive the block, forward it through the network, then fail to process it. The issue was first reported on github and was resolved within a few hours. The fix has landed in Prysm alpha.15 release and a recommended specification change has been submitted to the eth2 spec repository.

Prysm accounts revamp underway

For those who have run a validator, you may have noticed that our existing key management tooling isn’t the best user experience, so we’re been working on cleaning it up! A revamp of our account management has been long overdue since the introduction of EIP-2335. With EIP-2335, a standardized keystore directory and file format are standardized for all clients to follow a similar strategy to make swapping between clients seamless! While we’ve been refactoring the key management tooling, we’ve also been making sure our new wallet tooling is much easier to use, including displaying account information in a clean, color-coded form, on top of a streamlined prompt entry for account creation and modifications.

You can track our progress here:

Merged Code, Pull Requests, and Issues

Big hot/cold state refactoring complete

In the last biweekly update, we mentioned a major refactoring of our state memory management to reduce the frequency of disk operations with regards to saving states. This new feature started with a user report that a Prysm client took several minutes to sync after restarting a fully synced node and ended up with two weeks of design discussion, refactoring, and optimization. We’re really pleased with the initial results (+10% sync speed) and expect this feature to land in the next Prysm release.

Blockchain service refactor and huge bump in test coverage

One of the audit feedback was concerning the around block chain package. The block chain package is one of the critical components of Prysm and it handles beacon block and attestation processing. The package needs to be simple, well understood and with well test coverage. Ever since the feedback, we have been fine combing the block chain package to make it simpler and have fewer interfaces. The summary of the overall work is listed here:

  • 6537 — Clean up regular block processing path
  • 6562 — Clean up initial sync block processing path
  • 6599 — Clean up initial sync batch block processing path

As of overall test coverage improvements, this remains work in progress. We have started narrowing down the testing gaps and improving one file at a time.

  • 6613 — Improve test coverage for receive block path
  • 6618 — Improve test coverages for logging and tree handler

The overall progress can be tracked via 5783

Block batch verify in initial sync

One of the major improvements required to speed up chain synchronization has to do with how we verify block signatures. The naive way of processing blocks is to perform sequential processing of valid state transitions + valid signatures. Every time we verify an individual signature, we incur a significant performance cost. Instead, we can verify state transitions sequentially and in process, aggregate batches of signatures which we only verify once at certain checkpoints. This can help drastically improve the speed of sync for mainnet. Our team has been working on this critical feature and the improvements are noteworthy. Developer Afri Schoedon, who coordinates the eth2 multiclient testnets, has recently published a report on Prysm’s performance in the Altona testnet published here.

With full signature verification, Prysm is on par with Lighthouse, the rust implementation of eth2. We are confident we will continue improving our benchmarks as we approach a mainnet release.

Remote Signer Reference Implementation for Prysm

As part of our effort to revamp how validator accounts work in Prysm, we have been working on getting great reference examples for anyone running our client to understand how to use our tools. In particular, many users want to run their validator clients in the cloud but want to run a remote signing server elsewhere that is able to securely store keys and sign data as needed. In the Prysm validator client, we initialize a gRPC client that connects securely to a remote signer as desired by the user. For reference purposes, we created an example remote signer server on Github here, licensed under Apache 2. You can also write your own remote signer servers in most popular programming languages that support gRPC, as long as your server implements the required API schema for a remote signer defined in Prysm here. We will be adding comprehensive documentation to regarding how to use the remote signer and all other accounts v2 related functionality.

Upcoming Work

Rate Limiting and Peer Scoring

Rate limiting and peer scoring refactoring work has been started. Our aim is to make sure that Prysm beacon node is more robust, and is capable of performing its duties in presence of slow, broken, or even malicious peers.

The current rate limiting system is already capable of banning peers that send requests too aggressively, and we are extending it to become more sophisticated in how good/bad peers are detected and what actions are taken to prevent misbehaviour. In addition to banning, we are devising a reputation system that will score peers based on a wide range of different performance metrics (average blocks returned, blocks successfully processed etc), throttling peers that violate carefully selected thresholds. Peers that persistently violate such thresholds will be banned for longer periods of time.

If you are interested in tracking the progress of new peer scoring system please follow the #6622 issue on GitHub.

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Eth 2.0 Dev Update #54 — So Close to Official Multi-client Testnet! was originally published in Prysmatic Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Eth 2.0 Dev Update #51— v0.12 testnet on the horizon

New v0.12.1 Version Testnet Incoming

With the recent releases of v0.12.0 and v0.12.1, the Prysmatic Labs team has been hard at work to align Prysm with new scoped features and bug fixes. We are happy to announce that the restart of Topaz testnet for v0.12.1 is just around the corner. If you have any preference on what this testnet should be called, please suggest that in our Discord, we’d love to hear them! This release contains the long-anticipated update to the BLS signature standard. It also contains a tweak on how inactivity penalty is applied to performing validators. Yay! No more seeing balance decreasing for the performing validators during those tough times. We have also updated our libp2p Gossipsub to version 1.1. This version contains great features on addressing various attack vectors. v0.12.1 will be the target for multi client testnet and mainnet assuming no bugs found during this test period. The amount of v0.12 work done on Prysm can be tracked via #5935 and #6076. Be sure to keep an eye out as we announce the testnet restart plans early next week.

Community Feedback Survey Results

Thanks to some feedback from Ethereum Foundation researcher, Protolambda, we started thinking carefully about the biggest pain points for contributing to the Prysm project and decided to run a small community survey asking everyone about their experience running a node, interacting with our team, and reading our documentation.

And it ended up yielding excellent results! The biggest feedback comes from the pain of learning and using our build system, Bazel, for external contributions or to integrate Prysm into other Go projects. Other feedback revolves around account management: i.e., how do I move my account to a different machine? How do I change my password? How can I run tens of validators on one computer?

Another highly requested feature was being able to run any sort of eth1 node with Prysm, such as Nethermind, which is now fixed in our latest master branch! Overall, people’s experience with our team seems fairly positive, but integrating with our codebase, testing it, and understanding its complexity received mostly negative reviews. Code health is going to be a big priority for us going into the mainnet release. If you still have some feedback you want to give or didn’t get to fill out the survey, please take a look here.

Merged Code, Pull Requests, and Issues

Full support for Nethermind + other eth1 node implementations for Prysm

Previously, Prysm nodes required access to both an http eth1 endpoint as well as a websocket one. This limited the options of which eth1 client to use for Prysm to only go-ethereum, which was unfortunate as we noticed many users requesting Nethermind support. We took this feedback seriously and refactored our eth1 code to only use an http endpoint, meaning you can provide any endpoint you want to your node as long as it can read smart contract logs from eth1. This allows users to run Nethermind, Besu, go-ethereum, or parity for Goerli eth1 testnet support!

Additionally, the Prysmatic Labs team runs a goerli eth1 endpoint for users to try, which is enabled by default in Prysm beacon nodes as In the spirit of decentralization, we are encouraging all our users to either run their own eth1 node, or use a reliable third party provider such as Infura. We have updated our instructions for how to do this in our official docs portal here.

Full support for building, testing, and running Prysm with the native Go tool

Although Bazel is great for our team and monorepos in general, it has a massive learning curve for anyone wanting to contribute to the codebase with pure Go. For example, with our build system, you’re not able to:

  • Run go get, go build, or go test
  • Use a lot of popular IDEs or text editors except for VSCode with a Bazel plugin or IntelliJ IDEs
  • Import Prysm packages into other Go projects easily

For an open source organization, not providing these benefits is a huge no-no. Instead, we decided to offer pure Go support. This is huge because now anyone can do a `go get` and start developing in their favorite text editor, or importing Prysm packages into their Go projects. We’re really curious to see what people will build! You can also run tests in pure Go with `go test -v ./…`. We have updated our contribution guidelines on how to develop Prysm with pure Go here. All the hard work that went into this feature was done by our teammate Preston Van Loon. You can check the pull request which enabled pure Go support for Prysm here.

Initial synchronization improvements

Several important init-sync related PRs have been added, with the main purpose of improving performance and robustness of the process.

The most important step towards better robustness is #5975, where system was challenged to overcome a very long period of skipped blocks (a week of skipped blocks)

The current implementation is capable of handling such cases, even better, all this without excessive network requests and without overwhelming peers (load is balanced using weighted round robin routine).

Another important improvement was related to how batch requests are processed: previously, the initial synchronization queue was epoch based, meaning that it relied on the number of blocks per epoch, when deciding on the size of block batches to sync. This limitation is fixed now, and the queue works with batches of arbitrary size (64 blocks, currently). These and other improvements allowed for better blocks/sec values (85+ blks/sec on average). Everyone is invited to test the updated synchronization; do not forget to use the ` — dev` flag to have all the optimizations enabled:

Upcoming Work

Code Health Improvements

As we are heading towards the milestone for large scale multi client testnet, it’s natural there will be more eyes looking into the Prysm codebase and more newcomers wanting to contribute. That’s a great thing! Which means It becomes very important that Prysm’s codebase maintains a high standard for readability and clarity. We are in the process of updating the docs portal, user guides, and general node logging for better user experiences. We are also refactoring some of the most used prysm service for easy reasoning. Expect lots of great changes on those items in Q3. Also thank you to Protolambda for providing some awesome feedback, and thank you to Quantstamp for pointing out some great issues so far

Preparations for coordinated multiclient

With the restart of v0.12 Topaz testnet, we are also one step closer to large scale coordinated multi client testnet. Our plan is to relaunch Topaz, which will serve as a good testbed for other clients to also sync with v0.12 of the specification. Although it will be majority Prysm at launch, the goal is to allow other client implementations to easily run on the testnet as well. We will keep you all posted on our progress.

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Eth 2.0 Dev Update #51— v0.12 testnet on the horizon was originally published in Prysmatic Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Eth 2.0 Dev Update #46 — “Slasher running & Integrated fuzz testing”

Eth 2.0 Dev Update #46 — “Slasher running & Integrated fuzz testing”

Our biweekly updates written by the entire Prysmatic Labs team on the Ethereum Serenity roadmap.

🆕 New Testnet Updates

🔹Slasher running in production catching slashable offenses

Our slasher implementation is now capably working in production, running in our cloud deployment in order to catch all slashable attestation events happening in the Prysm testnet. That means if your validator is wired up to do a conflicting double vote or a surround vote, we’ll catch you and you’ll get forcefully ejected from the validator set and gradually slashed! This was a huge milestone for the team and something that we will continuously improve throughout the next testnet restart. We were able to simulate and catch another slashable event, this time a surround vote as seen here: At the moment, however, our slasher only detects slashable events as they happen and is not performing historical chain detection. That is, if a slashable event happened at slot 1, our current slasher won’t catch it. Ivan from our team is now working on plugging in historical chain detection and getting it functional.

Monitoring slasher through Grafana

📝Merged Code, Pull Requests, and Issues

🔹Able to build all Prysm libraries + dependencies from source, including low-level cryptography

One of the key principles at Prysmatic Labs is to create builds as hermetic as possible. What this means is that building Prysm should be exactly reproducible across different machines. Prysmatic Labs achieves this goal by removing dependencies on system level libraries and building everything from original source code. This way, we can ensure that the same version of a library or dependency is used across builds. For example, building Prysm from source downloads a specific version of golang compiler, whether or not the host machine had that already installed! One of our most recent challenges was to build our BLS cryptography library, which is primarily c++, within our go project. While we have been using precompiled library archives for BLS, some differences in linux OS versions have caused issues which could only be resolved by compiling from source. If you’re on linux, try building Prysm with — config=llvm to build BLS entirely from source files.

🔹Fuzz testing Prysm’s core functions is now working

Continuous fuzz tests on

Our friends at Sigma Prime have been working hard on differential fuzz testing across all of the Ethereum 2 implementations for core functionality. Inspired by their achievements, we’ve taken to adding continuous fuzz testing as part of our continuous integration pipeline. At present, we have 7 fuzz targets which are tested with coverage based random inputs! In other words, the fuzz testing platform uses a set of valid data as a seed corpus, mutates the inputs slightly, executes the target with the new input and checks if a new code path was discovered. This type of testing can discover unknown edge cases which crash the application. Just this week, the fuzz testing has found two fatal bugs within Prysm’s core methods. One of these bugs involved populating a certain cache with certain inputs that would lead to an overflow. This issue is truly something that we could not have foreseen and would have been extremely difficult to reproduce without the input data. As we advance towards mainnet, we will be embracing fuzz testing for as much of the application as possible, especially mission critical methods which are exposed to unknown inputs.

🔹Revamping of beacon state management

Currently in our design, a node prunes all states prior to the finalized checkpoint, while this design was easy to implement, it’s neither fully desirable nor sustainable. The nodes no longer store historical states, and the nodes do not have the capability to retrieve state that’s prior to the finalized check point. With that limitation, it comes with the following downfalls.

It limits debuggability. If a user wants to retrieve historical data to debug, it would not be able to do that. This increases the disparity between the regular node and the archival node. We have seen complaints when an archival node’s functionality was applied on a regular node. This disables a node to perform state sync. With state sync, having access to some amount of historical state will be important. Finally, this increases man testing power to test both regular and archival nodes.

The current proposal is to implement a wrapper service on top of the state DB with separation to maintain both hot and cold sections of the state. By efficiently splitting the states into two sections. In both hot and cold sections, full BeaconState objects are only stored periodically and intermediate states are reconstructed by quickly replaying blocks on top of the nearest state.

To replay blocks , we omit redundant signature checks and ssz hashing calculations. For the hot section, the full state where blocks are replayed from are referred to as epoch boundary states. The frequency at which the hot section stores BeaconState is fixed to one per epoch. For the cold section, the full state where blocks are replayed from are referred to as archived points. The frequency at which the cold section stores BeaconState is configurable via cli flags.

This proposal enables flexibility for end users to explore space and time trade-offs. Frequent archived points occupy more disk space but accelerate the loading and computing of historical states. Whereas infrequent restore points occupy less space but also slow down the loading and computation of historical states.

Follow the remaining issues for bugs follow up and further optimizations.

Wanna give a shout out to Handelaar and Butta from our Discord. They have been on the front line with me testing out this feature in the wild. While this has been buggy, we are seeing good results! Thanks guys! ❤️

🔹Separating initial sync block fetching from block processing

This week we have merged #5096 into master, which features a major headway in initial sync refactoring. This update introduced a concurrent model of syncing, when blocks are processed as they are being fetched. Number of issues were addressed as well (stucking on a single block, or looping on a range of blocks without any progress).

Updated sync process is not fully complete, as it still has a number of quirks and inefficiencies, that’s why we have put it behind the flag i.e. by default we still sync using previous implementation. During the next week more polishing will be done to the algorithm, to improve correctness, performance and memory footprint.

🔹Dynamic attestation subnets complete

A huge blocker for multiclient testnets that are large in scale is reducing network spam of validator attestations. If there are 100k validators, your node will be receiving a huge amount of attestations per second via gossipsub p2p. Instead, we should only be subscribing to unaggregated attestations our validators care about in order to put into blocks. We can do this by splitting this up into committees our validators are part of. That is, if our validators are part of committees 3 and 4, we should only care about listening for unaggregated attestations for those committees. To do this, we can subscribe to dynamic subnets for attestations. We have completed this feature and it is now behind a feature flag — enable-dynamic-committee-subnets. It will take precendence in our next testnet restart.

Imagine the biggest flame (1GB) above all goes away

P2P discovery using attestation subnets

Now that we subscribe dynamically to only certain gossip topics for unaggregated attestations our nodes care about, then we have to ensure we have peers that are on these subnets. Nishant from our team took the initiative of modifying our discovery protocol, discovery v5, to search for peers that contain the attestation subnets within their ENR record, which is a unique identifier for peers via p2p that contains important metadata for eth2. We have wrapped up this feature and tested it locally via a development chain with beacon nodes that could discover each other successfully based on the topics they cared about. You can see more in the PR #4989

🔜 Upcoming work

🔹Optimizing attestation aggregation in prysm

The purpose is to aggregate validator votes (attestations) efficiently, to maximize validator’s profitability. The difficulty arises from the fact that it is a known NP-hard problem and we need to come up with a heuristic that will be both efficient and performant.

Currently, we use the most naive strategy and are looking to improve it significantly in preparation for mainnet. There are a number of proposed strategies, and during the course of next few weeks we need to decide on which strategy to use and come up with a benchmarked implementation.

🔹Testnet restart with spec version v0.11

We have been heads down the last few weeks revamping Prysm with latest spec releases. First v0.10.1, then v0.11.0. Given both releases have breaking changes that make it incompatible with current testnet, we will perform a testnet restart once all is done and well-tested. We are super excited about the new testnet restart after the current testnet has been running for over three months.

The v0.11.0 is also the current target for multi client testnet. This release was significant as it represented a post audit phase 0 spec. The two releases contains limited consensus changes, it mostly contained networking changes with new features, DoS protections and simplifications.

Follow the remaining v0.11.0 todos in #5119 and top priority testnet issues in #4781


🔹Eth2 Research Explainer Paper Published, With Formal Proofs

After a significant amount on behalf of the eth2 research team, a formal paper on eth2 known as “Combining GHOST and Casper” has been published on ArXiv here. We are thrilled to see such a great write-up on all the work that went behind creating the specification for eth2, and we believe this will open up contributions into eth2 to the wider research sphere.

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Eth 2.0 Dev Update #46 — “Slasher running & Integrated fuzz testing” was originally published in Prysmatic Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Eth 2.0 Dev Update #44 — “More Optimizations”

Eth 2.0 Dev Update #44 — “More Optimizations”

Our biweekly updates written by the entire Prysmatic Labs team on the Ethereum Serenity roadmap.

🆕 New Teammate

We are happy to announce after receiving a very talented batch of applicants, we have hired our new full-time teammate at Prysmatic: Victor Farazdagi!

Victor previously received his master’s degree from Georgia Tech, worked as a lead Golang engineer at Status, and has experience extending the go-ethereum client’s LES and Whisper protocols. We believe his skillset will ramp up the quality of our Prysm client and offer new perspectives to solving challenging problems in the eth2 research space, welcome Victor!

📝 Merged Code, Pull Requests, and Issues

BeaconState redesign: partial copy and shared references

The beacon state management is the root of many of our performance bottlenecks since configuring the testnet to run on a mainnet scale. As such, we have been working diligently to find ways to shave off kilobytes of redundantly allocated data to improve sync times, CPU load, and overall memory usage. Go, the programming language Prysm is written in, has no concept of fully immutable data structures. What this means is that two or more copies of an object may reference the same underlying object so when you mindfully mutate a copy of the object, you might be mutating another copy in some unknown routine! The only way around this is to provide a deep copy of data structure, however this becomes expensive with the beacon state as it is copied dozens of times throughout various routines and is relatively large and growing over time. So we have designed a mechanism by which we utilize a hybrid of deep copy, shallow copy, and copy-on-write strategies for sharing beacon states across multiple routines with minimal overhead.

Multiple beacon states can share references to large allocated arrays.

This feature along with the recent full beacon state management rewrite PR have been an incredible improvement on memory and CPU usage. Check out the full design doc here and implementation PRs 4699 and 4785.

Comprehensive storage improvements

Prysm has an experimental flag for greatly improving sync times by storing more states in memory than on disk. One major bug we found was that we were holding onto many of these states in memory after initial sync was complete. These states only served a purpose during initial sync and often accounted for gigabytes of allocated RAM. This issue was easily resolved upon discovering it.

User reported memory dropped significantly after PR 4817!

Other observations we have seen are that the cached states in memory can sometimes grow to the point of exhausting the syncing node. This issue is being tracked in issue 4813 and should be resolved soon. After resolving that, we can enable this in memory state management during initial sync for all nodes by default. For now, it is only enabled with a feature flag `— initial-sync-cache-state`.

Fork choice radical improvements

As we have mentioned in previous update, switching from a naive fork choice implementation to proto array fork choice implementation has significantly improved fork choice performance and lowered CPU utilization. With the recognization on this significant improvements, we have gone on to make proto array fork choice the default fork choice on the Prysm beacon node. See #4778 for further detail. Doing so, we also got rid of 4000 lines of code doing so. Hooray for code health improvement!

In parallel, we also have been making micro improvements in our block chain service code to better use this new fork choice implementation. One good example is a node often checks the validity of an object by whether the corresponding block has been successfully processed. The naive way is to check whether the block exists in DB. Another less known way but a more efficient way to check if the block exists in a fork choice store. Check fork choice store is ~10x faster than DB. See #4821

BenchmarkHasBlockDB-12 381 ns/op

BenchmarkHasBlockForkChoiceStore-12 40.0 ns/op

Better attesting summary reporting

Stakers often ask “Why aren’t my validators making $$?” That is a fair question but also a hard one to answer without dissecting into details on how validators attested. We updated the validator performance API to include useful information such as `inclusion_slots` and `correctly_voted” fields. This enables stakers to query information like the slot of when the validator’s attestation got included in the chain, or whether the validator correctly voted for source, target and head. We started working on this by updating the API definition #103

We then updated API implementations to support this feature #4845

With the log:

“[2020–02–12 12:48:00] INFO validator: Previous epoch voting summary correctlyVotedHead=false correctlyVotedSource=true correctlyVotedTarget=true epoch=7619 inclusionDistance=1 inclusionSlot=243747 pubKey=0x875ce51a3821e5a2”

Which can be read as:

“Validator 0x875ce51a3821e5a2’s attestation from previous epoch was included in block at slot 243747, it took 1 slot for this attestation to get included. The attestation correctly voted for source, target and incorrectly voted for head

Another log that showcase summary for total validators:

“[2020–02–12 12:48:00] INFO validator: Previous epoch aggregated voting summary attestationInclusionPercentage=0.64 correctlyVotedHeadPercentage=0.81 correctlyVotedSourcePercentage=1.00 correctlyVotedTargetPercentage=1.00 epoch=7619”

Which can be read as:

“Out of all the validators in this client, 64% of their attestations got included in blocks. Out of the attestations include in the block, 100% voted correctly for source and target. 64% voted correctly for head”

Safer concurrency in Prysm

There are thousands of things happening in a beacon node every second, with validators submitting blocks, peers requesting and receiving data via p2p, making sure fork choice is being applied, tons of reads and writes from caches and the database, etc. Go is an excellent language for blockchain clients because of its powerful concurrency primitives such as channels and goroutines, making it easy to spawn concurrent processes which communicate with each other easily. A common requirement when designing concurrent processes is to declare locks around certain values when accessing or writing to them. Unfortunately, excessive use of locking can lead to higher latency and other problems if the code is poorly designed. We have been working hard to ensure safer concurrency in Prysm through careful use of locks when we need them and ensuring there are no race conditions throughout our code base. We improved concurrent access in fork choice here: #4784 and attestation concurrency here: #4833 which we believe makes Prysm safer under conditions of high validator per beacon node load.

Block tree visualization

Team member Terence has implemented a graphviz visual display of Prysm’s fork choice voting algorithm in real time. This tool has been great for debugging and for satisfying our immense curiosity of the current health of the testnet. Visit http://localhost:8080/tree with your local beacon chain node to see this block tree for yourself!


Slashing operations pool for Prysm beacon nodes

Our testnet previously was unable to insert slashings into a block, but that is soon going to change with our recent slasher progress! Beacon chain nodes now have a slashing pool that proposers can query for any pending slashings that need to be put into a block.

Our slasher is currently undergoing a reorganization (more details later), but once that is complete we will see beacon nodes adding slashings to the pool for insertion into blocks!!

🔜 Upcoming work

Real slashing occuring in the Prysm testnet

We’re super eager to ship our slasher implementation, which is essentially a watchtower for beacon nodes that detects every block and vote that comes through the node and is able to discern slashable events. If one is found, it then relays a packaged proposer_slashing or attester_slashing object back to the beacon node to be applied to validators via the state transition function. Unfortunately, slashing is still a super expensive operation and our current slasher design needs a fair bit of work before we are comfortable running it in the testnet. Now, we are working on a tracking issue that improves slasher via careful refactoring, testing, and benchmarking

We believe within the next two weeks we will be able to confidently start including slashing operations in beacon blocks in our running testnet.

Comprehensive bug fix roundup, optimizations, and multiclient testnet work

We need multiclient to be stable before we can consider a mainnet launch of eth2, and with various teams improving their production readiness, being on the latest spec versions, and aligning on networking, we believe things are ready to ramp up the effort. We opened up a tracking issue above to get all the missing TODOs done before focusing on a multiclient effort over the coming weeks. Not only is it important for another client to sync with us, but also for that client to product blocks and successfully contribute towards finalizing the chain. Having an evenly distributed multiclient testnet in terms of nodes and validators would also be a worthy goal to aim for before the real launch, and we’ll be working with other teams to achieve this goal carefully.

More memory optimizations

As we are looking forward to apply more optimizations, to lower the memory and CPU usages when running Prysm nodes. The biggest bottleneck we are working to solve is utilizing more shared references for our commonly used arrays. Constantly copying large arrays leads to a larger amount of memory required to run the beacon node, and can lead to frequent OOMs. Another method we are looking at is pooling together and re-using memory from already destroyed objects so that we do not need to allocate more memory when creating new objects. Since go runtime does not release memory back to the OS immediately, when frequently creating objects with large memory footprints this can lead to OOMs; it would be good to recycle this already allocated memory instead. Expect more update on this next time!


New awesome features added to Bitfly’s eth2 block explorer for the Prysm testnet

The awesome team at Bitfly has been working hard on adding new features to their fantastic ETH2.0 block explorer. They now have support for a few charts about the beacon chain, including great stats on block proposal history, active validator count, and more! They also have a great realtime block visualization and a validator leaderboard! 🎉🎉🎉Thank you Bitfly!

New awesome features added to etherscan eth2 block explorer for Prysm testnet

Another excellent block explorer for our Prysm testnet is Etherscan! They’ve added a really useful tracking site for ETH1 deposits, very neat for anyone curious on the kind of deposits being made regularly.

On top of this they have a dope finality history tracker that records historical finality history so it’s easy to tell when finality or performance issues occur. Very useful for our testnet while we’re working on fixing current finality issues :). It also shows the gaps in finality at any given moment of the chain. Thanks Etherscan!

See us at ETHDenver! 👋

Several members of the team will be around ETHDenver this weekend. Come find Raul, Preston, Terence, and Ivan during the event! Let us know if you plan to build something ETH2 related to claim the Ethereum Foundation sponsored additional bounties!

Interested in Contributing?

We are always looking for devs interested in helping us out. If you know Go or Solidity and want to contribute to the forefront of research on Ethereum, please drop us a line and we’d be more than happy to help onboard you :).

Check out our contributing guidelines and our open projects on Github. Each task and issue is grouped into the Phase 0 milestone along with a specific project it belongs to.

As always, follow us on Twitter or join our Discord server and let us know what you want to help with.

Official, Prysmatic Labs Ether Donation Address


Official, Prysmatic Labs ENS Name


Eth 2.0 Dev Update #44 — “More Optimizations” was originally published in Prysmatic Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.