Welcome to the first article in the series of publications on the Solana validators performance research by P2P Validator. In our opinion, performance of the Solana network validators is one of the most important aspects which determine the network growth and sustainability. Our team has done a deep dive into this topic and we want to share insights gained to benefit the Solana community.
The research is devoted to the analysis of the two most important metrics reflecting Solana network validators’ performance: downtime duration (node delinquency/unavailability duration) and skip rate (measuring how frequently a node fails to produce a transaction block which is subsequently confirmed by consensus on the network).
In this article we reveal the first part of the research findings regarding analysis of downtime. You can find Part 2 here, covering block skip rate analysis results.
All data used for analysis in the research were obtained from publicly available sources such as Solana JSON RPC API, Solanabeach API, Validators.app API and are relevant for Mainnet beta epochs №194-236 unless another epoch or time period is explicitly specified.
Solana is a relatively new (went live in March 2020) public high-performance distributed blockchain platform curated by Solana Foundation (non-profit organization headquartered in Geneva, Switzerland) along with professional blockchain developers, organizations and individuals running validator and RPC nodes and DevOps specialists from all over the world who are dedicated to the decentralization, growth, and security of the Solana network.
Solana is one of blockchains that aims to be fast and scalable, without compromising its security or decentralization. Its theoretical throughput limit of 50k transactions per second (TPS) which is twice more than VISA’s limit, which means it can be used for many real-time applications in various business areas. Solana mainnet has already handled more than 35 billion transactions with current throughput exceeding 2000 TPS (see Figure 1) due to high demand for its capabilities and various use cases including: ultra-fast on-chain payments, token creation and distribution, staking through delegation to network validators, smart-contracts, NFTs issuance and trading. Solana ecosystem also provides many different DeFi services such as decentralized exchange, token swaps, liquidity farming and bridging ensuring cross-chain interoperability with other blockchains.
There are currently more than 1000 independent validators and 800 RPC nodes (see Figure 2) which comprise a physical layer for above mentioned functionality while making the network highly secure and decentralized. Each validator supports the network's operation by providing high-end hardware resources and properly configuring their systems to keep the network running as fast and smooth as possible.
Validators receive SOL tokens from stakers, participate in the consensus-based process of transactions validation, get rewards proportional to delegated stake amount and distribute these rewards to stakers (proportionally to staked shares) charging a variable commission. The more stake is delegated to a validator, the more this validator (and its delegators) earns and is more frequently chosen to process new transactions on the ledger and, thus, exposed to greater hardware and network load. Thus, on the one hand, validators are economically motivated to keep their hardware and software running without interruptions, and, on the other, to timely update Solana software and to improve their nodes and network connection as their stake and the Solana network load increase.
It is normal for a node to be temporarily unavailable/offline sometimes as every technical system needs periodic maintenance and reconfiguration. Typical reasons for server unavailability are usually quite simple such as planned reboot to update host configuration or software, an emergency (power outage), network problems in the data center or at the provider.
The longer a node is unavailable, the fewer staking rewards and transaction fees it receives. Staking rewards are paid proportionally to node’s vote transactions count which it cannot post if it is offline or functioning incorrectly. Validator downtime negatively affects its delegates’ rewards, which is why one should consider checking validator recent downtime duration history before delegating to it.
During periods of downtime an unavailable validator is assigned the “delinquent” status which can be checked using Solana CLI solana validators command or by parsing corresponding json response (solana --output json validators). By constantly fetching statuses of all validators on the network it is possible to measure delinquency periods durations which is a good approximation for downtime duration for further quantitative analysis. The downtime data analyzed is available through the public Redash dashboard.
There are many factors influencing downtime duration and these are typical ones:
Despite most of these factors can not be measured directly, we have managed to collect and analyse some important on-chain data related to the topic, which allowed to quantitatively describe several aspects regarding Solana network nodes unavailability such as downtime duration statistics over time, its variability across nodes as well as duration of node software updates.
Here we illustrate retrospective downtime statistics of Solana nodes that were active in the period from epoch №209 (5th of August, 2021) to epoch №236 (17th of October, 2021). Historical data allow to reveal trends in the dynamics of downtime making it easier to understand the normal behavior of the metric as well as to identify abnormal fluctuations.
The descriptive statistics for downtime duration by epochs are presented in the Figure 3 below. Quantile values of 5%- and 95%-level reflect the maximum downtime among the top 5 and top 95 percent of validators, respectively, for each epoch. Average downtime is the simple arithmetic mean and median defines a downtime duration which divides the top 50 and worst 50 percent of validators.
As can be seen from the chart above, typical average downtime duration for a node is around 1.5 hours, which is quite low, while median downtime duration is almost always zero (which means that most nodes usually don’t experience shutdowns). Also there were several epochs (№214, 223 and 234) with high downtime duration upticks mainly due to simultaneous upgrades of Solana software version. Epoch 223 is especially interesting as it is known that on 14 of September, 2021, the Solana network experienced a severe overload which led to the network halt, and after a successful network restart almost all the nodes had to update to a new Solana version with the necessary fixes.
As many factors affect downtime duration, it varies greatly across validators within the same epoch. The dispersion measures indicate the metric’s spread magnitude which is slightly changing over time as shown in Figure 4.
It can be seen from the chart above that downtime duration dispersion across validator nodes is dropping slightly over time which indicates that validators, on average, have both lower downtime durations and lower deviations of the metric from the mean.
Since the leading validators with a large stake amount take much more financial risks compared to the smaller ones, their nodes' technical characteristics are far better than of the majority. Therefore, it makes sense to compare performance of the superminority set of validators (the minimal set of validators that together control more than 33.33% of the total stake) with the rest falling into the supermajority set with 66.66% of total stake (see Figure 5).
As the charts above represent, the supermajority is usually much worse in terms of average downtime duration, especially after epoch №220, especially during hard times like epoch №223, when the Solana network halted and most validators had to perform major software updates.
In contrast, superminority validators (especially P2P Validator) have an average downtime duration and an average number of downtimes (see Figure 6 below) that are much lower than for the supermajority, and there is a much smaller probability that a validator from the superminority set is offline for more than 5% of total epoch duration (see Figure 7 below).
As it was described previously, downtimes may happen due to Solana node software updates as well as due to hardware upgrades and unexpected halts. The available on-chain data allows to distinguish between downtimes related to software updates and related to other causes and compare downtime duration distributions for the supermajority and superminority groups of validators.
According to the distributions of downtime duration not related to software updates (see Figure 8), validators groups are quite similar apart from the fact that supermajority validators are more likely to have very long outages that greatly increase the average value of downtime duration (69 vs. 34 minutes for the superminority group). It should be noted that even if the P2P Validator goes down (or delinquent), on average it happens for an extremely short time of 1.5 minutes.
Concerning downtimes due to software updates (see Figure 9), the distributions for the groups differ considerably: for the superjmajority group there is much more variability in downtime duration when compared to the superminority and again supermajority validators frequently have much longer update times leading to higher average (195 vs. 76 minutes for superminority group). Superminority validators including the P2P Validator demonstrate high consistency in update duration presumably due to specific administration standards developed by professional engineers who operate these validators.
Different Solana node software versions vary significantly in the complexity and duration of the installation process, which directly affects the downtime duration associated with updates. Figure 10 below shows the average update time of Solana node software versions by validators from the supermajority and superminority groups.
Of all the most used versions of Solana node software, the update to version 1.6.25 took the longest for both supermajority (4.5 hours on average) and superminority (3.5 hours on average) validators. Long updates to versions 1.7.11 and 1.7.15 were performed only by validators from the supermajority group and took approximately 2-3 hours to complete. Overall, validators from the superminority group usually perform the updates significantly faster ensuring less rewards losses for them and their delegators.
Downtime duration is a very important metric as it reflects Solana validators operators efficiency and influences rewards received by validators and their delegators as well as overall network’s stability and security. Solana Foundation and the network validators do everything they can to improve performance of nodes and quality of software that control nodes operation, and we can say with confidence that they do it very well, especially validators from superminority group thanks to the experience and professionalism of DevOps engineers.
Authors of the report would like to express gratitude and appreciation for the P2P Validator team whose guidance, support and encouragement have been invaluable throughout the research. We would also like to thank Stephen Akridge, co-founder of Solana, Ruud van Asseldonk, software engineer at Chorus One, and Robert Dörzbach, product manager of the Solana Beach, for helpful advice, comments and corrections.
Information presented in this report and referenced sources are for educational purposes only. It is not financial/investment advice. Seek a licensed professional for any financial advice. Authors of the report made every reasonable effort to ensure the accuracy and validity of the information provided. However, as price points, conditions, and information are continually changing, authors reserve the right to change at any time without notice, information contained in the report and make no warranties or representations as to its accuracy or up-to-dateness.
Authors of the report are employees of P2P Validator company which provides professional services and consulting for highly secure non-custodial staking across more than 25 blockchain networks, including the Solana network with mainnet and testnet validator nodes as well as RPC nodes. Therefore, P2P Validator is not a neutral party with its own business interests in the Solana ecosystem. Nevertheless, authors did their best to make the report as objective as possible with the main purpose in mind being to educate and inform the community.
P2P Validator is a world-leading non-custodial staking provider securing more than 4 billion USD value from over 20,000 delegators across 25+ high-class networks. We are early investors in Solana and have supported the network from the first block taking part in all stages of testing and voting.
Stake SOL with us: p2p.org/solana
Get the latest posts delivered right to your inboxSubscribe
Data analyst @ p2p.org (previously dentsu inc.), graduated from Moscow State University of Psychology and Education, PhD, father, blockchain enthusiast & investor.
<p></p><p>We have created a step-by-step guide to help you stake Elrond (EGLD) using the Elrond web wallet.</p><h3 id="create-your-elrond-wallet">Create your Elrond Wallet</h3><ol><li>First you need to go to the wallet <a href="https://wallet.elrond.com/">website</a>. Once there you will be prompted to begin the process. Start by pressing the “Let’s get started” box:<br></li></ol><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/-DMtPCmaZa9_qX6EyjGjPYxJLh_7uwx_I6Hat_r8Ua5gW2Uddz41bablvlNxKEnmPli80f0GR5aG-hzCkw5J5HEPy4dLxJvppunG3S-Cm-Da0MzHjSmfY-s_1U-3xsG7Q2arb5u8" class="kg-image" alt loading="lazy"></figure><p>2. You will then be presented with some information about the wallet. Be sure to read it, and once you are done press the “Next step” box and finally the “Thanks for the info” box:</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/5H-OsTIShKlHy_eRYfun58-P6PjwyHQdt2nudHpvuZN7uSudB0SHzSQJdmqOgi4s6sxmxrW0gvyRzx1HSG37k3Z0h9aRdRO8_vM20j-CWeoA_boynNsyJ8MKMIBWUYAAJa0rKbBY" class="kg-image" alt loading="lazy"></figure><p>3. Now you can click on the “create wallet” box:</p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/NmV6ejIv_3QeyJAujlY4tvTC-I6WxBLYFu5VD0XWGqvRCZz4Lf9hWmmQilIBNEx4bxBWAr0DoUSlNME74MS81KVZh3erUa65zdZJ7ozPJBiKR9Z7w3y5gpfTh2E9WXT84dC-ujvf" class="kg-image" alt loading="lazy"></figure><p>4. Read and select both prompts to continue:<br></p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/13iL-5Uq-W5DcR7td3AabOdfEmC-MHbNqvSsIdWdMb9IfZ0_YcBCYUMVAtKtaRTuwNiFVAbsh2DeG2-KOiOQc35WnvUbukjL-5lozHHy1Z_QWTPPoTB0MtNg_rE067MYf44wJqtH" class="kg-image" alt loading="lazy"></figure><p>5. Write down your seed phrase in a secure place that only you can access. This is used to gain back access to your wallet if you lose it. If you lose the seed phrase you may lose the funds stored in the wallet. Warning: Never share this with anyone.</p><p>6. Enter the words in the corrected order as prompted by the screen: </p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/9ubcwmhitRnN9ndfhmmK06k1POn9fT9oRmoOJRRvkdrKgA5Du9aLVj9E404Jb6LZj90ags3L40VJOv5IsiDNN_JA-TTYNZ-38mWZWzt6otT9YCmUnT5PZqIY-KkVVWpvAaenAH8r" class="kg-image" alt loading="lazy"></figure><p>7. Next, create a password for your wallet (it must contain at least one number and a special character):</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/Nhei9GK81gVRFlABqHBh7xWuRigxOpqWMlyDAX90GLzTev1KEV6VY9HXtepPLZicpSSrxByFhi9w28FtxYlqEdQ-Xf1tq2XoK-G6qGCpz6gDrKlXhGyAH386g0K3jNMGvuRBc0B_" class="kg-image" alt loading="lazy"></figure><p>8. And you are all set! The wallet has been created. You might notice that a file was downloaded to your device. This file is required to access your wallet, so keep it safe. If you lose it you can recover access using the seed phrase. </p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/_2kTJiKkaLCsqiudVKUYXpTJErWhi_38c5EKmV9SnOiUzHWAbdF65QeF99xVv8Thnqb2JPA7EmFvsYnm-ouxYIwYQki54mNa8hXZAjCc4707bulPEZUKLWlbvfRrCZv100oG0GtE" class="kg-image" alt loading="lazy"></figure><h3 id="stake-egld">Stake EGLD</h3><p>1. First you need to go to the Stake section of the wallet. Select Stake on the Menu on the right-hand side of your screen:</p><figure class="kg-card kg-image-card"><img src="https://lh3.googleusercontent.com/zmLgPY8D122Ua0Y-0ZTlcGmXjIVWDKH83Ni2OFVqebd2Ue2g9sboawBE5MZjakqa2LOPdI6su65pZn8pBbaHzzJNS5OeN0dGV9oGAZKDv6h57eXMKvU4Ivmyaxfrj57sFFYliBU6" class="kg-image" alt loading="lazy"></figure><p><br></p><p>2. You will be taken to the Staking Dashboard, where you can find information about your stake and rewards (once we complete the process). Once there press “stake now” and a new window will open:</p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/Si7lCVtJz_tyvUE01Ky9oHituMgk-O8Gf65oichfhmKxCNWhzmziMsDd3BuWhP4yMqMrP9cpzRSnpyZz3XEnVxW-lxRtE1Hne7FyvHZvSl7y5yFhKUnQ6eCm9kTSgq8_wlUYPKQG" class="kg-image" alt loading="lazy"></figure><p></p><p>3. Find P2P validator from the list and select it. <br></p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/EnDGlvUwkzog5m5k-qzJBi7SROJzK_gKc3aSLpmvotSKD4AbLeHPAwAw0o6zPa9QmVqgWybDNoqOeC_iwvTQG7kquskLqXCg5DuwTHgPqGS-Cg3JE7LmEp-2ecwugC48LWCR9Q7_" class="kg-image" alt loading="lazy"></figure><p>4. Input the amount you want to stake and press continue. Please note that a minimum of 1 EGLD is required to start staking:</p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/K8D0BR_mgU-1nlybIN5i4UQZMCOKlPzw5F66M6nQP9NRUPhezkhp5-Jh6qRPrfTWfcygymRVqsirV6yZiaZ9pj-6dIYNvU6MwJCkv25Kc96aPwI_gR3TVuCwYFbv8qrME4cYqwAU" class="kg-image" alt loading="lazy"></figure><p>5. In the next screen, you can review all the transaction information. If everything is in order, press the “confirm” button:</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/lK08z3d41D9BKsYjwzcAyi9S6d0Ct0RGpt4-kJHLWHppKBqQEBI7rnKkPud11oRQMe5fTrkEcCZqN5gRCzPjw3WyNSZQ94XKlqHYzzrreKXf66QP6MxxOZlwOYz3IOnfeJ_sE2-k" class="kg-image" alt loading="lazy"></figure><p>6. After a few seconds your transaction will be processed: </p><figure class="kg-card kg-image-card"><img src="https://lh3.googleusercontent.com/TT5lzr4sSOM6VtQQP6ozEJSkzdnDsHhRlDOB9g6sr1ld22xmJJ1ALotZ4UQRrb-83iGvGyzuVYiy05GEIWxasfhAnScRTvf9_ToImq0LSh0gl5a93KOQ4aM1Al_TSbIGplEFFhTZ" class="kg-image" alt loading="lazy"></figure><p><br>Your funds are now staking. You will start receiving rewards at the end of the next epoch with each epoch lasting 24 hours. Please note that you will have to manually claim your rewards.</p><hr><h3 id="about-p2p-validator"><strong>About P2P Validator</strong></h3><p><a href="https://p2p.org/"><em>P2P Validator</em></a><em> is a world-leading <strong>non-custodial staking provider</strong> with the best industry practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only high class staking opportunities securing more than 4<strong> billion of USD</strong> value at the time of the latest update.</em></p><p><em>P2P Validator is <strong>trusted by over 24,000 delegators</strong> across 25+ networks. We are a major player in all networks we support because of our experience, commitments and our <strong>reputation</strong>. We pay special attention to the process of <strong>governance</strong>.</em></p><hr><p><em>Want to stake Elrond (EGLD) with us? Visit </em><a href="https://p2p.org/elrond">https://p2p.org/elrond</a><em> to find out more about Elrond staking.</em><br><br><em>If you have any questions, feel free to join our</em><a href="https://t.me/P2Pstaking"><em> Telegram chat</em></a><em>, we are always open for communication.</em></p><p></p>
<p><br><strong><em>We are excited to share that Vega has now launched the restricted mainnet - you can now stake VEGA and receive rewards! Still in the early phases and backed by top validators, it’s a project that can shake the Defi ecosystem. </em></strong></p><p>Traditional centralised financial markets have left us paralysed. Traders and investors have no choice other than giving full trust to third parties who fix high costs and stifle innovation by controlling what markets and products can be traded. The Vega network works to solve these shortcomings through the use of its permissionless and agnostic proof of stake network, bringing efficient trading of derivative products (comparable to centralised exchanges) to the DeFi ecosystem. Check out Barney Mannerings (founder of VEGA) explain more about VEGA at the Kryptonight London <a href="https://www.youtube.com/watch?v=DhGD8brS3r0&t=125s">here</a>. <br></p><blockquote><em>“The Vega chain will be the Web3's native derivatives layer, bringing fairness, front-running protection and capital efficiency to blockchain. Starting with an Ethereum bridge Vega plans to support assets from all major blockchains.”</em> <br>David Siska - Vega research team @Vega Protocol</blockquote><p>At P2P Validator our team provides exhaustive analysis to support top-tier staking networks. We have been seed investors and early contributors in top networks such as Solana, Cosmos, Tezos among many others and have a flair for selecting striving networks to stake with. We have now added VEGA (their native token) to our list of carefully selected products - both locked and unlocked VEGA can now be staked, giving you the opportunity to earn more! <br></p><blockquote><em>“Most importantly, Vega and P2P share a lot of core values, and we really appreciate Vega’s built-into incentives that empower decentralisation, and fair, transparent distribution of rewards. We also support Vega’s goal of solving fundamental problems for the centralised finance, and share their intent to open the opportunity to create and launch markets to a broad audience.” </em><br>Dmitry Argunov - Vega Product manager @ P2P Validator</blockquote><p>As with most staking services, the earlier phases provide the highest rewards. Vega are placing a fixed amount of VEGA into a treasury for staking rewards that will be distributed among the staked VEGA. Less staked VEGA to distribute it to, the higher the rewards per VEGA staked. As Vega trading volume increases, a percentage of the infrastructure fee - fees that are charged on trades executed on the platform - will be shared for staking rewards. <br><br>While staking VEGA tokens, you will not only be helping secure the network and earn rewards along the way, you can also participate in governance and shape the future of the Vega project. </p><hr><h3 id="useful-vega-resources">Useful VEGA resources</h3><ul><li><strong><strong>Website:</strong></strong> <a href="https://vega.xyz/">vega.xyz</a></li><li><strong><strong>Guides: </strong></strong><a href="https://p2p.org/economy/vega-vega-staking-guide-using-macs/">For Mac</a> or <a href="https://p2p.org/economy/vega-vega-staking-guide-using-windows/">Windows</a> </li><li><strong>What to Expect from Restricted Mainnet: </strong><a href="https://blog.vega.xyz/what-to-expect-from-restricted-mainnet-616086d9fdaf">https://blog.vega.xyz/what-to-expect-from-restricted-mainnet-616086d9fdaf</a></li><li><strong><strong>Whitepaper:</strong></strong> <a href="https://vega.xyz/papers/vega-protocol-whitepaper.pdf">vega.xyz/papers/vega-protocol-whitepaper.pdf</a></li><li><strong><strong>Github:</strong></strong> <a href="https://github.com/vegaprotocol/">github.com/vegaprotocol/</a></li><li><strong><strong>Twitter:</strong></strong> <a href="https://twitter.com/vegaprotocol">twitter.com/vegaprotocol</a></li><li><strong><strong>Discord</strong>: </strong><a href="https://discord.com/invite/3hQyGgZ">https://discord.com/invite/3hQyGgZ</a></li><li><strong><strong>Blog</strong></strong>: <a href="https://blog.vega.xyz/">blog.vega.xyz/</a></li><li><strong><strong>Community:</strong></strong> <a href="https://community.vega.xyz/">community.vega.xyz/</a></li></ul><hr><h3 id="about-p2p-validator">About P2P Validator</h3><p><a href="https://p2p.org/"><em><em>P2P Validator</em></em></a><em><em> is a world-leading <strong><strong>non-custodial staking provider</strong></strong> with the best industry practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only high class staking opportunities securing more than </em>4<em><strong><strong> billion of USD</strong></strong> value</em> a<em>t the time of the latest update</em>.</em><br><br><em><em>P2P Validator is <strong><strong>trusted by over </strong></strong></em><strong>24</strong><em><strong><strong>,000 delegators</strong></strong> across 25+ networks. We are a major player in all networks we support because of our experience, commitments and our <strong><strong>reputation</strong></strong>. We pay special attention to the process of <strong><strong>governance</strong></strong>.</em></em></p><hr><p>Want to stake VEGA with us? Visit <a href="https://p2p.org/vega">https://p2p.org/vega</a> to find out more about Vega staking and our special offer.<br><br>If you have any questions, feel free to join our<a href="https://t.me/P2Pstaking"> Telegram chat</a>, we are always open for communication.</p>