Solana
Solana Validators Performance Research, part 1: Downtime Analysis
<p>Welcome to the first article in the series of publications on the Solana validators performance research by P2P Validator. In our opinion, performance of the Solana network validators is one of the most important aspects which determine the network growth and sustainability. Our team has done a deep dive into this topic and we want to share insights gained to benefit the Solana community. </p><p>The research is devoted to the analysis of the two most important metrics reflecting Solana network validators’ performance: downtime duration (node delinquency/unavailability duration) and skip rate (measuring how frequently a node fails to produce a transaction block which is subsequently confirmed by consensus on the network). </p><p>In this article we reveal the first part of the research findings regarding analysis of downtime. You can find <a href="https://www.stakingrewards.com/journal/solana-validators-performance-research-report-part-2-skip-rate-analysis/?ref=p2p.org">Part 2 here</a>, covering block skip rate analysis results. </p><h2 id="table-of-contents">Table of contents</h2><!--kg-card-begin: markdown--><ul>
<li><a href="#preface">Preface</a></li>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#solana-validators-downtime">Solana validators downtime</a></li>
<li><a href="#factors-influencing-downtime-duration">Factors influencing downtime duration</a></li>
<li><a href="#downtime-data-analysis">Downtime data analysis</a></li>
<li><a href="#nodes-downtime-duration-by-epochs">Nodes downtime duration by epochs</a></li>
<li><a href="#supermajority-and-superminority-validators-comparison">Supermajority and superminority validators comparison</a></li>
<li><a href="#downtime-duration-distribution-for-updates-and-other-causes">Downtime duration distribution for updates and other causes</a></li>
<li><a href="#average-update-time-by-solana-software-versions">Average update time by Solana software versions</a></li>
<li><a href="#summary">Summary</a></li>
<li><a href="#acknowledgements">Acknowledgements</a></li>
<li><a href="#disclaimer">Disclaimer</a></li>
<li><a href="#sources-of-data">Sources of data</a></li>
</ul>
<!--kg-card-end: markdown--><h2 id="preface">Preface</h2><p>All data used for analysis in the research were obtained from publicly available sources such as Solana JSON RPC API, Solanabeach API, Validators.app API and are relevant for Mainnet beta epochs №194-236 unless another epoch or time period is explicitly specified.</p><h2 id="introduction">Introduction</h2><p>Solana is a relatively new (went live in March 2020) public high-performance distributed blockchain platform curated by Solana Foundation (non-profit organization headquartered in Geneva, Switzerland) along with professional blockchain developers, organizations and individuals running validator and RPC nodes and DevOps specialists from all over the world who are dedicated to the decentralization, growth, and security of the Solana network.</p><p>Solana is one of blockchains that aims to be fast and scalable, without compromising its security or decentralization. Its theoretical throughput limit of 50k transactions per second (TPS) which is twice more than VISA’s limit, which means it can be used for many real-time applications in various business areas. Solana mainnet has already handled more than 35 billion transactions with current throughput exceeding 2000 TPS (see Figure 1) due to high demand for its capabilities and various use cases including: ultra-fast on-chain payments, token creation and distribution, staking through delegation to network validators, smart-contracts, NFTs issuance and trading. Solana ecosystem also provides many different DeFi services such as decentralized exchange, token swaps, liquidity farming and bridging ensuring cross-chain interoperability with other blockchains.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/VYf4yl-UeoEH2-bfH8v7ACSsiTxl2n6d4TM46f_sAKeOUwW_rRCkSelwlOGzzqb9uTXHyj5YQKAK-Wk3XSZDJZLg5EK94MuogDL1J-DWXv5oW8awbwR-tptoGxwmCvY2LYTYlg3L" class="kg-image" alt loading="lazy"><figcaption><em>Figure 1. Solana’s TPS on 27.10.2021 (see <a href="https://explorer.solana.com/?ref=p2p.org">explorer.solana.com</a> for live data).</em></figcaption></figure><p></p><p>There are currently more than 1000 independent validators and 800 RPC nodes (see Figure 2) which comprise a physical layer for above mentioned functionality while making the network highly secure and decentralized. Each validator supports the network's operation by providing <a href="https://docs.solana.com/running-validator/validator-reqs?ref=p2p.org">high-end hardware resources</a> and properly configuring their systems to keep the network running as fast and smooth as possible.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh4.googleusercontent.com/78LCSkFGLBansse0T67-5AmbeRFdQkV4tnzyPE06EXNouo_Kr28DMAXKOv7DiFUo-7m05j0ntYPc0RbblbubK4aLwMSUfr3jgFb17XdETt55bJar-v6s19xi2HrZsfJCpkTQK0jq" class="kg-image" alt loading="lazy"><figcaption><em>Figure 2. Solana validator nodes map (see <a href="https://solanabeach.io/?ref=p2p.org">https://solanabeach.io/</a> for live data).</em></figcaption></figure><p></p><p>Validators receive SOL tokens from stakers, participate in the consensus-based process of transactions validation, get rewards proportional to delegated stake amount and distribute these rewards to stakers (proportionally to staked shares) charging a variable commission. The more stake is delegated to a validator, the more this validator (and its delegators) earns and is more frequently chosen to process new transactions on the ledger and, thus, exposed to greater hardware and network load. Thus, on the one hand, validators are economically motivated to keep their hardware and software running without interruptions, and, on the other, to timely update Solana software and to improve their nodes and network connection as their stake and the Solana network load increase.</p><h1 id="solana-validators-downtime">Solana validators downtime</h1><p>It is normal for a node to be temporarily unavailable/offline sometimes as every technical system needs periodic maintenance and reconfiguration. Typical reasons for server unavailability are usually quite simple such as planned reboot to update host configuration or software, an emergency (power outage), network problems in the data center or at the provider.</p><p>The longer a node is unavailable, the fewer staking rewards and transaction fees it receives. Staking rewards are paid proportionally to node’s vote transactions count which it cannot post if it is offline or functioning incorrectly. Validator downtime negatively affects its delegates’ rewards, which is why one should consider checking validator recent downtime duration history before delegating to it.</p><p>During periods of downtime an unavailable validator is assigned the “delinquent” status which can be checked using Solana CLI <em>solana validators</em> command or by parsing corresponding json response (<em>solana --output json validators</em>). By constantly fetching statuses of all validators on the network it is possible to measure delinquency periods durations which is a good approximation for downtime duration for further quantitative analysis. The downtime data analyzed is available through the <a href="https://redash.p2p.org/public/dashboards/ZEW9RvuBXPdYHUU5aC7DfVjY8DJaXQrGenG2HUmo?org_slug=default&ref=p2p.org">public Redash dashboard</a>.</p><h2 id="factors-influencing-downtime-duration">Factors influencing downtime duration</h2><p>There are many factors influencing downtime duration and these are typical ones:</p><ul><li>node operator reaction time (node operators may or may not use specialized monitoring and alerting systems);</li><li>node operator skill (imagine the difference between inexperienced enthusiasts and mature professionals who are working with such high-load systems for years);</li><li>time to repair breakdowns in the power grid or communication network (which does not depend on a node operator);</li><li>time needed to debug and fix specific configuration errors or replace hardware parts;</li><li>complexity and duration of software update (i.e., different Solana versions take different time to install), node startup duration, etc.</li></ul><p>Despite most of these factors can not be measured directly, we have managed to collect and analyse some important on-chain data related to the topic, which allowed to quantitatively describe several aspects regarding Solana network nodes unavailability such as downtime duration statistics over time, its variability across nodes as well as duration of node software updates.</p><h2 id="downtime-data-analysis">Downtime data analysis</h2><p>Here we illustrate retrospective downtime statistics of Solana nodes that were active in the period from epoch №209 (5th of August, 2021) to epoch №236 (17th of October, 2021). Historical data allow to reveal trends in the dynamics of downtime making it easier to understand the normal behavior of the metric as well as to identify abnormal fluctuations.</p><h3 id="nodes-downtime-duration-by-epochs">Nodes downtime duration by epochs</h3><p>The descriptive statistics for downtime duration by epochs are presented in the Figure 3 below. Quantile values of 5%- and 95%-level reflect the maximum downtime among the top 5 and top 95 percent of validators, respectively, for each epoch. Average downtime is the simple arithmetic mean and median defines a downtime duration which divides the top 50 and worst 50 percent of validators.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh3.googleusercontent.com/aQCsTug3snvF4n1VUW3AXUBYvC6FytRw5Q2ICuSuMsZMr8RGjpyI-ylB6CZK8qU8WR5EKHnBNMPfMBsV6CXTN7UM_sMO3k_QPjq5JhoVDPJW0dEU6FTJUr0skrGCbjcKdpVQAB5m" class="kg-image" alt loading="lazy"><figcaption><em>Figure 3. Downtime mean (cyan line), median (lilac), 5%-quantile (red), 95%-quantile (green) and its actual values for each node (black transparent dots) over epochs.</em></figcaption></figure><p><br>As can be seen from the chart above, <strong>typical average downtime duration for a node is around 1.5 hours</strong>, which is quite low, while <strong>median downtime duration is almost always zero</strong> (which means that most nodes usually don’t experience shutdowns). Also there were several epochs (№214, 223 and 234) with high downtime duration upticks mainly due to simultaneous upgrades of Solana software version. Epoch 223 is especially interesting as it is known that on 14 of September, 2021, the Solana network experienced a <a href="https://jumpcrypto.com/reflections-on-the-sept-14-solana-outage/?ref=p2p.org">severe overload which led to the network halt</a>, and after a successful network restart almost all the nodes had to update to a new Solana version with the necessary fixes.</p><h3 id="dispersion-of-downtime-duration">Dispersion of downtime duration</h3><p>As many factors affect downtime duration, it varies greatly across validators within the same epoch. The dispersion measures indicate the metric’s spread magnitude which is slightly changing over time as shown in Figure 4.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/pYyFq6WJf7ZUvigy7YMAFPcT0e9OISiG4_mNYWZLs-N_c7vUrs09jUb9wtgCxgH4OqNIAppwlrjif3vupB0ZWDAtSc2mv8gI8ixFAkvcVVYs859GnOtTiXvGN5SQSDjPA2ql9Hjv" class="kg-image" alt loading="lazy"><figcaption><em>Figure 4. Measures of dispersion of downtime duration over time: interquartile range (red line) representing difference between 25%- and 75%-quantiles of the metric and standard deviation (cyan) representing average deviation from mean.</em></figcaption></figure><p><br>It can be seen from the chart above that downtime duration dispersion across validator nodes is dropping slightly over time which indicates that validators, on average, have both lower downtime durations and lower deviations of the metric from the mean.</p><h3 id="supermajority-and-superminority-validators-comparison">Supermajority and superminority validators comparison</h3><p>Since the leading validators with a large stake amount take much more financial risks compared to the smaller ones, their nodes' technical characteristics are far better than of the majority. Therefore, it makes sense to compare performance of the superminority set of validators (the minimal set of validators that together control more than 33.33% of the total stake) with the rest falling into the supermajority set with 66.66% of total stake (see Figure 5).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/pXXTHF4CC5szdub_hF8cYrIu_BpisJxblMbSODq5p4IWU7gdhGI5XXOyDGOLbATya7vLvaH0rDLWXJsLZY_D1OiAD9MpKSRR3ZkrB_oXBSagqZKNeAxlE1dik8nVo9moZvIN3aC6" class="kg-image" alt loading="lazy"><figcaption><em>Figure 5. Average downtime duration by epochs for superminority validators set (blue line), supermajority (green) and P2P Validator (red).</em></figcaption></figure><p><br>As the charts above represent, the supermajority is usually much worse in terms of average downtime duration, especially after epoch №220, especially during hard times like epoch №223, when the Solana network halted and most validators had to perform major software updates.</p><p>In contrast, superminority validators (especially P2P Validator) have an average downtime duration and an average number of downtimes (see Figure 6 below) that are much lower than for the supermajority, and there is a much smaller probability that a validator from the superminority set is offline for more than 5% of total epoch duration (see Figure 7 below).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/vUWC-dbCA4C7q5xb2U29_ch4guExlxtY1-5QMdK8vTS9Ek_3lpPFnPFqV12qM2u8ErrpDUtgMfD4333AjhqiP1cPMqHfxGfZpaI-AmZGitsKPgLKJz5_dvesr5OMehcU1Mljfhvi" class="kg-image" alt loading="lazy"><figcaption><em>Figure 6. Average number of downtimes over epochs №209-236 for the superminority (cyan line) and supermajority set of validators (red).</em></figcaption></figure><p></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh4.googleusercontent.com/ZQeA4Y0DFh4ontzVLbMRC2MNw_6Dh1_7oe8is52gxvxSWfROVrXbE2PjzlrbIOoT-hVp8pu9CkfuyhAH9ovBd2dUq1-yLrexjo2xwOKjXbJdsylkLYtfzUxD1Zh6J9r_hvkCsNLU" class="kg-image" alt loading="lazy"><figcaption><em>Figure 7. Share of validators with downtime duration greater than 5% of total epoch duration over epochs №209-236 for superminority (cyan line) and supermajority set of validators (red).</em></figcaption></figure><h3 id="downtime-duration-distribution-for-updates-and-other-causes"><br>Downtime duration distribution for updates and other causes</h3><p>As it was described previously, downtimes may happen due to Solana node software updates as well as due to hardware upgrades and unexpected halts. The available on-chain data allows to distinguish between downtimes related to software updates and related to other causes and compare downtime duration distributions for the supermajority and superminority groups of validators.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh5.googleusercontent.com/vKMR9AxmKV1-dm-jSA0f_mSFiEZV_oR0ur9g8_R5666r4z1vSn8OJcikdkq9La4l99IWwzLZgv-1OATphpDFbdEh2izdijhIDTmYTJ4GlpPPr7-vKAa-0HoaZXwhoNmzYu1ilpBJ" class="kg-image" alt loading="lazy"><figcaption><em>Figure 8. Downtime duration distribution (in logarithmic scale) unrelated to software updates for supermajority (green line), superminority (blue) set of validators and P2P Validator (red). Dashed lines of corresponding colors show the average downtime duration written nearby</em></figcaption></figure><p><br>According to the distributions of downtime duration not related to software updates (see Figure 8), validators groups are quite similar apart from the fact that supermajority validators are more likely to have very long outages that greatly increase the average value of downtime duration (69 vs. 34 minutes for the superminority group). It should be noted that even if the P2P Validator goes down (or delinquent), on average it happens for an extremely short time of 1.5 minutes.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh6.googleusercontent.com/Ho-gbXZDhA6SyHlPnJxSGrHE2F5x-Cn8HGwX0K30wuoibCH6Ay_Rx1Obq8D4Se3MoUATYUxICIKq-7RuKkZhcct1ARAUruNtCaGi4_1lNBsDEhJYpGS3Fao5osWpA1lV4oeWyYhC" class="kg-image" alt loading="lazy"><figcaption><em>Figure 9. Downtime duration distribution (in logarithmic scale) related to software updates for supermajority (green line), superminority (blue) set of validators and P2P Validator (red). Dashed lines of corresponding colors show the average downtime duration written nearby.</em></figcaption></figure><p><br>Concerning downtimes due to software updates (see Figure 9), the distributions for the groups differ considerably: for the superjmajority group there is much more variability in downtime duration when compared to the superminority and again supermajority validators frequently have much longer update times leading to higher average (195 vs. 76 minutes for superminority group). Superminority validators including the P2P Validator demonstrate high consistency in update duration presumably due to specific administration standards developed by professional engineers who operate these validators.</p><h3 id="average-update-time-by-solana-software-versions">Average update time by Solana software versions</h3><p>Different Solana node software versions vary significantly in the complexity and duration of the installation process, which directly affects the downtime duration associated with updates. Figure 10 below shows the average update time of Solana node software versions by validators from the supermajority and superminority groups.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lh3.googleusercontent.com/0ahrMb4jlGfbkVuMN5FNxFwdjqx5563dDEDqWSLqdpz0w9edfuUv7_Y6QWh0FFM704pbpyQHtmjKPTskVDj54VjWDgnPdARp5nSOnUiUueOuJq6BJbVUjpo45J66poJ8ZK26HX6c" class="kg-image" alt loading="lazy"><figcaption><em>Figure 10. Average update time for different Solana node software versions.</em></figcaption></figure><p><br>Of all the most used versions of Solana node software, the update to version 1.6.25 took the longest for both supermajority (4.5 hours on average) and superminority (3.5 hours on average) validators. Long updates to versions 1.7.11 and 1.7.15 were performed only by validators from the supermajority group and took approximately 2-3 hours to complete. Overall, validators from the superminority group usually perform the updates significantly faster ensuring less rewards losses for them and their delegators.</p><h2 id="summary">Summary</h2><p>Downtime duration is a very important metric as it reflects Solana validators operators efficiency and influences rewards received by validators and their delegators as well as overall network’s stability and security. Solana Foundation and the network validators do everything they can to improve performance of nodes and quality of software that control nodes operation, and we can say with confidence that they do it very well, especially validators from superminority group thanks to the experience and professionalism of DevOps engineers.</p><h2 id="acknowledgements">Acknowledgements</h2><p>Authors of the report would like to express gratitude and appreciation for the <a href="https://p2p.org/?ref=p2p.org">P2P Validator</a><strong> </strong>team whose guidance, support and encouragement have been invaluable throughout the research. We would also like to thank <a href="https://twitter.com/stephenakridge?ref=p2p.org">Stephen Akridge</a>, co-founder of Solana, <a href="https://twitter.com/vmulps?ref=p2p.org">Ruud van Asseldonk</a>, software engineer at Chorus One, and <a href="https://twitter.com/RDorzbach?ref=p2p.org">Robert Dörzbach</a>, product manager of the Solana Beach, for helpful advice, comments and corrections.</p><h2 id="disclaimer">Disclaimer</h2><p>Information presented in this report and referenced sources are for educational purposes only. It is not financial/investment advice. Seek a licensed professional for any financial advice. Authors of the report made every reasonable effort to ensure the accuracy and validity of the information provided. However, as price points, conditions, and information are continually changing, authors reserve the right to change at any time without notice, information contained in the report and make no warranties or representations as to its accuracy or up-to-dateness.</p><p>Authors of the report are employees of P2P Validator company which provides professional services and consulting for highly secure non-custodial staking across more than 25 blockchain networks, including the Solana network with mainnet and testnet validator nodes as well as RPC nodes. Therefore, P2P Validator is not a neutral party with its own business interests in the Solana ecosystem. Nevertheless, authors did their best to make the report as objective as possible with the main purpose in mind being to educate and inform the community.</p><h2 id="sources-of-data">Sources of data</h2><ol><li><a href="https://app.swaggerhub.com/apis-docs/V2261/solanabeach-backend_api/0.0.1?ref=p2p.org">https://app.swaggerhub.com/apis-docs/V2261/solanabeach-backend_api/0.0.1</a></li><li><a href="https://docs.solana.com/developing/clients/jsonrpc-api?ref=p2p.org">https://docs.solana.com/developing/clients/jsonrpc-api</a></li><li><a href="https://www.validators.app/api-documentation?ref=p2p.org">https://www.validators.app/api-documentation</a></li><li><a href="https://redash.p2p.org/public/dashboards/ZEW9RvuBXPdYHUU5aC7DfVjY8DJaXQrGenG2HUmo?org_slug=default&ref=p2p.org">https://redash.p2p.org/public/dashboards/ZEW9RvuBXPdYHUU5aC7DfVjY8DJaXQrGenG2HUmo?org_slug=default</a></li></ol><hr><h2 id="about-p2p-validator">About P2P Validator</h2><p>P2P Validator is a world-leading non-custodial staking provider <strong><strong>securing more than </strong>4<strong> billion USD value from over </strong>2<strong>0,000 delegators across 25+ high-class networks</strong></strong>. We are early investors in Solana and have supported the network from the first block taking part in all stages of testing and voting.</p><hr><p><strong><strong>Web</strong></strong>: <a href="https://p2p.org/?ref=p2p.org">p2p.org</a><br><strong><strong>Stake SOL with us</strong></strong>: <a href="https://p2p.org/solana?ref=p2p.org">p2p.org/solana</a><br><strong><strong>Twitter</strong></strong>: @p2pvalidator<br><strong><strong>Telegram</strong></strong>: <a href="https://t.me/P2Pstaking?ref=p2p.org">t.me/P2Pstaking</a></p>
ALL
Agoric
Aptos
Auth
Avail
Avalanche
Axelar
Babylon
Bitcoin
BTC
Cardano
Celestia
Chainlink
Cosmos
Crescent
Cyberway
DAOBet
DVT
Dymension
Economy
EigenLayer
Elrond
Ethereum
Evmos
Explain Like I'm Five
Guide
HR
Kava
Kusama
Lido
LSP
Manta
Mantle
Marlin
Matic
Mina
Moonbeam
MultiversX
Near
NuCypher
Oasis
Origin
Partnership
Persistence
Pocket
Polkadot
Polygon
Quicksilver
Regen
Renzo
restaking
RPC Node
Serum
Solana
SSV.Network
Staking
Sui
symbiotic
Terra
Tezos
TheGraph
Threshold
Update
Vara
Vega
