Persistence (XPRT) Staking Guide - [Keplr + Ledger]

<h2 id="summary"><strong>Summary</strong></h2><p>One of our Validator nodes went offline at the end of era 3281 until the start of era 3282 as the result of the corruption of a blockchain database that was being used. This led to the inability to produce blocks for <strong>2h:45m.</strong></p><h2 id="what-happened"><strong>What happened</strong></h2><p>A database that we were using to run our validator node crashed. We immediately took action and started deploying a new node using a recent snapshot of our disk. Once fully synced, new keys were generated and signed off before turning the validation service back on. As we created a new validator with the rotation of session keys, we were forced to chill for an additional epoch.</p><h2 id="customer-impact"><strong>Customer Impact</strong></h2><p>Delegators who nominated this validator and had their stake allocated to it will receive lower rewards than it could have potentially earned for 2 epochs.</p><h2 id="what-went-wrong"><strong>What went wrong?</strong></h2><p>Inadequate tools for this particular event. We did not deploy the db from our backups (pruned) because on occasion, we faced issues with our automation running too long. Additionally, we had not recovered the session keys since it’s always a risky option and the situation at hand did not warrant taking such risks.</p><h2 id="what-went-well"><strong>What went well?</strong></h2><p>It was immediately notified that the node stopped producing blocks and the root cause was identified almost instantly. This allowed us to swiftly amend the occurrence, and with the help of fully automated key rotation/verification and signing cycle, we were able to get a fully operational node running in an effective manner.</p><h2 id="lessons-learnt-and-action-plan"><strong>Lessons learnt and action plan</strong></h2><p>Improve our incident handling procedures and find a faultless and rapid fix for these kinds of events. We already improved and reached faster spin up for fully synced nodes and will implement a solution for safe session keys management.</p><p>P2P takes full responsibility for the event that led to the weak performance and we are sorry for the inconvenience. Please be assured that P2P is taking actions to eliminate even a small probability of such an event occurring in future.</p><hr><p><em>If you have any questions feel free to join our</em><a href="https://t.me/P2Pstaking?ref=p2p.org"><em> Telegram chat</em></a><em>, we are always open for communication.</em></p>

<h2 id="tldr"><strong>TLDR</strong></h2><p><br>Due to a sudden spike of transactions, mainnet validator node ran out of disk space, leading to inability to produce blocks and corruption of the node state.</p><h2 id="what-happened"><strong>What happened?</strong></h2><p><br>Disk usage grew quite fast and mainnet pool p2p-org.poolv1.near stopped producing blocks since ~07:00 UTC on 16th of January. Epoch ended ~10:00 UTC and by that time the node had surpassed the epochal kick-out threshold for downtime. Therefore, it was scheduled for temporary kick-out for the next two epochs. While resolving the issue, the validator remained offline for the following epoch resulting in one additional forfeit epoch. Overall, the node was offline for ~3,25 epochs and the validator pool lost 4 full epochs of staking rewards.</p><h2 id="what-went-wrong"><strong>What went wrong?</strong></h2><p><br><strong>The potential impact of the issue was underestimated.</strong></p><p>It was expected to rely on a recent state snapshot that would be readily available. We had no cold backup of a recent node state while available backup nodes were subject to the same issue leading to a loss of access to a synced node. In fact, the official public back-up archives were corrupted too. A <a href="https://github.com/near/nearcore/issues/6095?ref=p2p.org">GitHub issue</a> was created afterwards.</p><p><strong>Monitoring was insufficient.</strong></p><p>Near validation infrastructure was undergoing an overhaul, some monitoring facilities were offline. It was expected that the amount of space used on disk would grow more or less linearly. With ~100 GiB of free space it could last for a month. Space clogged up in a matter of days while disk monitoring was not set appropriately to catch the spike and warn in advance.</p><h2 id="what-went-well"><strong>What went well?</strong></h2><p><br>It was immediately notified that node stopped producing blocks and the root cause was identified almost simultaneously. Quite a few validators were affected by the issue, and the community was very helpful.</p><h2 id="impact-on-clients"><strong>Impact on clients</strong></h2><p><br>All our Near delegators were affected and lost four epochs of staking rewards. To compensate our delegators in full and mitigate their loss, P2P waived the fees until the end of February.</p><h2 id="lessons-learned"><strong>Lessons learned</strong></h2><p><br>We should have had better monitoring and collecting disk usage metrics from all nodes at all times including mainnet, backup and Near RPC node. It is important to ensure that back-up nodes are running &amp; synced at all times. In addition, it is important to establish the process of making cold snapshots of the node state on a regular basis and spread this practice to all available networks.</p><p>P2P takes full responsibility for the event that led to the weak performance and we are sorry for the inconvenience. Please be assured that P2P is taking actions to eliminate even a small probability of such an event occurring in future.</p><hr><p><em>If you have any questions feel free to join our</em><a href="https://t.me/P2Pstaking?ref=p2p.org"><em> Telegram chat</em></a><em>, we are always open for communication. </em><br><em>Special thanks to Evgeny Kuzyakov &amp; DenysK for providing a state snapshot and general support.</em></p><hr>