ALL Agoric Aptos Auth Avalanche Axelar Cardano Celestia Chainlink Cosmos Crescent Cyberway DAOBet DVT Dymension Economy EigenLayer Elrond Ethereum Evmos Explain Like I'm Five Guide HR Kava Kusama Lido LSP Manta Mantle Marlin Matic Mina Moonbeam MultiversX Near NuCypher Oasis Partnership Persistence Pocket Polkadot Polygon Quicksilver Regen Renzo restaking RPC Node Serum Solana SSV.Network Staking Sui Terra Tezos TheGraph Threshold Update Vara Vega
/
Near

Near Incident - Post Mortem

Post preview image

TLDR


Due to a sudden spike of transactions, mainnet validator node ran out of disk space, leading to inability to produce blocks and corruption of the node state.

What happened?


Disk usage grew quite fast and mainnet pool p2p-org.poolv1.near stopped producing blocks since ~07:00 UTC on 16th of January. Epoch ended ~10:00 UTC and by that time the node had surpassed the epochal kick-out threshold for downtime. Therefore, it was scheduled for temporary kick-out for the next two epochs. While resolving the issue, the validator remained offline for the following epoch resulting in one additional forfeit epoch. Overall, the node was offline for ~3,25 epochs and the validator pool lost 4 full epochs of staking rewards.

What went wrong?


The potential impact of the issue was underestimated.

It was expected to rely on a recent state snapshot that would be readily available. We had no cold backup of a recent node state while available backup nodes were subject to the same issue leading to a loss of access to a synced node. In fact, the official public back-up archives were corrupted too. A GitHub issue was created afterwards.

Monitoring was insufficient.

Near validation infrastructure was undergoing an overhaul, some monitoring facilities were offline. It was expected that the amount of space used on disk would grow more or less linearly. With ~100 GiB of free space it could last for a month. Space clogged up in a matter of days while disk monitoring was not set appropriately to catch the spike and warn in advance.

What went well?


It was immediately notified that node stopped producing blocks and the root cause was identified almost simultaneously. Quite a few validators were affected by the issue, and the community was very helpful.

Impact on clients


All our Near delegators were affected and lost four epochs of staking rewards. To compensate our delegators in full and mitigate their loss, P2P waived the fees until the end of February.

Lessons learned


We should have had better monitoring and collecting disk usage metrics from all nodes at all times including mainnet, backup and Near RPC node. It is important to ensure that back-up nodes are running & synced at all times. In addition, it is important to establish the process of making cold snapshots of the node state on a regular basis and spread this practice to all available networks.

P2P takes full responsibility for the event that led to the weak performance and we are sorry for the inconvenience. Please be assured that P2P is taking actions to eliminate even a small probability of such an event occurring in future.

If you have any questions feel free to join our Telegram chat, we are always open for communication.
Special thanks to Evgeny Kuzyakov & DenysK for providing a state snapshot and general support.

Subscribe to P2P-economy

Get the latest posts delivered right to your inbox

Subscribe
Admin

Read more
Persistence Persistence (XPRT) Staking Guide - [Keplr + Ledger]

<p></p><p>This tutorial helps you stake and manage <a href="https://p2p.org/persistence?ref=p2p.org">XPRT</a> tokens using the <a href="https://www.keplr.app/?ref=p2p.org">Keplr</a> Browser Extension together with your <a href="https://www.ledger.com/?ref=p2p.org">Ledger</a> device.<br><br><strong>This guide will help you</strong></p><ol><li>Set up your Keplr Wallet</li><li>Deposit your XPRT tokens to your wallet</li><li>Stake XPRT to earn staking rewards</li><li>Manage your XPRT stake and grow your portfolio<br></li></ol><p><strong>Before you start:</strong></p><ol><li>Download and install the newest version of <a href="https://www.ledger.com/ledger-live/download?ref=p2p.org">Ledger Live</a> on your desktop machine.</li><li>Update your <a href="https://support.ledger.com/hc/en-us/articles/360013349800-Update-Ledger-Nano-X-firmware?docs=true&ref=p2p.org">Ledger</a> device to the newest firmware.</li><li>Install the latest version of the Persistence App on your Ledger device.<br></li></ol><p>See below for our step-by-step guide.</p><h2 id="i-setting-up-your-keplr-wallet">I. Setting up your Keplr Wallet</h2><ol><li>Open the Keplr Browser Extension and press ‘Import Ledger’.</li></ol><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image--14-.png" class="kg-image" alt loading="lazy" width="685" height="449" srcset="https://p2p.org/economy/content/images/size/w600/2022/02/image--14-.png 600w, https://p2p.org/economy/content/images/2022/02/image--14-.png 685w"></figure><p>2. Type the account name you wish to use, then click ‘Next’.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-2.png" class="kg-image" alt loading="lazy" width="645" height="520" srcset="https://p2p.org/economy/content/images/size/w600/2022/02/image-2.png 600w, https://p2p.org/economy/content/images/2022/02/image-2.png 645w"></figure><p>3. Plug in and unlock your Ledger device. Open the Persistence app and click ‘Next’.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-3.png" class="kg-image" alt loading="lazy" width="458" height="597"></figure><p>4. Your Keplr account should now be successfully connected to your Ledger device.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-4.png" class="kg-image" alt loading="lazy" width="641" height="458" srcset="https://p2p.org/economy/content/images/size/w600/2022/02/image-4.png 600w, https://p2p.org/economy/content/images/2022/02/image-4.png 641w"></figure><p>Now you will need to deposit XPRT into your wallet.</p><h2 id="ii-deposit-persistence-xprt-into-your-keplr-wallet">II. Deposit Persistence (XPRT) into your Keplr wallet</h2><p>1. Open the Keplr Browser Extension and choose your Ledger account (via the ‘human icon’ on the top right).</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-29.png" class="kg-image" alt loading="lazy" width="710" height="108" srcset="https://p2p.org/economy/content/images/size/w600/2022/02/image-29.png 600w, https://p2p.org/economy/content/images/2022/02/image-29.png 710w"></figure><p>2. Find your Persistence wallet by selecting the drop down menu on the top.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-15.png" class="kg-image" alt loading="lazy" width="352" height="570"></figure><p>3. Copy your address by clicking on it as indicated in the image below. Alternatively you can select "Deposit" to find your address QR code. </p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-16.png" class="kg-image" alt loading="lazy" width="351" height="536"></figure><p>Once you have deposited your XPRT and you own a balance on your Keplr wallet, you are now ready to start staking! </p><h2 id="iii-staking-your-xprt">III. Staking your XPRT!</h2><p>1. To start staking select "Stake". You will be prompted to the web version of the Keplr wallet where you will see a list of validators. </p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-17.png" class="kg-image" alt loading="lazy" width="351" height="536"></figure><p>2.  Choose your validator (P2P.ORG - P2P Validator) then click ‘Manage’ and ‘Delegate’ in the next picture.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-19.png" class="kg-image" alt loading="lazy" width="2000" height="544" srcset="https://p2p.org/economy/content/images/size/w600/2022/02/image-19.png 600w, https://p2p.org/economy/content/images/size/w1000/2022/02/image-19.png 1000w, https://p2p.org/economy/content/images/size/w1600/2022/02/image-19.png 1600w, https://p2p.org/economy/content/images/size/w2400/2022/02/image-19.png 2400w" sizes="(min-width: 720px) 720px"></figure><p>3. Choose the amount of XPRT you want to stake and click ‘Delegate’.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-22.png" class="kg-image" alt loading="lazy" width="489" height="523"></figure><p>4. Set your preferred fee and select "Approve". </p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-25.png" class="kg-image" alt loading="lazy" width="353" height="576"></figure><p>5. Check the information on your Ledger device and click ‘Approve’ on it.</p><p>6. Use the Dashboard within the <a href="https://wallet.keplr.app/?ref=p2p.org#/cosmoshub-4/stake">Keplr Web Wallet</a> to see whether your delegation was successful.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-27.png" class="kg-image" alt loading="lazy" width="354" height="599"></figure><p>Now let’s move on to managing your staking assets.</p><h2 id="iv-managing-your-persistence-xprt-stake">IV. Managing your Persistence (XPRT) stake</h2><h3 id="claim-your-xprt-staking-rewards"><strong>Claim your XPRT staking rewards</strong></h3><p>1. Open the Keplr Browser Extension and click ‘Claim’.</p><figure class="kg-card kg-image-card"><img src="https://p2p.org/economy/content/images/2022/02/image-28.png" class="kg-image" alt loading="lazy" width="354" height="599"></figure><p>2. Set your preferred fee and select "Approve". </p><p>3. Plug in your Ledger device, unlock, and open the Persistence app on it, then click ‘Next’ in the Browser Extension pop-up.</p><p>4. Review the transaction on your Ledger device and confirm it.</p><h3 id="compound-your-xprt-staking-rewards-to-increase-your-return"><strong>Compound your XPRT staking rewards to increase your return</strong></h3><p>To compound, you simply have to claim your XPRT rewards and go through the staking process again as in section III. That’s it!</p><p>Now you know how to delegate and compound your XPRT staking rewards using the Keplr Browser Extension together with your Ledger device.</p><p>If any questions arise, whether on XPRT staking or not, feel free to contact us via <a href="https://twitter.com/p2pvalidator?ref=p2p.org">Twitter</a>, <a href="https://t.me/P2Pstaking?ref=p2p.org">Telegram</a>, or <a href="https://p2p.org/?ref=p2p.org#contact-us">email</a>.</p><p><br><strong><strong>About P2P Validator</strong></strong><br><a href="https://p2p.org/?ref=p2p.org"><em><em>P2P Validator</em></em></a><em><em> is a world-leading <strong><strong>non-custodial staking provider</strong></strong> with the best industry practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only high class staking opportunities securing more than 3<strong><strong> billion of USD</strong></strong> value at the time of the latest update.</em></em></p><p><em><em>P2P Validator is <strong><strong>trusted by over 24,000 delegators</strong></strong> across 25+ networks. We are a major player in all networks we support because of our experience, commitments and our <strong><strong>reputation</strong></strong>. We pay special attention to the process of governance. <strong><strong>P2P </strong></strong></em>has<em> </em>the<em> intention to contribute and provide long term support to the</em> Persistence (XPRT)<em> network.</em></em></p><p><br></p>

Admin

from p2p validator

Agoric Staking Agoric with Keplr wallet

<p><br>In this article we will guide you through the process of staking Agoric governance token (BLD) using the Keplr wallet.</p><hr><p>To begin, head over to the<a href="https://wallet.keplr.app/?ref=p2p.org"> Keplr wallet</a> website.</p><p>1. You will be prompted to install the Keplr browser extension. Once you have done so, open the extension and either create an account or login by importing an account already created.</p><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/a1LuAKHz6UdLyFhh_bIEYFHc70PvZd-QO6nhf_9hExPuie_LQWe5fNi0i7byB9NJrnjwTrIhawubDvwHWR88m6jxYz7Pe9uOrjZ9M2k0leR6WHQRHEPW_P80xOG_TQjJlmYzcJbh" class="kg-image" alt loading="lazy"></figure><p>2. Once your account is set up, you now need to navigate to the Agoric section and select Stake. Click Manage to proceed with delegation.</p><figure class="kg-card kg-image-card"><img src="https://lh3.googleusercontent.com/_3ks08mqYw3PSsLvhfatnqJuJATt2wuqvYmpCYQtJ387Xjd-Wc-3EFjEyq8H9XhNgRIjMs_LNXYH7oLjOGVN4KVtHKjRCBes-rBWPeTUqfihDsnSvxxbA_76tD37z6u0mwV9n-zz" class="kg-image" alt loading="lazy"></figure><p>3. Press Delegate.</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/XuoONXjcXmDQhTiciBNWdg2uhZ4IYAZqn_ON94fKyMotwzumv1wQ6iMOo2xen4zmO7RmEdEP5MGL8DfSJ1UfB_EXxQf27W_-ElPCHkM5e-IDoM1S-ZDcJw-q7_8pjBfDr7WEyQ8P" class="kg-image" alt loading="lazy"></figure><p>4. Input the amount and press Delegate.</p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/PankhoHKPLWX_EOdFKt5Qjzvid6r5IXlw0v62EBbMqCT4Q4T_vBTZJpnOmWaaQFuKh79lJ80sFb0nMY7V8ZFmMS0Gxg3uGmSnx_XYUk16Y0HW56J0Msr6LEsurtu1i1Uy9F344g5" class="kg-image" alt loading="lazy"></figure><p>5. Confirm the transaction in Keplr Wallet to start staking!</p><h3 id="about-p2p-validator"><strong>About P2P Validator</strong></h3><p><a href="https://p2p.org/?ref=p2p.org"><em>P2P Validator</em></a><em> is a world-leading <strong>non-custodial staking provider</strong> with the best industry practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only high class staking opportunities securing more than 3<strong> billion of USD</strong> value at the time of the latest update.</em></p><p><em>P2P Validator is <strong>trusted by over 24,000 delegators</strong> across 25+ networks. We are a major player in all networks we support because of our experience, commitments and our <strong>reputation</strong>. We pay special attention to the process of governance. <strong>P2P invested in Agoric</strong> with intention to contribute and provide long term support to the network.</em></p><hr><p><em>Want to stake Agoric (BLD) with us? Visit</em><a href="https://p2p.org/elrond?ref=p2p.org"><em> </em></a><a href="https://p2p.org/agoric?ref=p2p.org">https://p2p.org/<em>agoric</em></a><em> to find out more about Agoric staking.</em></p><hr><p><em>If you have any questions,  feel free to join our</em><a href="https://t.me/P2Pstaking?ref=p2p.org"><em> Telegram chat</em></a><em>, we are always open for communication.</em></p>

Admin

from p2p validator