Staking Agoric with Keplr wallet

<h2 id="tldr"><strong>TLDR</strong></h2><p><br>Due to a sudden spike of transactions, mainnet validator node ran out of disk space, leading to inability to produce blocks and corruption of the node state.</p><h2 id="what-happened"><strong>What happened?</strong></h2><p><br>Disk usage grew quite fast and mainnet pool p2p-org.poolv1.near stopped producing blocks since ~07:00 UTC on 16th of January. Epoch ended ~10:00 UTC and by that time the node had surpassed the epochal kick-out threshold for downtime. Therefore, it was scheduled for temporary kick-out for the next two epochs. While resolving the issue, the validator remained offline for the following epoch resulting in one additional forfeit epoch. Overall, the node was offline for ~3,25 epochs and the validator pool lost 4 full epochs of staking rewards.</p><h2 id="what-went-wrong"><strong>What went wrong?</strong></h2><p><br><strong>The potential impact of the issue was underestimated.</strong></p><p>It was expected to rely on a recent state snapshot that would be readily available. We had no cold backup of a recent node state while available backup nodes were subject to the same issue leading to a loss of access to a synced node. In fact, the official public back-up archives were corrupted too. A <a href="https://github.com/near/nearcore/issues/6095?ref=p2p.org">GitHub issue</a> was created afterwards.</p><p><strong>Monitoring was insufficient.</strong></p><p>Near validation infrastructure was undergoing an overhaul, some monitoring facilities were offline. It was expected that the amount of space used on disk would grow more or less linearly. With ~100 GiB of free space it could last for a month. Space clogged up in a matter of days while disk monitoring was not set appropriately to catch the spike and warn in advance.</p><h2 id="what-went-well"><strong>What went well?</strong></h2><p><br>It was immediately notified that node stopped producing blocks and the root cause was identified almost simultaneously. Quite a few validators were affected by the issue, and the community was very helpful.</p><h2 id="impact-on-clients"><strong>Impact on clients</strong></h2><p><br>All our Near delegators were affected and lost four epochs of staking rewards. To compensate our delegators in full and mitigate their loss, P2P waived the fees until the end of February.</p><h2 id="lessons-learned"><strong>Lessons learned</strong></h2><p><br>We should have had better monitoring and collecting disk usage metrics from all nodes at all times including mainnet, backup and Near RPC node. It is important to ensure that back-up nodes are running &amp; synced at all times. In addition, it is important to establish the process of making cold snapshots of the node state on a regular basis and spread this practice to all available networks.</p><p>P2P takes full responsibility for the event that led to the weak performance and we are sorry for the inconvenience. Please be assured that P2P is taking actions to eliminate even a small probability of such an event occurring in future.</p><hr><p><em>If you have any questions feel free to join our</em><a href="https://t.me/P2Pstaking?ref=p2p.org"><em> Telegram chat</em></a><em>, we are always open for communication. </em><br><em>Special thanks to Evgeny Kuzyakov &amp; DenysK for providing a state snapshot and general support.</em></p><hr>

<p>Agoric is a secure JavaScript smart-contract platform that allows developers to reuse a set of trusted components to build decentralized applications (dApps). Smart-contracts power the new efficient economy that doesn't rely on a third party while all the interactions within the system are fair and transparent. <strong>Critical smart-contract bugs result in million dollar losses for developers and users of dApps.</strong> They are still considered risky and difficult to use.</p><p>Agoric uses an object-capabilities (ocap) security architecture defining constraints on how references to the objects are obtained. It simplifies building an application allowing developers to focus on it's purpose and makes interaction with contracts much safer. <strong>With the Zoe framework developed by Agoric even a buggy contract won't lead to the loss of funds put inside as they become automatically escrowed.</strong></p><p>Another well-known problem is a lack of developers in crypto. Many exciting use cases are still to be invented. Inflow of new talents can boost the ecosystem’s growth adding to composability. JavaScript remains the most fast-growing programming language having over 10 millions developers all over the world. <strong>We believe that a composable framework built by Agoric is capable of attracting a significant portion of JavaScript developers into the space.</strong></p><p><em>We are excited to announce that P2P Validator joined Agoric mainnet as a node operator. Agoric vision is fully in line with our mission to make decentralized finance secure, accessible and simple for everyone.</em></p><p>Our team has more than 3 years of experience in validating Tendermint based networks. P2P Validator maintains highly-available nodes, uses advanced alerting and monitoring systems providing 24/7 technical support. <strong>We are early Agoric investors having our own skin in the game.</strong></p><p>Currently Agoric is in phase 0 of mainnet. We have prepared a <a href="https://p2p.org/economy/staking-agoric-with-keplr-wallet/">staking guide</a> to simplify delegation process. Feel free to visit <a href="https://p2p.org/agoric?ref=p2p.org">p2p.org/agoric</a> to find out more.</p><h3 id="about-agoric"><strong>About Agoric</strong></h3><p>Agoric is developing a secure distributed ocap platform for smart-contracts and market-oriented programming. It provides a reusable library of safe composable DeFi components to build decentralized applications at scale. It is built by the team that is deeply involved in the JavaScript and Cosmos ecosystem. Agoric is supported by notable venture funds including Polychain, Outlier Ventures, Placeholder, Compound and many others.</p><p>Learn more by visiting the<a href="https://agoric.com/?ref=p2p.org"> Agoric website</a>,<a href="https://twitter.com/agoric?ref=p2p.org"> Twitter</a> or<a href="https://t.me/agoricsystems?ref=p2p.org"> Telegram</a>. If you’re a developer, explore possibilities<a href="https://agoric.com/develop/?ref=p2p.org"> here</a> and join the discussion on<a href="https://agoric.com/discord?ref=p2p.org"> Discord</a>.</p><h3 id="about-p2p"><strong>About P2P</strong></h3><p><a href="https://p2p.org/?utm_source=p2p_agoric_announcement&utm_medium=blog_post&utm_campaign=announcement_about_p2p_agoric&utm_id=2">P2P Validator</a> is a world-leading staking provider with the best industry security practices and proven expertise. We provide comprehensive due-diligence of digital assets and offer only top-notch staking opportunities. At the time of the latest update, <strong>more than 3 billion of USD value is staked with P2P Validator by over 20,000 delegators across 25+ networks.</strong> We are early Agoric investors committed to provide a long term support for the ecosystem.<br></p>