Oasis Staking Guide: Stake Oasis (ROSE) with Ledger

<p>We are pleased to announce a collaboration with <a href="https://robonomics.network/?ref=p2p.org">Robonomics</a> for the launch of parachains on the Kusama (KSM) network. Through this partnership, <a href="https://p2p.org/?ref=p2p.org">P2P Validator</a> will roll out, maintain and support infrastructure for Robonomics parachain when Robonomics wins a slot within a Kusama parachain auction. </p><p>P2P Validator will help to ensure a smooth launch and sustained support of Robonomics parachains through rolling out an optimal number of collators and full nodes to organise all associated parachain data. In addition to this, we will work with Robonomics to produce new block candidates by passing them to the relevant validators across the Kusama relay chain.</p><p>Robonomics is an open-source server-less IoT platform on top of Ethereum and Polkadot. Its main goal is to bring the economy of robots into the 4th industrial revolution.</p><p>Kusama parachain is an excellent opportunity for Robonomics to scale up to 1 million IoT devices connected in real-time without any centralized regulators.</p><p>Through this partnership, P2P aims to demonstrate our technical infrastructure capabilities across the Kusama blockchain. As a leading non-custodial staking provider securing more than $3 billion in assets across 25+ staking networks, we are thrilled to work with Robonomics on this exciting project. </p><hr><h2 id="useful-resources">Useful Resources</h2><p>To find out more about Robonomics crowdloan campaign, visit the <a href="https://robonomics.network/kusama-slot/?ref=p2p">dedicated crowdloan page</a>. Don’t forget to unbond your KSM to participate in the crowdloan. </p><p>To learn more about Polkadot and Kusama, please visit P2P’s support centre: “<a href="https://help.p2p.org/en/articles/5161573-kusama-ksm-staking-faq?ref=p2p.org"><strong>Kusama Staking FAQ</strong></a>” &amp; “<a href="https://help.p2p.org/en/articles/5094261-polkadot-dot-staking-faq?ref=p2p.org"><strong>Polkadot Staking FAQ</strong></a>”.</p>

<h3 id="summary">Summary</h3><p>115 validators of different staking providers received "unapplied slash" during the era 2249 in Kusama due to not sending "I'm online" signal. 45 out of 115 validators are P2P validators where a possible slash would be around 0.6% of active stake.</p><p>Unapplied status means that it can be reverted through governance action, which is exactly what happened in that case.</p><h3 id="customer-impact">Customer Impact</h3><p>All slashings have been reverted and no customer funds are impacted thanks to councils who supported the request and voted for the <a href="https://kusama.polkassembly.io/motion/295?ref=p2p.org">motion 295</a> to cancel the slashing.</p><h2 id="what-happened">What Happened</h2><p>Almost all of the active P2P Validator nodes were running <strong>0.9.0 version</strong>. Monitoring showed perfect performance although sometimes we detected CPU spikes during the election.</p><p>On May 13, there was a release note issued for the <a href="https://github.com/paritytech/polkadot/releases/tag/v0.9.1?ref=p2p.org">0.9.1 version</a>. The release had “Low (upgrade at your convenience)” upgrade priority. P2P started to roll out this version gradually because this is our SoP for low priority upgrades.</p><p>On May 14, we got a notification via Kusama Validator Lounge chat which said: <em>“Just a heads up that the<strong> </strong>9010 runtime upgrade will happen in ~55 minutes at block 7,468,792. If your node is not on at least version v0.9.0, it will not be able to sync after that”</em></p><p>After the runtime 9010 had been applied, part of our monitoring services based on @polkadot-js/api stopped working due to it being an older version than required with the fresh runtime upgrade. We had an alert which would  have been triggered if all CPU cores had 95%+ load at once, but as only one core had been overloaded, it didn't fire. Therefore, we didn’t figure out that a  number of our nodes would get slashed. It took about 40 mins to understand what exactly had happened.</p><p><strong>See diagrams:</strong></p><p>CPU time of node under v0.9.0 that experienced CPU load during the election.</p><figure class="kg-card kg-image-card"><img src="https://lh4.googleusercontent.com/xbMcNNQasSYF-CWTDqRaczqDUKVhZcf2FFbTC5_ubeGcZs1mpMGCES6MOxYgZxW0NtLCNl4Nzcb5qNp5IzYk0BcgHQCrp2s0qVazGGMvct2QbkIgbpMlDX0dfNGcSPE373RGV0pJ" class="kg-image" alt loading="lazy"></figure><p>CPU time of node under v0.9.1 - no spikes</p><p></p><figure class="kg-card kg-image-card"><img src="https://lh5.googleusercontent.com/uqe0ePHHepSaTDa5rueOthMvifsFhezANDS7Y-vkQ5QxbfSO1QuPJbHiqVcwf9I8Xshh7pd2uzS-VWtzbZEGtJzgO02JojrjRAAaOEZwpaVdiAMNg835F0KHctcxPkuSUEab1XGO" class="kg-image" alt loading="lazy"></figure><p>Logs records nodes running on v0.9.0</p><p></p><p>Looking at logs it becomes clear that after the runtime upgrade, nodes were experiencing some issues with performance. We caught 61780 events, which is a huge number and very unusual.</p><figure class="kg-card kg-image-card"><img src="https://lh3.googleusercontent.com/bQTU71RRrNxVcfTAW8tKOgg-FQmrsrbYq3k5v2DeFqyyh4rWq3oyb-hbUq30BfzL5QdeuCda8XisWvr1C5i62oYkFPMpADfKWv5zhcdzbYGrC6tieoJzZXwTHDd2syjP0qB2Xk9l" class="kg-image" alt loading="lazy"></figure><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/tyErRh-yx8nhKGrEDIWdEcEwdmMKQfH-bv0w0VCNaZ_kG3yFmS2-7wNJq-EJmERFnoiUl1tV8LWSXpINXw8lt6hOoJ_Zyf5z9QoQpmTplkZUNfNLAHs2HalIv42I7QvVeE4PYT59" class="kg-image" alt loading="lazy"></figure><p><br>During CPU spikes, our validators were not able to send "heartbeat" events. Therefore all 115 validators got slashed due to the concurrent unresponsiveness of a bunch of validators, see offence <a href="https://wiki.polkadot.network/docs/en/learn-staking?ref=p2p.org#slashing">Level 2</a>.</p><p></p><p><strong>In order to avoid further slashing, P2P team undertook the necessary steps as follows:</strong></p><p><strong>May 14, 21:30:00</strong></p><ul><li>Fix monitoring service</li></ul><p><strong>May 15, 00:30:00</strong></p><ul><li>Validation is recovered on all P2P validators</li></ul><p><strong>May 15, 00:40:00</strong></p><ul><li>Investigation in collaboration with other nodes operators and the Kusama dev team occurred.</li></ul><p><strong>May 15, 05:00:00</strong></p><ul><li>All Kusama nodes have upgraded to v0.9.1 that solved CPU spikes issue</li></ul><p><strong>May 15, 19:00:00</strong></p><ul><li>Set “--wasm-execution Compiled” flag for all Kusama nodes<br></li></ul><p>----</p><p>Slashing was reverted by one of council member Raul Romanutti who proposed cancellation of slash submitting the <a href="https://kusama.polkassembly.io/motion/295?ref=p2p.org">motion 295</a> and no user funds were affected.</p><p>----</p><h3 id="lessons-learnt">Lessons Learnt</h3><ul><li>We should have  more resilient monitoring for our nodes and infrastructure</li><li>Follow polkadot-js releases to upgrade libraries on time</li><li>Look closely at network status during runtime upgrades</li></ul><h3 id="what-s-next">What’s Next</h3><ul><li>We are going to work on new infrastructure and improvements for monitoring services that will help us to avoid such cases.</li></ul>