Due to an old version of an indexer agent combined with a short-time node desynchronisation, closed allocation resulted in a zero proof of indexing on February 6th 2021. Due to this rewards for the epoch vanished. Lost rewards will be fully compensated by waiving our fee and sharing our own indexing rewards with our delegators.
An oversight in an upgrade process led to the indexer-agent version being out of date. A minor hiccup in Ethereum node operation coincided with the allocation closing, resulting in the closing of an allocation without a proof of indexing being submitted. Rewards will be fully compensated over the course of 12 days by charging 0% fee and sharing our own indexing rewards.
We provide regular reward stats on a daily/weekly basis for delegators with locked GRT and thus had no monitoring of epoch rewards after closing an allocation. This led to a minor delay in reaction as reports are filled automatically without human intervention. Right after realising that rewards had been lost we set up an internal investigation.
Additionally was the absence of a well-reviewed upgrade procedure for new indexer releases. The upgrade procedure had no clear standard and checklist of actions to follow. New allocations were opened and no technical mistakes occurred, but existing monitoring that allocation was insufficient. We noticed that monitoring for a subgraph syncing delay had insufficient resolution time-wise so failed to catch a small hiccup in Ethereum node operation.
We close and open allocations on a daily basis so we lost only one day of rewards. We collect statistics on stake changes every day and spotted a problem with reward crediting. We also thank our delegators who have noticed the issue and pointed concerns out in our Telegram channel.
All our Graph Network delegators lost one day of indexing rewards. To compensate and mitigate their loss, P2P will waive the fees for 12 days in addition to distributing all the rewards from our own indexer stake for that period.
We should have had monitoring for new version releases and checked that software is up to date before taking actions. To avoid such issues in the future we have defined a clear written procedure for upgrades and set monitoring for all necessary releases. This case revealed flaws in our existing monitoring so we have now initiated a deep analysis of all metrics improving it from ground up and adding necessary alerts to support this.
We have decided to make reviews for all standard operating procedures more rigorous by allocating more engineering time to these activities. Testing and improvement will be continued on the new testnet where we are already set up. All the crucial operations, upgrades and innovations that can potentially lead to a financial loss will be implemented on testnet first.
P2P takes full responsibility for this and we are sorry for the inconvenience. Please be assured that P2P is taking actions to eliminate even a small probability of such an event occurring in future.
Research & Analytics at p2p.org.
This post will outline incentives for Solana token holders to stake SOL and earn rewards by securing the network.Stake your SOL and earn up to 8% in staking rew...
from p2p validator
We are pleased to announce our collaboration with Advanced Blockchain AG to manage their Polkadot (DOT) staking activities - validating their 7-figure USD equiv...
from p2p validator