Table of Contents
This article is our submission to Lido’s Ethereum censorability monitor grant.
Ever since the Ethereum merge, MEV-boost has become a significant part of the ecosystem. At the same time, the US government via the Office of Foreign Assets Control (OFAC) have imposed sanctions on certain digital addresses. MEV-relays are now divided between those which are OFAC-compliant and those which are not.
The main goal of this article is to demonstrate the influence of this censorship on blockchain degradation and propose a solution to monitor the censorship problem in the Ethereum blockchain.
For the purpose of this paper, we will refer to the time difference between when a transaction enters the mempool and is included in a block as "delay".
Our sources of data:
After the data was processed, we created a large dataset. The table below contains a description of the main variables:
This dataset is available on Google BigQuery in the table `p2p-data-warehouse.p2p_public.eth_mev_censored`.
Based on the data described above, we’ve created a dashboard with the main characteristics of Ethereum transactions.
Dashboard description
Our dashboard has 6 parts:
Our main goal is to estimate the level of blockchain degradation, i.e. longer time to verify transactions and higher transaction costs, that may be caused by censorship. We hypothesize that longer delays and higher transaction costs could be a sign of censorship.
We hypothesize that our main metrics (delay and transaction costs) could be statistically different in the following subgroups:
We also want to check the level of censorship employed by Lido validators so we are going to check the following hypothesis:
We want to highlight a certain amount of transactions whose high time delay could not be explained by normal network conditions. Such transactions will be suspect of being subject to censorship. To realise this, we must take into account the following transaction properties.
High delay
We will start by choosing all the transactions over a certain threshold for the time delay in seconds.
Successful transactions
Next, we will only consider successful transactions since failure could be a reason for the delay.
Low transaction fees
Another reason for a transaction to have a high delay could be low fees. That is why we should account for that and start by checking the transaction fee.
Previous transaction pending
Sometimes transactions could delayed simply because a previous transaction from the same sender had not yet finished. We use nonce parameters to exclude these transactions in our analysis.
After forming the truncated dataset, we will try to find out the reasons for the high delay in censored transactions: government-driven or ethical censorship. We will check the receiver and sender addresses against the sanctioned list and share OFAC/ethical censoring MEV-relays. Results across the full daily dataset can be found in our dashboard.
The code required to reproduce our results is available here
Our sample dataset has 4 798 993 transactions and of those, 153 847 (~3%) are failed transactions.
The delay for most transactions does not exceed 27 seconds (95% quantile) and almost every transaction has been delayed for only one block (block_diff = 1).
Most transactions have fees with a skewness of zero. The difference between the 99.5% quantile and the 95% quantile is greater than 4 times the fee. The table below shows the transaction time delay and fees.
Main quantiles for delay and fees
We used a rank Spearman correlation analysis due to the data having quite a few outliers that did not reveal any non-obvious relationships between the variables.
Relays: Approximately 87% of all transactions within the sample are from the top 5 relays: flashbots (43%, OFAC-compliant), max_profit (22%, non-compliant), ultra_sound (11.3%, non-compliant), agnostic_gnosis (6.62%, non-compliant) and blocknative (4.2%, OFAC). The table below showcases the main quantiles for delay among different relays.
Main quantiles for top 5 relays based on delay
Notice that flashbots relay has the biggest delay in the 5% quantile (2 seconds vs 1 second). This could partly be explained by the fact that flashbots relay has the biggest share of all relays for MEV transactions on Ethereum. To avoid this, we will use random sampling and check hypotheses on equal-size samples (in section 4.2).
Apart from the time difference in seconds, it is quite interesting to analyze the difference in the number of blocks as a time delay metric. The plot and table below show cumulative empirical probability to enter the Nth block or earlier for a random transaction. It is calculated as a share of transactions to be included in the Nth or earlier block across compliable and non-compliable MEV relays.
Empirical cumulative probability to be included in Nth block or earlier
Empirical cumulative probability to be included in Nth block or earlier for the block numbers
As the plot and table above show, it is really hard to identify any differences between the probability to be included in the Nth block between transactions under OFAC-compliant MEV-relays and non-compliant MEV-relays. Differences in probability for ethically censoring MEV-relays could be because of a low sample size.
In order to check the hypotheses correctly, in every statistical test we will generate random samples with an equal number of observations (10000).
Let’s check hypotheses 1-3. To do this, we will use Mann-Whitney and bootstrapped tests because data samples do not follow a normal distribution (We proved this fact, using a Shapiro-Wilk test and QQ-plot).
We start by using the Mann-Whitney test because of its robustness for distribution with outliers. We also backed up our results with bootstrap statistical tests.
Mann-Whitney test results on the full sample
These are the main conclusions that can be derived from the Mann-Whitney tests:
Next, we used a bootstrap test to approve the results above and, in addition, to check multiple comparison hypotheses among different relays (hypothesis 4). Also due to the fact that we use equal size samples, we can estimate the differences between metrics.
Bootstrap tests results for two subgroups
We deleted observations after the 95% quantile to calculate sample means more accurately
The plots below show the bootstrapped distribution of the sample mean differences.
Bootstrapped statistical tests proved statistically different only for MEV / non-MEV subgroups. MEV transactions are faster by 1.16 seconds on average.
Despite the censored transactions being longer by 0.12 seconds on average, this is not statistically significant. The same conclusion is valid for Lido validators - there are no statistical differences.
We then run multiple comparison tests among relays to check hypothesis 4. The tables below introduce the results of bootstrapped statistical tests with Bonferroni correction.
Relays (top 5) pairs with a significant time delay difference
Bootstrap multiple comparisons test showed that Agnostic Gnosis and Blocknative MEV-relays have lower time delay after deleting all the outliers. Note that Agnostic's relay is non-compliant but Blocknative's relay is. In addition, Flashbots MEV-relay, which has the biggest share in MEV-boost-producing blocks, has a higher time delay on average, compared to Agnostic and Blocknative.
To check hypotheses 5-6 from section 3.1, we will use a Bayesian approach. The main statistic here is the chance to beat all (probability to be best). The difference is statistically significant when the chance to beat all exceeds 95%.
We then check the probability of some transactions to be included in the Nth block in the case of OFAC/not-OFAC samples.
Results of testing probabilities to be included in Nth block between OFAC and non-OFAC transactions
There is no difference in the chance to be included within the 2nd and 10th blocks for OFAC-compliant MEV relays and non-compliant relays. The same is true for the probability to be included after the 10th block.
Results of testing probabilities to be included in an OFAC / non-OFAC block
As the table above shows, the probability to be included in an OFAC block is higher for non-Lido validators.
Here we want to select transactions whose high delay could be explained by censorship. To decrease false positives, we must take into account cases in which delay is caused by any other reason. We will use the term “missing blocks” to describe blocks produced between when the transaction first appears in the mempool and the transaction is confirmed.
Criterion 1. We assume that transactions for which the delay is greater than 8 seconds (50% quantile) are suspected of being censored.
Criterion 2. Next, let’s suppose that failed transactions (error_dummy = 1 in our dataset) are not censored.
Criterion 3. We only consider transactions for which the fees in Ethereum are higher than 50% quantile within the block.
Criterion 4. We consider transactions where some “from address” is used more than once.
Criterion 5. We must also account for how full a block is. We will compare gas usage and gas limit for every block.
Criterion 6. We should only consider transactions for which the maximum base fee is higher than the block base fee.
Criterion 7. The nonce for every transaction with a longer mempool timestamp must be higher for every sender.
OFAC criterion. The address of the receiver or the address of the sender must be in OFAC sanctioned list.
The monitoring of all transactions is on our dashboard in the section “Potential censoring transactions”. The analysis of the data can be found in the Google collab.
In this article, we attempt to estimate how censorship affects blockchain degradation within the Ethereum ecosystem. The main metrics we used to judge this were “the time difference between when a transaction entered the mempool and it got confirmed” and “the number of blocks created between the time a transaction entered the mempool and it got confirmed”.
MEV-boost and OFAC censorship
Since the Ethereum merge, the share of validators that uses MEV-boost relays has increased from 10% to 87% since November 2022. However, since the beginning of 2023, the number of blocks produced by OFAC-compliant MEV relays has decreased from approximately 75% to 40%, which translates to a decrease in the share of blocks produced by flashbots.
Lido validators
Despite every Lido validator using MEV-boost, the use of OFAC-compliant MEV relays has gone down from 57% on February 10, 2023, to 42% on February 24, 2023.
Non-Lido validators
Among other validators, the use of OFAC-compliant MEV relays almost did not change and stayed at around 50% from February 10 to February 24.
Our statistical analysis also shows that the probability to be included in an OFAC block is higher for non-Lido validators when compared to Lido validators.
Effect on blockchain degradation
In general, censorship has an insignificant effect on the Ethereum blockchain degradation.
The delay distribution does not differ statistically between OFAC-compliant and non-compliant relays. However, there are some differences among MEV relays. Agnostic (non-compliant) has the lowest time delay and Blocknative (compliant) showed the second lowest transaction delay. But these relays have, at the same time, the lowest share among all the top 5 MEV relays in blocks built over this period.
We also did not notice a statically significant difference between the delay distribution between Lido and other validators.
We did not reveal any dependencies between censorship and transaction cost.
We tried to account for false positive cases of when a high delay is caused by other reasons besides censorship. Among these reasons, we can highlight low transaction cost, failure, repeated transactions from one sender, gas used, and having to wait for transactions with a lower nonce.
This is not our final research regarding the effects of censorship on blockchain degradation in the Ethereum network. We are planning to investigate it further, while addressing the following:
Mempool quality data
Our main limitation was parsing manually mempool data rather than getting full ready-to-use mempool data. Therefore our time delay metrics could be biassed.
Rocketpool data
To compare Lido validators versus Rocketpool validators.
Empty blocks
To improve time delay metrics, considering empty blocks.
Quality of MEV-relays data
Currently, we have some problems collecting MEV-relay data. We are planning to improve this process.
<p>We are pleased to announce that P2P has launched an MEV-enabled client on Solana validators which will allow P2P stakers to receive extra rewards from MEV starting today. <br></p><p>Maximum extractable value (MEV) is a growing capability in blockchain. It refers to the maximum profit a validator can make through its ability to include, exclude, or reorder transactions in the blocks it creates. By running a MEV-enabled client on validators, P2Ps gain access to MEV Opportunities, which can increase staker APR while having a positive impact on the quality of the Solana network.</p><h3 id="mev-boosted-client-adoption">MEV-boosted client adoption</h3><p>To maximize MEV rewards for stakers, P2P will use the open-source Jito-Solana Client, which has been <a href="https://jito-foundation.gitbook.io/mev/audits/audits">audited by Neodyme</a>, a leading security and blockchain auditing firm. Jito-Solana represents a meaningful improvement to Solana’s validator software. It was intentionally designed to maximize MEV rewards and optimize their distribution to network validators and stakers. In addition, Jito-Solana was designed to combat spam and improve network efficiency. One of the ways that Jito-Solana achieved this is through its optimization of transaction processing. Jito-Solana offers more efficient transaction processing by bundling transactions and optimizing transaction ordering, which reduces the number of duplicated and unnecessary transactions and enables faster processing times.</p><p>For implementation, we conducted an in-depth analysis of the impact of the Jito MEV client on Solana’s network performance and adoption among Solana validators. You can find the full report <a href="https://p2p.org/economy/jitos-mev-boosted-client-adoption-impact-on-solana-validators-performance/">here</a>. </p><p>The key takeaways are:</p><ul><li>The Jito client has been gaining traction among Solana validator operators, as reflected by the growing number of validators running the client (currently 80), their total active stake (~55M SOL), and their market share (~15%).</li><li>We also observed a decrease in the average active stake per validator using the Jito client which indicates that more validators with smaller stakes are adopting the software. This trend is a positive sign as it shows that even smaller validators can successfully run the Jito client. The Jito client democratizes access to MEV with equal treatment for all validators. The growth in adoption by lower-stake validators demonstrates a strong interest in MEV opportunities from a community that was previously unable to access these benefits.</li><li>The total Jito-generated MEV rewards are currently relatively low due to limited adoption and lack of MEV searcher participation. This situation should improve with broader adoption of the client and as searchers become more accustomed to the new tools. We observed a significant increase in MEV rewards after January 2023 suggesting that a large DeFi participant started using Jito's MEV extraction tools and we believe that others will soon follow.</li><li>Jito-enabled validators have shown higher average uptime, a higher number of vote credits, and a significantly higher stake-weighted average block production rate.</li><li>Careful statistical analysis showed that block production rate for validators that switched to the Jito client has increased significantly. The increase is likely due to the more efficient transaction processing enabled by the Jito client's optimized block engine.</li></ul><figure class="kg-card kg-image-card"><img src="https://lh6.googleusercontent.com/YjlzfIMapEo0xV1CVCjMjZYeX1VMVKYrb-_mYKrB5D0_Ts3e6SqmHxcAnZOSO2fhMr1gA0lWNRPABEWSp62KhP5ZeU4P4AlhJgCPmj_ro9r-SV-hoEPLfV-tWq5oj-L-JpZ0aIhJU2h-0GSLwgDuUwI" class="kg-image" alt loading="lazy" width="624" height="361"></figure><p>Based on our research, we believe that the Jito client is a valuable addition to the Solana ecosystem, providing validators and their delegates with a new revenue stream from MEV capabilities while also improving the Solana network stability. Feel free to view the validator performance data, MEV validators rewards, MEV stakers rewards, and more using the public P2P Validator dashboard at: <a href="https://reports.p2p.org/superset/dashboard/jito_client_adoption/">https://reports.p2p.org/superset/dashboard/jito_client_adoption/</a>.</p><h3 id="security-assurance">Security assurance</h3><p>To implement the MEV client on P2P nodes, automation was prepared and tested to seamlessly switch between Jito and the standard Solana client with 0-downtime in this case. Infrastructure security is the most important thing and we would like to add details about the possible security risks validators may face when switching to the Jito client.</p><p>Basically, Jito client is a set of patches applied to a standard Solana client, so theoretically it may have new vulnerabilities or performance degradations in some situations on current or future releases. We discussed these issues with Jito labs and found that 2 independent auditors checked Jito code base and checked the stability of the client on our testnet validator.</p><p>Although the probability of validator performance degradation caused by Jito client is not zero, we don’t think it is a big problem since we always monitor the performance of our validators against cluster average performance statistics (vote success rate, block skip rate) with automated alerting if performance issues are detected. Generally, our monitoring system will allow us to make quick decisions about falling back to a standard Solana client.</p><p>There are certain things that must be taken into account when switching to a standard Solana client:</p><ul><li>Solana Foundation releases a new client with a request to update immediately. Usually, it happens, when the Foundation wants to close a vulnerability or fix stability issues. Jito labs need days or weeks to release its patched version (since it requires a lot of testing) so in such cases, we will switch to the standard client and wait for the Jito client release.</li><li>We detect abnormal behavior of a validator according to our monitoring metrics or detect significant performance degradation (for example vote success rate is lower than the cluster average).</li><li>We receive information from public or closed sources about a vulnerability revealed in the Jito client. In this case, all our validators immediately get moved to a standard client and we will get in touch with Jito labs to clarify the situation.</li></ul><p>If switching to a standard client happened and it has been going on for more than 2 weeks we will notify you about that on our <a href="https://twitter.com/P2Pvalidator">Twitter</a>, so please subscribe to our Twitter to be up-to-date about our services</p><p>We believe that adopting the Jito client will drive MEV rewards. Performance should also improve even further as the client becomes more widely adopted and as searchers become accustomed to the new tools.</p><h3 id="sharing-mev-rewards-with-sol-stakers">Sharing MEV rewards with SOL stakers</h3><p>New and old P2P stakers alike have the opportunity to increase their revenue through extra MEV rewards. P2P will take an 8% commission from MEV rewards, distributing 92% of the rewards to stakers, according to their share.</p><p>The Jito Tip Distribution Program is the on-chain program that collects and distributes MEV rewards to eligible validators and stakers. At the end of an epoch, the MEV rewards earned by a validator are stored in a special account derived from the validator's vote account and the corresponding epoch.</p><p>Once the epoch is over, the validator generates a data structure containing the rewards claims for all validators and stake accounts and uploads it on-chain. Then validators and stakers receive the MEV rewards in the form of an automatic airdrop performed by Jito to the validators’ vote accounts and stakers stake accounts. The claim action is permissionless, meaning others could execute it if Jito failed to do so properly. Note that distributed rewards are not automatically staked (auto compounded) and the stake account owner is required to withdraw or stake the airdropped rewards.</p><p>Stakers and validators can check their rewards using the <a href="https://reports.p2p.org/superset/dashboard/jito_client_adoption/">P2P Validator’s dashboard with Jito & MEV </a>statistics or <a href="https://jito.retool.com/embedded/public/e9932354-a5bb-44ef-bce3-6fbb7b187a89">Jito’s MEV rewards dashboard</a>. Anyone can check their rewards using public explorers, the details to do so are available in <a href="https://jito-foundation.gitbook.io/mev/mev-payment-and-distribution/faqs">Jito’s documentation</a>.</p><h3 id="about-p2p">About P2P</h3><p>P2P Validator began in 2018 with a mission to positively influence the development of POS technologies. At the time of the latest update, more than 750 million USD value is staked with P2P Validator by over 35,000 delegators across 40+ networks. We work closely with each network we support to push the developments of each project to new limits. <br></p><p>Beginning as seed investors and validating from the genesis block, we have shown tremendous support to the Solana ecosystem since day one and are now trusted with over $120m under management. Our proficiency is shown not only by our excellent validating track record & our published research papers written on network performance (<a href="https://www.stakingrewards.com/journal/solana-validators-performance-research-report-part-1-downtime-analysis/">Downtime</a>, <a href="https://www.stakingrewards.com/journal/solana-validators-performance-research-report-part-2-skip-rate-analysis/">Skip Rate</a>) to improve the network health and development, but also by our involvement across projects including <a href="https://portalbridge.com/#/transfer">Wormhole Bridge</a>, <a href="https://pyth.network/">Pyth</a>, and <a href="https://neon-labs.org/">Neon EVM</a> to help build Solana’s network infrastructure.<br></p>
from p2p validator
<p>RPC stands for Remote Procedure Call, and it refers to a protocol that allows a computer program to request services from a server in a network. In the cryptocurrency sphere, RPC is commonly used for communication between client applications and blockchain nodes. An RPC node serves as an interface for developers to interact with the blockchain, allowing them to access its data and execute transactions. The RPC protocol provides a standardized way for clients to communicate with the node, allowing them to query the blockchain, submit transactions, and perform other operations. By using an RPC node, developers can build and deploy decentralized applications on networks like Ethereum.</p><p>RPC nodes can be divided into two main areas: <a href="https://p2p.org/economy/shared-and-dedicated-rpc-nodes/">shared or dedicated</a>. Shared nodes are less secure and less reliable compared to dedicated nodes, but they are also less expensive and require less maintenance. However, dedicated nodes are the preferred choice for organizations requiring a high-security level, reliability, and performance. When working with dedicated RPC nodes, it's important to ensure that they are properly maintained and managed.</p><p>The maintenance of a dedicated Ethereum RPC node can be done in two ways: in-house or by outsourcing. In-house maintenance involves having a dedicated team responsible for managing the Ethereum node. This team is responsible for setting up, configuring, and maintaining the node, as well as troubleshooting any issues that may arise. This approach requires significant investment not only in hardware and software but also in terms of personnel and can be time-consuming and resource-intensive.</p><p>Outsourcing, on the other hand, involves contracting a specialized service provider to handle the maintenance and management of a node. This approach allows the organization to benefit from the expertise and resources of the service provider, who is responsible for ensuring the node is up-to-date, secure, and functioning properly. By outsourcing the RPC node maintenance we can reduce costs, improve security, and free up valuable time and resources that can be redirected to more critical tasks.</p><p>The following are some key benefits of outsourcing RPC node maintenance.</p><h3 id="saving-time-and-resources">Saving Time and Resources</h3><p>Maintaining an RPC node requires continuous monitoring and troubleshooting to ensure that the node is up-to-date and functioning properly. This can consume a significant amount of time and resources from a company's development and operations teams. By outsourcing this task to a specialized service provider, companies can free up valuable time and resources that can be redirected to more critical tasks.</p><h3 id="reliability-and-security">Reliability and security</h3><p>Outsourcing the maintenance and management of Ethereum RPC nodes can provide a high level of reliability and security, ensuring that the node is always up-to-date, secure, and functioning properly. Service providers specializing in Ethereum RPC node maintenance offer expertise, round-the-clock monitoring, quick response times, and regular maintenance to minimize the risk of downtime and technical issues. By outsourcing, organizations can improve the reliability of their systems, reduce risk, and increase operational efficiency, leading to improved customer satisfaction.</p><h3 id="reduced-costs">Reduced Costs</h3><p>In-house maintenance of an RPC node requires significant resources, including hardware, software, and personnel. By outsourcing it, companies can reduce their costs and minimize the need for expensive hardware and software investments. Additionally, they can avoid the cost of hiring and training employees to handle maintenance tasks.</p><p>For example, let's assume that an in-house DevOps engineer earns $100,000 per year, and it takes approximately 300 hours per year to maintain an Ethereum RPC node (research, updates, support, etc.). So the cost of in-house maintenance would be around $16500 for salary alone. RPC nodes also require servers to function. For an Ethereum RPC node, the price will be around $500 per month.</p><p>It turns out that the actual cost of supporting one node would be around $22500. This approach, however, won't provide you with the confidence that it will work properly around the clock. In most cases, one engineer cannot adequately cover all tasks, since failures and problems can occur at any time (at night, on holidays, etc.). For this reason, a 24/7 team is necessary. </p><h3 id="conclusion">Conclusion</h3><p>Outsourcing Ethereum RPC node maintenance can provide a wide range of benefits, including saving time and resources, improved security, and reduced costs. By outsourcing this task to a professional service provider, companies can focus on their core business functions while ensuring that their RPC node is properly maintained and secure. This can result in significant cost savings and a more efficient use of resources.</p><p>Unleash the full potential of your Ethereum or other network projects with our dedicated RPC node maintenance services. Partner with us at P2P and let us take care of the technicalities while you focus on developing innovative, game-changing products. Whether you're diving into the world of Web3 infrastructure or simply need reliable maintenance, we've got you covered. Contact us now to experience a hassle-free and seamless journey towards success!</p><p>Contact us at [email protected]</p>
from p2p validator