Security Analysis of Encrypted Mempool

Abstract

With the rapid growth of Decentralized Finance (DeFi), the challenge of Maximum Extractable Value (MEV) has become increasingly significant-particularly on Ethereum. MEV allows malicious actors to manipulate transaction order within blocks, enabling exploitative strategies such as frontrunning and sandwich attacks. In response, recent research has proposed encrypted mempools, which conceal trans- action content until after ordering is finalized, thereby reducing the exploitable surface for MEV. This thesis investigates encrypted mempool, specifically the Shutter protocol, a threshold encryption-based approach designed to mitigate MEV by hiding transac- tion contents during the mempool phase. We analyze its core architecture, under- lying cryptographic mechanisms, and proposed extensions, including its integration into Ethereum’s Proposer-Builder Separation (PBS) framework. Through this study, we identify several vulnerabilities in both the base Shut- ter protocol and its proposed Ethereum integration. We demonstrate how certain behaviors of proposers, builders, and smart contracts can be exploited to launch front-running attacks on encrypted transactions. As a proof of concept, we have also implemented this attack on a local blockchain environment to showcase its feasibility in practice. In addition, we propose mitigation strategies to address these issues and highlight open problems that need further investigations.