Uncovering Transient Execution Vulnerabilities: Catalysts For Exploitation On Modern Processors

Modern processors, engineered for maximum performance, have adopted complex microarchitectural features such as speculative and out-of-order execution. These optimizations, while delivering substantial speedups, have directly enabled a new class of security vulnerabilities: transient execution attacks. Attacks like Spectre and Meltdown exploit these very features, allowing adversaries to bypass architectural security boundaries and exfiltrate sensitive data through microarchitectural side channels. The emergence of these vulnerabilities has fundamentally redefined system security. Early research and industry responses—such as speculation barriers and microcode updates—focused on mitigating specific Spectre and Meltdown variants. However, it is now evident that transient execution vulnerabilities are far more pervasive and intricate than initially recognized. The core issue lies in the disconnect between the architectural guarantees of processors and the exploitable realities of their microarchitectures. This thesis systematically exposes and analyzes a wide spectrum of classes of transient execution vulnerabilities. The complexity of modern CPUs, coupled with evolving software abstractions, has created a landscape rife with novel attack vectors that span every layer of the computing stack. Addressing these threats demands not only the identification of new vulnerabilities but also the development of robust methodologies for their analysis and mitigation across all abstraction levels. To achieve this, the thesis rigorously investigates transient execution vulnerabilities from multiple angles. At the software level, it demonstrates how operating system mechanisms—such as synchronization primitives—are compromised by speculative execution, revealing previously overlooked attack surfaces. At the architecture-microarchitecture boundary, it uncovers how violations of architectural invariants give rise to new transient execution threats. The research further dissects the microcode of complex instructions, exposing for the first time ever transient cross-core leakage even in the presence of all conventional defenses. At the hardware design stage, it introduces directed fuzzing techniques that proactively and automatically detect instances of this class of vulnerabilities before chip design reaches production. The contributions of this work are organized around four comprehensive re- search papers, each targeting a distinct abstraction level: software-level vulnerabili- ties in synchronization primitives in GhostRace (Chapter 2), microarchitectural violations of architectural invariants in Rage Against the Machine Clear (Chap- ter 3), transient cross-core leakage via microcode and off-core instructions in CrossTalk (Chapter 4), and hardware design-level detection using directed fuzzing in BugsBunny (Chapter 5). Collectively, these research papers establish that deep microarchitectural optimizations such as transient execution can be identified and exploited across all abstraction layers, demanding holistic solutions that span the entire computing stack. This thesis also charts clear directions for future research to mitigate, detect, and prevent these vulnerabilities. In the short-term, advancing software gadget discovery and exploitability analysis will systematically eliminate code patterns exploitable by transient execution attacks from existing software running on existing vulnerable processors. While in the long-term, first, developing robust microarchitectural domain isolation will sharply curtail data leakage across microarchitectural security domain boundaries. Second, enhancing pre-silicon design fuzzing will ensure that instances of this class of vulnerabilities are caught early in the hard- ware lifecycle and fixed before tape out. Finally, integrating AI-driven verification into chip design will automate and scale the detection of subtle hardware design security flaws, providing a proactive defense against not only transient execution vulnerabilities, but also emerging hardware-design security threats. In summary, this thesis provides a comprehensive perspective on how transient execution vulnerabilities arise and can be exploited throughout the computing stack. By connecting insights from both software and hardware domains, it aims to contribute to a deeper understanding of these challenges and to offer practical approaches for improving the security of modern systems against this important class of vulnerabilities of the last decade.