The Scourge of x86 Emulation: Why FEX’s Memory Model Hack Is Costly

FEX has a problem on its hands, and it is called x86-TSO. The company writes emulators for x86 systems, and the memory model those systems use is strictly ordered — every store is visible to every other processor, every load sees every earlier store. That is the opposite of how modern ARM CPUs work, where memory is weakly ordered and stores sit in caches until they are flushed out.

The result is a constant battle. FEX takes every x86 memory load and turns it into an ARM load-acquire instruction. Every x86 store becomes an ARM store-release instruction. That buys the emulator the same memory semantics as x86, but it is an expensive trick. ARM CPUs were not built for acquire and release instructions to be the vast majority of what they run, and FEX knows it.

The Memory Model Gap

A memory model is a set of rules for how memory accesses behave in relation to each other. x86-TSO is the strict end of the spectrum. A programmer can assume that when a store happens, it is visible to all processors in the system. A load sees all stores before it, logically completed or at least visible. The stores order the visibility of the loads, which is where the name comes from.

ARM’s model is the other end. Regular loads and stores are not coherent across processors by default. A store does not make its cacheline visible to other cores immediately. A processor loading data from memory that another processor has written to is not guaranteed to see the updated value. That is the weak consistency that lets ARM CPUs save power and run efficiently most of the time.

The two models are not the same. Consistency and atomicity are related but separate. ARM introduced load-acquire and store-release instructions to force ordering, and in C++ terms these map to std::atomic’s memory_order_acquire and memory_order_release. ARM calls the resulting model Release Consistency sequentially consistent (RCsc). Acquire loads must be observed sequentially without reordering. Store-releases must fulfill “barrier-ordered-before” semantics. The older memory barrier instruction was costly, and these new instructions removed it.

ARMv8.0-a and the Cost

FEX starts its emulation journey with ARMv8.0-a, where the strategy is simple: make all x86 loads into acquire instructions, all x86 stores into release instructions. That gives FEX the same memory semantics as x86, but the company admits it is being stricter than necessary. The microbenchmarks show the cost. It is exceedingly costly to emulate TSO this way, and ARM CPUs were not designed to have these rare instructions suddenly become the vast majority of what they execute.

The company tested this with a microbenchmark that is nice to the hardware. No tricky edge-cases, just accessing memory in the common case. The graph tells a few stories. The Load and Store columns represent the baseline performance number.

Ranking the Instructions

1. ARMv8.0-a’s emulation strategy: all x86 loads become acquire instructions, all x86 stores become release instructions.
2. The older memory barrier instruction: costly and replaced by the newer acquire and release instructions.
3. The newer load-acquire and store-release instructions: cheaper than the older barrier instruction, but still expensive when used as the vast majority of execution.

What We Know So Far

• x86-TSO is a strict memory model where stores are visible to all processors and loads see all earlier stores.
• ARM’s model is weakly consistent, meaning stores are not visible to other cores immediately.
• ARMv8.0-a emulation converts x86 loads to acquire instructions and stores to release instructions.
• Microbenchmarks show the approach is exceedingly costly.
• The older memory barrier instruction was replaced by load-acquire and store-release.

The Verdict

FEX has a working solution, but it is expensive. The company is aware of the cost and has microbenchmarks to prove it.

The lesson is that emulation is not free. Every abstraction costs something, and memory models are among the hardest abstractions to bridge.

The story is worth reading for anyone who cares about how memory behaves inside a computer.

Source material: “The scourge of x86 emulation,” fex-emu.com.

Arm · emulation · FEX · memory model · x86