Listen to this Post
Introduction:
Virtual Machine (VM) based obfuscation is a advanced protection technique that transforms original code into bytecode executed by a custom interpreter, making static analysis extremely difficult. Lifting binary code to LLVM Intermediate Representation (IR) enables security researchers to leverage LLVM’s powerful optimization and analysis passes for deobfuscation. This article explores the process of creating an LLVM-based devirtualizer for VM crackmes, bridging the gap between high-end reverse engineering workflows and accessible entry-level content.
Learning Objectives:
- Understand the architecture of VM-based protections and their impact on reverse engineering workflows
- Implement binary lifting to LLVM IR using open-source frameworks and custom passes
- Develop and test a devirtualization pass that simplifies VM dispatch loops into linear, analyzable code
You Should Know:
1. Understanding Virtual Machine Obfuscation
A VM-based protector replaces original instructions with a custom bytecode format and an interpreter (the VM) that fetches, decodes, and executes each bytecode opcode. This creates a “black box” where the real logic is hidden inside handler routines.
Step‑by‑step guide to identify a VM obfuscated binary:
- Static analysis: Look for a large loop with a switch statement dispatching to multiple handlers – typical VM dispatch pattern.
- Dynamic analysis: Trace execution flow – you’ll see repeated cycles through fetch/decode/execute phases.
- Use `objdump` or IDA Pro to locate the main dispatch loop. On Linux:
objdump -d ./crackme | grep -A 20 "<main>"
4. On Windows (PowerShell with Cygwin or WSL):
dumpbin /disasm crackme.exe | findstr "jmp"
5. Log memory accesses to identify the VM bytecode array and instruction pointer (VIP). Run under `strace` or a debugger:
strace -e trace=memory ./crackme
2. Setting Up LLVM for Binary Lifting
LLVM provides a robust infrastructure for representing, transforming, and generating code. To lift binary machine code to LLVM IR, you need a lifting frontend (e.g., McSema, RetDec, or Ghidra’s SLEIGH to LLVM).
Step‑by‑step setup on Linux:
Install LLVM and Clang (version 14+ recommended) sudo apt update sudo apt install llvm-14 clang-14 lld-14 libllvm-14-ocaml-dev Set alternatives (optional) sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-14 100 Install McSema (binary lifter) git clone https://github.com/lifting-bits/mcsema cd mcsema mkdir build && cd build cmake -DCMAKE_INSTALL_PREFIX=/usr/local .. make -j$(nproc) sudo make install
On Windows (using WSL2): Follow same Linux steps inside Ubuntu WSL. For native Windows LLVM, download from https://releases.llvm.org/ and add to PATH.
Verify installation:
clang --version opt --version LLVM optimizer pass tool llvm-dis --version
3. Lifting Binary Code to LLVM IR
Lifting translates raw machine instructions into LLVM IR, preserving control flow and data dependencies. For a VM crackme, you first lift the entire binary then isolate the VM bytecode handler.
Using McSema to lift a binary:
Disassemble the binary to CFG /d出来/mcsema-disass --disassembler /usr/bin/llvm-objdump --arch x86_64 --os linux --binary ./vm_crackme --output cfg.pb Lift to LLVM bitcode mcsema-lift --cfg cfg.pb --arch x86_64 --os linux --output lifted.bc --entrypoint main
Inspect the lifted IR:
llvm-dis lifted.bc -o lifted.ll less lifted.ll
Look for the dispatch loop – it will appear as a `br` instruction with a phi node for the VM instruction pointer. To manually extract the VM bytecode array, use a Python script with `capstone` and `pefile` (Windows) or `pyelftools` (Linux):
import pefile
from capstone import
pe = pefile.PE("vm_crackme.exe")
Find .text section
for section in pe.sections:
if b'.text' in section.Name:
code = section.get_data()
md = Cs(CS_ARCH_X86, CS_MODE_32)
for i in md.disasm(code, section.VirtualAddress):
print(f"0x{i.address:x}: {i.mnemonic} {i.op_str}")
4. Creating an LLVM Pass for Devirtualization
An LLVM pass analyzes and transforms IR. For VM devirtualization, you need to:
– Identify the VM dispatch loop (a loop containing a switch on the VM opcode)
– Inline the handler code for each possible opcode
– Replace the loop with straight-line code
Example skeleton of a LLVM pass (C++):
class VMDevirtualizerPass : public llvm::FunctionPass {
public:
static char ID;
VMDevirtualizerPass() : FunctionPass(ID) {}
bool runOnFunction(llvm::Function &F) override {
for (auto &BB : F) {
for (auto &I : BB) {
// Detect switch instruction with many cases (typical VM)
if (auto SI = dyn_cast<llvm::SwitchInst>(&I)) {
if (SI->getNumCases() > 10) {
return devirtualizeSwitch(SI, F);
}
}
}
}
return false;
}
bool devirtualizeSwitch(llvm::SwitchInst SI, llvm::Function &F) {
// 1. Trace back to bytecode fetch
// 2. For each case, extract the handler's IR
// 3. Inline handler code at the switch's location
// 4. Erase the switch and loop structure
return true;
}
};
Compile and run the pass:
clang++ -shared -fPIC -o libDevirtualize.so devirtualize.cpp `llvm-config --cxxflags --ldflags --libs` opt -load ./libDevirtualize.so -vm-devirt -S lifted.ll -o devirtualized.ll
Alternative (Python using llvmlite): For rapid prototyping, use `llvmlite` to parse IR and apply transformations.
- Testing the Devirtualizer on a Sample VM Crackme
Create a minimal VM crackme to validate your pass. Write a simple C program that uses a VM interpreter (e.g., a bytecode for XOR decryption). Compile with optimizations disabled.
Example VM crackme source (C):
include <stdio.h>
unsigned char bytecode[] = {0x01, 0x50, 0x02, 0x41, 0x03, 0x00}; // opcode, operand
int main() {
int ip = 0, reg = 0;
while (bytecode[bash] != 0) {
switch(bytecode[ip++]) {
case 0x01: reg += bytecode[ip++]; break;
case 0x02: reg ^= bytecode[ip++]; break;
case 0x03: printf("Result: %d\n", reg); break;
}
}
return 0;
}
Compile and lift:
clang -O0 -o vm_test vm_test.c ./mcsema-disass --binary ./vm_test --entrypoint main --output vm_test.cfg mcsema-lift --cfg vm_test.cfg --entrypoint main --output vm_test.bc opt -load ./libDevirtualize.so -vm-devirt -S vm_test.bc -o devirt.ll
Verify deobfuscation: The output IR should no longer contain the switch loop; instead, you’ll see sequential arithmetic and print calls. Compile the devirtualized IR to a native binary and confirm it produces the same output.
6. Advanced: Combining Symbolic Execution with LLVM
For complex VM handlers that mix data and control flow, augment your LLVM pass with symbolic execution using KLEE (built on LLVM). This allows you to discover all possible bytecode paths without brute force.
Install KLEE:
sudo apt install klee klee-utils
Concretize the VM bytecode array as symbolic:
// In your LLVM pass, mark the bytecode array as symbolic klee_make_symbolic(bytecode, sizeof(bytecode), "bytecode");
Run KLEE on the lifted IR:
clang -emit-llvm -c vm_crackme.c -o vm.bc klee --libc=uclibc --posix-runtime vm.bc
KLEE will generate test cases covering every possible bytecode sequence, effectively devirtualizing the VM by exploring all states. This is particularly powerful against VM protections that use opaque predicates or junk bytes.
What Undercode Say:
- LLVM IR is the universal translator for reverse engineering – lifting any architecture to LLVM opens the door to hundreds of analysis passes, from constant propagation to full symbolic execution.
- VM devirtualization is not just a crackme exercise – enterprise packers like VMProtect and Themida use similar techniques; understanding LLVM-based lifting gives defenders and attackers a common framework.
- Entry-level content is rare but critical – Sven Rath’s blog post addresses a niche where academic papers are too dense and forum posts too scattered. Practical, working code examples accelerate learning for the next generation of security researchers.
- Automated deobfuscation is shifting from pattern matching to IR rewriting – as more malware adopts custom VMs, traditional signature-based detection fails. LLVM passes offer a generic, reusable method to flatten control flow and inline handlers.
Prediction:
Within the next 18 months, we will see open-source frameworks that combine LLVM lifting with machine learning to automatically classify and devirtualize unknown VM bytecode handlers. Training datasets from millions of crackmes will enable models to predict handler semantics directly from IR patterns. This will force commercial protectors to adopt dynamic, self-modifying VMs that mutate per execution, leading to an arms race between LLVM-based deobfuscation and polymorphic VM generators. Security teams will increasingly embed LLVM passes into their malware analysis pipelines as a first-class step, making intermediate representation the new standard for binary normalization.
▶️ Related Video (90% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Sven Rath – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
