Listen to this Post
Introduction:
Direct system calls are a cornerstone of advanced Windows programming, often used in malware development and red teaming to bypass user-mode API hooks. However, as demonstrated in a recent deep-dive, invoking a syscall like `NtAllocateVirtualMemory` without respecting the Application Binary Interface (ABI) leads to silent corruption and crashes. This article dissects the journey of building a direct syscall stub, revealing how a single misplaced argument can misalign registers and break the stack, ultimately teaching the critical importance of ABI compliance in low-level Windows internals.
Learning Objectives:
- Understand the Windows System Call ABI and its critical role in kernel-mode transitions.
- Learn to reverse engineer `ntdll.dll` to extract syscall IDs and build functional stubs.
- Implement a dynamic syscall dispatcher using C and inline assembly to maintain ABI integrity.
You Should Know:
1. Reverse Engineering the Syscall Interface
The journey begins inside ntdll.dll, the user-mode gateway to the Windows kernel. Each native API, such as NtAllocateVirtualMemory, consists of a small stub that loads the syscall number (Service ID) into the `eax` register and executes the `syscall` instruction. To replicate this, we must extract these IDs and replicate the exact prologue and epilogue.
Step‑by‑step guide:
- Use a disassembler like IDA Pro or Ghidra to locate `NtAllocateVirtualMemory` within
ntdll.dll. The function typically appears as:mov eax, 0x18 ; Syscall ID (varies by Windows version) mov r10, rcx syscall ret
- Extract the syscall ID for the target function. This ID can change between Windows builds, making version-agnostic approaches necessary.
- Create a header file with the syscall prototype and a helper function to retrieve the ID dynamically by parsing the PEB at runtime. Example C structure:
typedef NTSTATUS(NTAPI pNtAllocateVirtualMemory)( HANDLE ProcessHandle, PVOID BaseAddress, ULONG_PTR ZeroBits, PSIZE_T RegionSize, ULONG AllocationType, ULONG Protect );
2. The ABI Mismatch Trap
The initial failure occurred because an extra argument (syscall_id) was passed as if it were a standard parameter. In the Windows x64 ABI, the first four integer arguments are passed via rcx, rdx, r8, and r9. The syscall number, however, must reside in `eax` before the `syscall` instruction. Adding an extra C function argument shifted the register assignment, causing the kernel to read garbage values from the wrong registers.
Step‑by‑step guide to understand the ABI:
- The Windows x64 calling convention mandates that the caller reserves shadow space (32 bytes) on the stack before calling a function.
- For a direct syscall, the stub must preserve the original arguments in
rcx,rdx,r8, `r9` while loading the syscall ID intoeax. - Incorrect implementation leads to:
- Registers misalignment: The kernel’s `KiSystemService` routine expects the syscall number in `eax` and parameters in the standard registers.
- Stack corruption: Improper handling of the return address and shadow space can cause crashes upon return from kernel mode.
Example of a correct inline assembly stub in MSVC:
NTSTATUS NtAllocateVirtualMemory_Stub(HANDLE ProcessHandle, PVOID BaseAddress, ULONG_PTR ZeroBits, PSIZE_T RegionSize, ULONG AllocationType, ULONG Protect) {
NTSTATUS status;
DWORD syscall_id = get_syscall_id(L"NtAllocateVirtualMemory"); // Dynamically resolved
__asm {
mov eax, syscall_id
mov r10, rcx ; Save first argument to r10 as per ABI
mov rcx, ProcessHandle
mov rdx, BaseAddress
mov r8, ZeroBits
mov r9, RegionSize
push [rsp+0x28] ; Push remaining arguments onto stack
sub rsp, 0x20 ; Shadow space
syscall
add rsp, 0x20
pop r10 ; Clean up stack
mov status, eax
}
return status;
}
3. Building a Dynamic Syscall Dispatcher
To avoid hardcoding syscall IDs and writing individual stubs for hundreds of functions, a dynamic dispatcher can be implemented. This approach leverages a custom assembly function that accepts the syscall ID and variable arguments, then sets up the correct context.
Step‑by‑step guide:
- Write a generic assembler function `SyscallDispatcher` that takes the syscall ID in `ecx` and a pointer to an argument array in
rdx. - Inside the assembly, move the arguments from the array to the appropriate registers and stack positions.
- Use a loop to push any arguments beyond the first four onto the stack.
- Execute the `syscall` and return.
- This method centralizes the ABI logic and simplifies calling multiple syscalls. A simplified version in MASM:
SyscallDispatcher PROC mov eax, ecx ; Syscall ID mov r10, rdx ; Arg array pointer mov rcx, [bash] ; Arg 1 mov rdx, [r10+8] ; Arg 2 mov r8, [r10+16] ; Arg 3 mov r9, [r10+24] ; Arg 4 ; Push args 5+ if any sub rsp, 0x20 syscall add rsp, 0x20 ret SyscallDispatcher ENDP
4. Debugging the Crash with WinDbg
When the syscall crashes, kernel debugging is essential. Using WinDbg, you can set breakpoints on `nt!KiSystemService` to inspect the values received by the kernel.
Step‑by‑step guide:
- Attach WinDbg to the target system in kernel mode (or use a virtual machine).
- Set a breakpoint:
bp nt!KiSystemService. - Run the crashing application and observe the registers.
- Check if the syscall number in `eax` matches the expected value.
- Dump the parameter registers (
rcx,rdx,r8,r9) and compare them with what you intended. - The mismatch will immediately highlight the ABI violation.
5. Security Implications: Bypassing User-Mode Hooks
Direct syscalls are often used in offensive security to bypass EDR/AV user-mode hooks placed in ntdll.dll. By calling the syscall directly, you circumvent the hooked functions. However, modern EDRs have started monitoring syscalls themselves using kernel callbacks and ETW, making this technique less stealthy.
Step‑by‑step guide for red teamers:
- Use direct syscalls to invoke `NtCreateThreadEx` and `NtWriteVirtualMemory` for process injection.
- Ensure the syscall IDs are resolved at runtime to maintain compatibility across Windows versions.
- Combine with indirect syscalls (jumping to the `syscall` instruction from a clean copy of
ntdll) to evade more sophisticated hooks. - Example of indirect syscall in C:
// Find the syscall instruction address in a fresh ntdll BYTE syscall_ptr = (BYTE)GetProcAddress(ntdll, "NtAllocateVirtualMemory") + 0x12; // Use assembly to call that address after setting eax
6. Linux vs. Windows Syscall ABI
Understanding the Windows ABI in contrast to Linux highlights platform-specific nuances. Linux’s x86_64 syscall ABI uses `rax` for the syscall number and rdi, rsi, rdx, r10, r8, `r9` for parameters, with no shadow space requirement. This difference often trips developers transitioning between the two.
Comparison commands:
- On Linux, view syscall definitions with: `man 2 syscall`
– On Windows, use the `dumpbin /exports C:\Windows\System32\ntdll.dll` to list exported functions, though syscall IDs are not listed. - Use tools like `SysWhispers` to generate syscall stubs for Windows.
7. Hardening and Detection
For defenders, detecting direct syscalls involves monitoring for anomalies such as `syscall` instructions originating outside ntdll.dll. Kernel drivers can use `PsSetCreateThreadNotifyRoutine` and `ObRegisterCallbacks` to inspect thread creation and handle operations, looking for suspicious call stacks.
Step‑by‑step guide for detection:
- Deploy a kernel driver that hooks the `syscall` entry point or uses ETW (Event Tracing for Windows) to log system call invocations.
- Analyze the caller address: if it is not within
ntdll.dll, flag it as potential direct syscall abuse. - Use tools like `Sysmon` with Event ID 10 (ProcessAccess) and custom configuration to detect anomalous access patterns.
What Undercode Say:
- Direct syscall implementation requires meticulous adherence to the Windows x64 ABI, as even a single extra argument can cause catastrophic failure.
- Dynamic syscall dispatchers and indirect syscalls offer flexibility and evasion capabilities but add complexity that must be carefully debugged.
- The low-level internals of `ntdll.dll` and kernel mode remain a critical frontier for both offensive and defensive security professionals.
Prediction:
As EDR solutions evolve, the battle will shift further into kernel space, with vendors implementing more robust syscall monitoring and anomaly detection. The use of direct syscalls will likely become less effective without combining them with other evasion techniques, such as API obfuscation and kernel callout bypasses. Simultaneously, the demand for experts who understand the intricate details of Windows ABI and syscall internals will surge, making this knowledge a valuable asset in both red team and blue team roles.
▶️ Related Video (82% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Godwin Jose – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
