System Calls Explained

Analogy categoryAnalogies

What a System Call Actually Is

A system call is the only sanctioned doorway between your unprivileged (ring 3) program and the kernel running in ring 0. User code cannot directly touch a disk controller, write to another process's memory, or create a new process; it must ask the kernel to do it on its behalf. Assembly exposes this doorway directly: you load a syscall number into a register, load the arguments into other registers, and execute a single instruction that traps into the kernel.

🏏

Cricket analogy: It is like a batsman who cannot personally overturn an umpire's decision — he must request a DRS review through the on-field umpire, who alone has the authority to consult the third umpire and change the outcome.

The Linux x86-64 Calling Convention for Syscalls

On 64-bit Linux, the syscall number goes in RAX, and up to six arguments go in RDI, RSI, RDX, R10, R8, and R9, in that order. Note that R10 replaces RCX here specifically because the SYSCALL instruction itself clobbers RCX (it stores the return address there) and R11 (it stores RFLAGS there), so the ABI designers had to pick a different register for the fourth argument than the one used in the standard C calling convention. After the SYSCALL instruction executes, the kernel places its return value back in RAX; a negative value in the range of a valid errno (e.g., -EBADF) signals an error rather than a separate flag.

🏏

Cricket analogy: It is like a fixed batting order where the number 4 slot is reserved for R10 instead of the usual RCX, because RCX is busy being the designated runner (holding the return address) and cannot bat in that innings.

From int 0x80 to the syscall Instruction

Older 32-bit Linux code triggers system calls with the software interrupt INT 0x80, which looks up entry 0x80 in the Interrupt Descriptor Table and performs a full, relatively slow interrupt-gate transition. Modern 64-bit code uses the dedicated SYSCALL instruction (paired with SYSRET to return), a fast-path mechanism Intel and AMD introduced specifically for syscall entry that reads the target handler address from the LSTAR model-specific register instead of walking the IDT, cutting overhead significantly. Mixing conventions matters: a 64-bit binary that mistakenly issues INT 0x80 will hit the 32-bit compatibility syscall table, not the 64-bit one, silently invoking the wrong syscall numbers.

🏏

Cricket analogy: It is like the difference between an old-style manual scoreboard operator flipping numbered panels (INT 0x80's IDT lookup) versus a modern electronic scoreboard wired directly to the scoring computer (SYSCALL's LSTAR fast path).

nasm

; write(1, msg, len) then exit(0) using raw Linux x86-64 syscalls
section .data
    msg db "Hello from a syscall", 10
    len equ $ - msg

section .text
    global _start

_start:
    ; ssize_t write(int fd, const void *buf, size_t count)
    mov rax, 1          ; syscall number for write
    mov rdi, 1           ; fd = stdout
    mov rsi, msg         ; buf
    mov rdx, len          ; count
    syscall

    ; void exit(int status)
    mov rax, 60          ; syscall number for exit
    xor rdi, rdi          ; status = 0
    syscall

Syscall numbers are architecture-specific. The same operation, write(), is number 1 on x86-64 but number 4 on 32-bit x86, so tables like Linux's unistd_64.h and unistd_32.h must never be confused when hand-coding syscalls.

The SYSCALL instruction clobbers RCX and R11 unconditionally as part of its microarchitectural contract with SYSRET. If your assembly routine holds a live value in either register across a syscall, save it first — this is a common source of subtle, hard-to-reproduce bugs in hand-written syscall wrappers.

A system call is the controlled transition from user mode (ring 3) to kernel mode (ring 0) to perform privileged operations.
On Linux x86-64, the syscall number goes in RAX and arguments go in RDI, RSI, RDX, R10, R8, R9.
R10 is used instead of RCX for the fourth argument because SYSCALL uses RCX to store the return address.
SYSCALL/SYSRET is the fast, modern 64-bit mechanism; INT 0x80 is the legacy, slower 32-bit interrupt-gate mechanism.
The kernel returns results in RAX, with negative values in the errno range indicating failure.
SYSCALL always clobbers RCX and R11, so callers must save any live values in those registers beforehand.
Mixing 32-bit and 64-bit syscall conventions in the same binary invokes the wrong syscall table and produces silent bugs.