Operating System Protection: Isolation and Virtual Machines

A program needs enough freedom to calculate quickly without gaining control of the whole computer. User mode lets it run ordinary instructions directly. Kernel mode, also called supervisor mode, gives the operating system access to the instructions that control the machine.

Let hardware enforce privilege

The CPU carries a current privilege level. Addition, loads, stores and jumps normally run at full speed, subject to memory permissions. The kernel does not inspect each instruction in software. This is limited direct execution: run directly until a protected operation or an interrupt needs the kernel.

A privileged instruction traps if user code tries it. On x86-64, HLT, writing the page-table base register CR3, and installing an interrupt table with LIDT need kernel privilege. Otherwise an application could stop the CPU, replace its own memory boundaries or redirect interrupt handling.

Suppose an application executes HLT. The CPU raises a general-protection fault instead of halting. It enters the kernel's registered handler. Linux normally sends the process SIGSEGV, ending that process while other programs continue. The application cannot choose a convenient address inside the kernel and start executing there.

Cross the boundary through a controlled entry

Traps, exceptions and interrupts enter through addresses registered by the kernel. Hardware changes privilege and transfers control to the handler; kernel entry code preserves the state needed to return. A special return instruction restores user execution at the saved program counter. On x86-64 the return can use SYSRET or IRETQ.

The timer makes this control useful even when a program never asks for a service. Before running it, the kernel programs a timer. When the interrupt arrives, the scheduler can select another process. User code cannot disable interrupts or reprogram that timer, so an endless loop cannot keep a core forever.

Read privilege diagrams correctly

The numbering conventions differ: a lower x86 ring number means more privilege, while a higher ARM exception level means more privilege. Intel virtualization also adds VMX root and non-root operation; these are separate from the ring number.

Keep identity separate from CPU mode

Root is a user identity, UID 0, that kernel software recognizes. A root application's ordinary code still runs in user mode and enters the kernel through system calls. A device driver executing in kernel mode has much more direct power: a memory bug there can damage kernel state without passing a file-permission check.

The layer enforcing a boundary must control the layer inside it. The kernel enforces process boundaries with CPU modes and memory tables. A hypervisor similarly controls guest operating systems using hardware virtualization support.

A system call is a controlled request for a kernel service. Opening files, creating processes and sending network data all require decisions an application cannot safely make for itself. Entering the kernel gives the handler authority; it does not give the caller permission to do whatever it requested.

Follow the request

A C library wrapper such as open() places the call number and arguments in registers. It uses SYSCALL on x86-64, SVC on ARM64 or ECALL on RISC-V. The kernel then:

1. Enters through its registered entry point and saves user registers.
2. Selects the handler from the system-call table, rejecting an unknown number.
3. Reads the caller's identity and credentials from the current process.
4. Checks arguments, access rights and the objects involved.
5. Performs the operation or returns an error.
6. Restores the saved state and returns to user mode.

The identity comes from process credentials, not a user-supplied name. A caller cannot gain another user's rights by adding that user's name to a request.

Treat arguments as untrusted

A pointer is just a number supplied by the caller. The kernel must verify the whole requested user-memory range, including its length and access direction. Linux uses checked routines such as copy_from_user and copy_to_user to cross the boundary safely.

For read(fd, buf, 4096), a buf pointing into kernel memory must not become a destination for file data. Linux reports EFAULT for that bad user buffer. A different error has a different cause: write() through a descriptor opened with O_RDONLY fails with EBADF, because the descriptor was never granted write access. An ordinary denied request returns an error; it does not automatically end the process. In C, the wrapper commonly returns −1 and sets errno.

Check the object actually used

A time-of-check-to-time-of-use (TOCTTOU) race separates a decision from the object it was meant to protect. A privileged helper checks access("/tmp/report", W_OK), then opens the path. An attacker replaces the path with a symbolic link to /etc/passwd between those operations. The name is unchanged, but the opened object is different.

The repair is to perform the access under the intended credentials on the object actually opened. Drop privileges before opening, and use flags such as O_NOFOLLOW when symbolic links are unwanted. Copying arguments into kernel memory once also avoids repeatedly trusting mutable user data; the copied structure must still be validated.

Identify what must be trusted

A reference monitor mediates access. It must be invoked whenever access is acquired, protected from tampering and small enough to examine carefully. After the kernel grants a memory mapping, individual loads and stores go through the MMU's permission checks; they do not each make a system call.

The trusted computing base (TCB) is the hardware, firmware and privileged software whose correctness protection depends on. It includes the CPU, boot chain, kernel and trusted login components. Moving drivers or file systems into user-mode servers can shrink the kernel portion of this base, as microkernel designs do. A compromised ordinary application misuses its granted rights; a compromised trusted component may undermine the checks themselves.

Virtual memory protects programs as well as translating addresses. A process can reach the physical frames named by its active page table, with the permissions the kernel set. Another process's private frames are absent; deliberate shared-memory mappings are an explicit exception.

Check each memory access

The memory management unit (MMU) checks page-table entries on accesses. Important permissions include present or valid, writable, user-accessible and executable. On x86, U/S distinguishes user-accessible from supervisor-only pages; NX or XD disables execution. ARM calls its execute restriction XN.

A violation raises a page fault. The fault reports a failed hardware check; the kernel decides whether that failure is expected and repairable. Consider two stores to read-only pages:

• A process overwrites a code page that is not copy-on-write. The kernel treats it as an invalid write and normally sends SIGSEGV.
• After fork(), a process writes shared data marked copy-on-write. The kernel allocates a private copy, changes the mapping to writable and retries the instruction.

Both faults start from a protection bit. Kernel records explain why the mappings were read-only and determine the response. A page fault is therefore not automatically a crash or a request to fetch data from disk.

Protect kernel mappings in both directions

Traditional layouts map kernel memory into process address spaces as supervisor-only, making kernel entry convenient while blocking user-mode reads and writes. The 2018 Meltdown attack showed that some processors could leak such data through speculative execution. Kernel page-table isolation (KPTI) removes most kernel mappings while user code runs on affected systems.

Kernel code also needs protection against being redirected into user-controlled memory. x86 SMEP prevents kernel-mode execution from user pages. SMAP blocks kernel-mode data accesses to user pages except during deliberately authorized copy operations. ARM provides related protections through PXN and PAN. These checks reduce the ways an attacker can use a kernel bug to execute code or supply fake kernel data.

Read-only access protects integrity, not secrecy. If a process can read a mapped page, it can learn its contents even when writes fail. Confidential data must be inaccessible to that process: leave it unmapped or enforce an appropriate supervisor-only boundary.

Give devices their own boundary

Applications usually request device work through a driver. Device registers are normally kernel-only, though an authorized process can receive selected mappings, as with graphics drivers or Linux userspace I/O.

A device performing direct memory access (DMA) reads or writes RAM without sending those accesses through the CPU's MMU. CPU page permissions alone cannot stop a faulty device overwriting another process's frames. An IOMMU translates and restricts device addresses with per-device mappings. Intel VT-d and AMD-Vi supply this mechanism.

For a device assigned to a VM, the IOMMU restricts DMA to that VM's assigned memory. The CPU side needs a parallel boundary: a second stage of page translation maps guest physical addresses only to frames the hypervisor permits. Protecting a workload requires covering both CPU accesses and device accesses.

A memory-safety bug can become an attack when corrupted data changes the program's control flow. A stack buffer overflow, for example, can overwrite a saved return address. Operating-system defenses restrict what an attacker can do with that bug; they do not repair the faulty write.

Separate data from executable code

The NX bit marks data pages non-executable. Windows calls this Data Execution Prevention (DEP). Injecting instructions into a stack buffer is less useful when jumping there causes a fault.

W^X, write xor execute, is stricter: no page is writable and executable at the same time. DEP alone does not forbid a program from explicitly requesting such a mapping. Under W^X, a just-in-time compiler writes generated code into writable memory, then changes its permissions before executing it.

An attacker can instead reuse existing executable instructions. Return-oriented programming (ROP) chains short instruction sequences called gadgets. Address space layout randomization (ASLR) changes the locations of stacks, heaps, libraries and position-independent programs, making gadget addresses hard to predict.

With a library base chosen uniformly from $2^{28}$ positions, one blind guess succeeds with probability 1 in 268,435,456. With only $2^8=256$ positions and a layout preserved across worker crashes, trying distinct positions needs at most 256 attempts and 128.5 on average. A leaked address inside a known library reveals its base and gadget offsets in that library. A separately randomized stack can still remain unknown.

Reduce the services an attacker can request

Linux seccomp filters restrict the system calls a process may make. A browser renderer, for example, can be denied calls for launching programs or directly opening files. Filters are inherited by children and cannot be removed; later filters can tighten restrictions. An exploited renderer still has its allowed operations, so a filter is a boundary around services, not proof that the program is bug-free. OpenBSD pledge and the macOS App Sandbox provide related restrictions.

Check startup and preserve evidence

Secure boot checks signatures along the boot chain. Firmware checks a trusted boot loader; the loader checks the kernel; the kernel can require signed modules. A modified component without a trusted signature is refused. This helps stop persistent changes to boot software, but a valid signature does not prove the software has no vulnerabilities.

Measured boot records hashes of loaded components, commonly using a Trusted Platform Module (TPM). These measurements support attestation: another party can compare what started with an expected state. Measuring and refusing to start are different operations.

Audit logging records logins, authentication failures, permission denials, account changes and privileged actions such as sudo. Logs support detection and reconstruction. An attacker who controls a host may erase local records, so protected or remote logging preserves evidence beyond that host.

Attackers can combine an address leak, a memory corruption bug and a sandbox escape. Updates remove the underlying vulnerabilities that these defenses leave in place.

Successful login connects a human identity to kernel-held process credentials. Authentication checks the claimed identity; authorization decides what that identity may do. A principal may be a person or a service account, and processes act on its behalf.

Carry identity into running programs

Unix-like systems identify users by numeric UIDs and groups by GIDs. /etc/passwd maps names to UIDs, home directories and shells. Kernel comparisons use the numbers: two account names with one UID are one identity for access checks. UID 0 is root and bypasses most ordinary permission checks.

Windows uses security identifiers (SIDs) for users and groups. A process's access token records its user SID, group SIDs and privileges. On both platforms, the running program's credentials supply the identity for later decisions.

A typical Linux password login follows this sequence:

1. A privileged login service, such as login, sshd or a display manager, receives the credentials.
2. PAM, Pluggable Authentication Modules, checks them through configured authentication modules.
3. The service installs the user's supplementary groups and GID, then changes the UID to give up root.
4. It starts the shell. Children inherit these credentials through fork() and normally retain them across execve().

The ordering matters. Giving up root before setting groups can leave the process unable to replace its inherited privileged group list. Also, editing an account's group membership does not rewrite credentials in already running processes; a new login session gets the new list.

Understand a controlled privilege change

A process has a real UID, normally identifying who started it; an effective UID, used for most access checks; and a saved set-user-ID, supporting controlled changes back after a temporary reduction. Ordinary programs have all three equal.

The root-owned /usr/bin/passwd illustrates setuid. Its mode 4755, shown -rwsr-xr-x, makes the effective UID root when an ordinary user executes it. If alice has UID 1000, her real UID remains 1000 while the effective UID becomes 0. The program can update /etc/shadow and uses the caller's real identity to restrict whose password may change. A setgid executable similarly changes the effective group.

That small trusted program holds dangerous authority. A bug can expose root rights, so systems minimize setuid programs and drop privileges early. For a service that only needs a low-numbered port, CAP_NET_BIND_SERVICE grants a narrower Linux privilege than full root.

Administer without sharing one identity

sudo authenticates the invoking user, checks which commands that user may run as which target identity, and logs the use. Its policy is commonly in /etc/sudoers. Individual identities preserve accountability that sharing a root password loses. On Windows, UAC normally starts an administrator's programs with a filtered token and requests consent for elevation.

Everyday applications should run with ordinary rights, and services should have dedicated accounts such as www-data. An exploited document viewer then inherits those limited credentials instead of an administrator's full authority.

A password verifier needs to recognize a correct password without storing a recoverable copy. It stores a password hash with the information needed to repeat the check. Login recomputes the hash from the entered password and compares results; it never decrypts the stored hash.

Protect the stored verifier

On Linux, /etc/shadow holds protected password hashes. The widely readable /etc/passwd contains account information and an x in place of the password. A shadow password field records an algorithm identifier, cost parameters, a salt and the resulting hash. $y$ identifies yescrypt, used by default in Debian 11, Ubuntu 22.04, Fedora 35 and later; $6$ identifies SHA-512-crypt.

A stolen hash file enables an offline attack. The attacker hashes guesses on separate hardware without the login service's delays. A brute-force attack enumerates strings; a dictionary attack begins with likely words, leaked passwords and variations such as "Password1!".

For exactly eight lowercase letters, ${26}^8=\mathrm{208,827,064,576}$ strings are possible. At ${10}^{10}$ guesses per second, exhausting the space takes about 21 seconds. At ${10}^4$ guesses per second, it takes about $2.09\times {10}^7$ seconds, roughly 242 days. These are full-search times, not guarantees for an individual password. Twelve lowercase letters give about $9.5\times {10}^{16}$ strings, roughly 110 days even at the faster rate.

Make each account expensive to attack

A salt is a fresh random value stored beside each password hash. It makes equal passwords produce different hashes and defeats reusable precomputed tables. The salt is public; its job is to force separate work for different accounts.

Consider 1,000 accounts and a list of ${10}^6$ guesses, checked at ${10}^4$ hashes per second. Without salts, computing the list once takes 100 seconds and serves all accounts. Unique salts require ${10}^9$ computations, about ${10}^5$ seconds or 27.8 hours. A common password may still be found near the start of the list.

Password-hashing schemes such as bcrypt, scrypt, yescrypt, Argon2 and PBKDF2 have adjustable work costs. Scrypt, yescrypt and Argon2 also impose memory costs. A practical login can tolerate a fraction of a second; a large guessing campaign repeats that expense millions of times. A single fast SHA-256 computation lacks this deliberate cost.

Design the login policy around guessing

NIST SP 800-63B-4 (2025) requires salts of at least 32 bits and a password-hashing cost as high as practical. Its password guidance requires at least 15 characters for password-only authentication, or at least eight when the password is part of MFA, and recommends accepting at least 64 characters.

It rejects mandatory mixtures such as one digit and one symbol and routine forced password changes. It requires blocking common or compromised choices, limiting online guessing, and forcing a change when compromise is evident. Online rate limits protect the live service; slow salted hashing protects stolen verifiers. A longer password selected predictably from a common list can still be weak.

Passwords prove knowledge. Other authenticators prove possession of an object or a physical trait. The operating system eventually attaches the authenticated identity to processes, but the route to that identity affects how easily an attacker can impersonate the user.

Combine different factor types

The three factor types are something you know, such as a password or PIN; something you have, such as a phone, smart card or security key; and something you are, such as a fingerprint. Multi-factor authentication (MFA) combines different types. A password plus a security question repeats the knowledge factor.

Time-based one-time passwords (TOTP) use a shared secret and a time step, usually 30 seconds, to produce matching codes on an authenticator and server. RFC 6238 defines the method. A code expires quickly, but a fake login page can relay a fresh code before it expires. App codes avoid SIM swapping; SMS codes depend on control of the phone number and can be redirected if it is taken over.

Prove possession without sending the key

A public-key authenticator signs a fresh challenge with a private key. The server verifies it with a stored public key, which cannot itself be reused to sign a login. FIDO2 security keys and passkeys also bind authentication to the site's identity, so a response for a fake site cannot authenticate at the real one. An ordinary SSH key proves possession too, but should not be assumed to have the same website-binding mechanism.

Storage changes the risk. Device-bound keys in a hardware security key, TPM or secure enclave sign inside hardware without exposing the private key. Synced passkeys copy encrypted key material between devices through a platform account, making recovery easier while bringing that account into the trust boundary. An ordinary SSH private key is a file protected by permissions and, optionally, a passphrase.

Interpret biometric errors

A biometric system compares measured features with a stored template. It accepts when a similarity score meets a threshold; it does not require an identical image each time.

Two accepted impostors out of 10,000 give FAR 0.02%. Thirty rejections out of 1,000 genuine attempts give FRR 3%. Raising the similarity threshold lowers FAR while raising FRR. A more demanding match reduces successful impersonation but makes legitimate retries more common.

A leaked biometric trait cannot be changed like a password, and sensors need defenses against photos or copied fingerprints. A useful design uses biometrics locally to unlock a protected key, sending only a signature to the server.

Authenticate processes too

For a local Unix-domain connection, Linux SO_PEERCRED obtains peer credentials from the kernel instead of trusting a name the client sends. Across machines, services use keys, certificates or tokens stored under restricted service-account access. After login, ordinary OS access checks use the UID or access token; they do not ask whether the user typed a password or touched a key.

An access decision has three parts: a subject asks to perform an operation on an object. The subject is usually a process acting for a principal; objects include files, devices and other processes. An access matrix represents the rights in cells, with subjects as rows and objects as columns.

Store the same rights in two ways

Suppose alice can read and write report.txt, bob can only read it, and the backup service can read both report.txt and payroll.db. Bob can also read and write payroll.db. Alice and bob may use the printer; the backup service may not. Most cells in a real system's much larger matrix are empty.

An access control list (ACL) stores a column with its object. The list on report.txt names alice, bob and backup with their respective rights. A capability list stores a row with the subject. Bob holds tokens for reading the report, reading and writing payroll, and using the printer.

A capability names both an object and allowed operations and cannot be forged or enlarged by its holder. Authorized copying and delegation are possible. The kernel may hold the real token while user code holds an index; another design protects a token cryptographically.

Follow a descriptor after open

Unix combines both approaches. open() checks the caller against the file's permissions, then returns a file descriptor recording granted access. Later reads and writes check that open handle. A child can inherit it through fork(), or a process can receive it over a Unix-domain socket.

Alice opens app.log for writing. Its owner changes the mode to 444. Alice's already open descriptor can still write; a new attempt to open the file for writing encounters the new mode. Updating the ACL governing future opens does not cancel every capability already issued. Windows handles follow the same general pattern.

FreeBSD Capsicum makes this useful for confinement. After cap_enter(), a process cannot open an arbitrary absolute path. It can operate through existing descriptors, receive descriptors and use openat() beneath a directory descriptor it holds. A parent can therefore give a worker exactly the files it needs. Capability-based kernels such as seL4 and Fuchsia extend the idea to kernel objects generally.

Distinguish Linux's other meaning of capability

Linux capabilities divide root's privileges into categories. CAP_NET_BIND_SERVICE permits binding low-numbered ports; CAP_DAC_OVERRIDE bypasses ordinary file access checks. These privileges do not name one particular object, so they are different from an access-matrix capability. A single Linux privilege can authorize a class of operations across many objects.

Unix permissions attach rights to the owner, owning group and everyone else. To decide access, first identify the operation and the relevant object. Reading file contents, traversing a directory and deleting a name require different permissions.

Decode and select the bits

Read is 4, write 2 and execute 1. Add these values for each class: 754 means rwxr-xr--; 640 means rw-r-----. chmod 640 notes.txt sets a mode, while chmod g+w,o-r notes.txt edits selected bits. The owner or root may change the mode; changing ownership needs root authority, represented by CAP_CHOWN on Linux.

The kernel selects the first matching class: owner if the effective UID matches, otherwise group if a process group matches, otherwise others. It does not combine classes or retry a more generous one. Alice owns a mode-077 file: its owner bits are empty, so she cannot read it even though group and others have full rights. She can still change the mode as its owner.

Root normally bypasses read and write checks. For executing a regular file under ordinary Linux discretionary checks, at least one execute bit must be set. Mandatory policy can impose further restrictions.

Interpret directory and special bits

Directory read lists names; execute permits traversal and lookup; write together with execute permits creating, deleting and renaming entries. Removing a name changes its directory, so a mode-000 file can be deleted by someone with the needed directory rights.

Setuid adds octal 4000 to an executable's mode and changes the effective user identity on execution. Setgid adds 2000 and similarly changes the group; on a directory it makes new files inherit the directory's group. The sticky bit adds 1000. A directory such as /tmp, mode 1777, allows deletion or renaming only by the file owner, directory owner or a privileged process, with ordinary directory permissions still required.

Remove bits when creating files

A umask clears bits from the mode a program requests. It is a bit operation, not octal subtraction. With the usual requests of 666 for files and 777 for directories:

A umask cannot add execute rights that were absent from the file request.

Read richer access lists

POSIX ACLs add named users and groups. setfacl -m u:bob:r report.txt gives bob a named read entry; getfacl displays it and ls -l marks the extended ACL with +. The ACL mask limits named users, named groups and the owning group, so a named entry may show more rights than are effective.

Windows NTFS uses a security descriptor with an owner SID and a DACL of ordered allow or deny ACEs. Evaluation stops when a requested right is denied or all requested rights have been granted; rights left ungranted are denied. Preferred ordering places explicit entries before inherited ones, with deny before allow within each group. An explicit allow can therefore resolve a request before an inherited deny is reached. A separate SACL selects access events for auditing.

Different access-control models answer different management questions. Who may grant access? Can a compromised program share a user's data? How can an administrator change the rights of many employees without editing each file? One operating system can apply several models to the same request.

Distinguish owner choice from system policy

In discretionary access control (DAC), an object's owner decides who may use it. Unix modes and ACLs are familiar mechanisms. The weakness is that a malicious program acting as the user can exercise the user's rights, including copying readable data elsewhere.

In mandatory access control (MAC), an administrator's system-wide policy limits access independently of the owner's wishes. The kernel applies MAC in addition to DAC. Both checks must allow access, so a permissive file mode does not override a mandatory denial.

The Bell–LaPadula model protects confidentiality with ordered levels and two rules: no read up and no write down. A Secret process can read Confidential data and write to a Top Secret destination. It cannot read Top Secret data or write Confidential data. A program cannot move information from Secret to Confidential simply by reading and rewriting it. Biba reverses the read and write directions to protect integrity.

Confine a service even when it is root

SELinux labels files with types and runs processes in domains. Policy names which domain may perform which operations on each type; unlisted access is denied. A web server in httpd_t may read pages labeled httpd_sys_content_t while being blocked from user_home_t, even if ordinary Unix modes permit the read. ls -Z shows labels.

In enforcing mode, SELinux blocks and logs violations. In permissive mode, it logs without blocking. Root in a confined domain must still satisfy enforcing policy. UID 0 and unrestricted access are therefore different claims.

AppArmor confines programs with profiles based on file paths and allowed capabilities. It is commonly used on Ubuntu and SUSE. SELinux and AppArmor operate through Linux Security Module (LSM) hooks at security decisions. Android combines app-specific UIDs with SELinux; Apple systems also use application sandbox profiles.

Assign jobs instead of individual rights

In role-based access control (RBAC), permissions attach to roles and users receive role memberships. Moving a person between jobs changes memberships instead of many independent object permissions.

Suppose 50 employees each need 20 permissions. Direct assignment needs 1,000 user-permission links. If five roles each hold 20 permissions and every employee has one role, management needs 50 user-role links and 100 role-permission links: 150 total. The saving comes from shared job patterns, not from deleting required permissions.

Windows groups used as roles and group-based sudo policies implement this pattern. DAC suits owner-managed files, MAC contains services under centrally enforced rules, and RBAC organizes administrative assignments. A service may receive its job rights through a role while DAC and MAC both check its actual file access.

A virtual machine (VM) gives a complete operating system its own apparent computer. That operating system is the guest. The physical computer is the host, and the hypervisor, or virtual machine monitor (VMM), allocates real processor time, memory and devices to the guests.

Preserve behavior while retaining control

Popek and Goldberg's 1974 framework identifies three properties. Equivalence means programs behave as on a real machine, apart from timing and available resources. Resource control means the hypervisor remains in charge of the hardware. Efficiency means most guest instructions run directly on the processor.

The hypervisor schedules virtual CPUs (vCPUs) onto physical cores. A machine switch saves one VM's full execution state, including privileged registers and interrupt settings, then restores another's. A guest sees its own state even when it shares a physical core with other guests.

KVM turns the Linux kernel itself into a hypervisor and is usually classified as Type 1. A complete Linux host remains available, which makes the simple two-type distinction less clear. To reason about protection, identify which component really controls CPU and memory virtualization.

Use the machine abstraction

Server consolidation puts lightly used systems together. Twelve servers each averaging 5% of an eight-core computer need 60% of one such computer in aggregate. That average suggests consolidation is possible, but simultaneous peaks, RAM and device demand still require headroom.

A snapshot saves VM state for later restoration. Copying a disk image alone does not capture a running CPU and its memory. Live migration transfers a running VM to another host, copying memory and switching execution with a short pause. These operations are possible because the guest's machine state is represented as data the hypervisor manages.

VMs also support testing old systems, separating conflicting software and running different operating systems on compatible hardware. A VM can run a Windows guest alongside a Linux guest, each with its own kernel. Cloud server instances commonly use this boundary.

Separate virtualization from full emulation

An emulator can interpret or translate an instruction set for a different processor, such as ARM code on x86. Hardware virtualization normally runs guest instructions directly on a compatible processor and intervenes for sensitive events. This explains both its speed and its need for CPU support.

Separate guest kernels form a useful boundary: taking over one guest kernel does not directly give control of the host kernel. The boundary still depends on correct hypervisor and device-model code. A VM escape exploits that layer, while a side channel may leak data through shared hardware without granting ordinary memory access. Choosing a VM reduces the interfaces exposed to an untrusted guest; it does not make shared hardware infallible.

A guest kernel expects to control its machine. The hypervisor must preserve that appearance without handing over the real computer. CPU virtualization lets ordinary guest instructions run directly and intercepts operations that need a change to virtual machine state.

Trap and emulate virtual state

In classic trap-and-emulate, the guest kernel is deprivileged. Its attempt to load a page-table register or install a trap table traps to the hypervisor. The hypervisor updates the guest's virtual register or table and resumes execution. It does not let the guest replace the real hypervisor's control structures.

A guest system call can require several crossings in this design. The guest application traps into the real handler owned by the hypervisor. The hypervisor transfers execution to the guest kernel's recorded handler at reduced privilege. When the guest kernel returns, that privileged return traps again; the hypervisor completes the real return to the guest application.

Popek and Goldberg's condition for this technique is that every sensitive instruction must be privileged. Sensitive means it changes machine configuration or behaves differently with privilege. Privileged means it traps outside the most privileged mode. Silent differences cannot be emulated if the hypervisor never learns that they occurred.

Understand the classic x86 obstacle

Robin and Irvine's 2000 analysis identified 17 sensitive unprivileged x86 instructions. For example, POPF can restore the interrupt-enable flag with sufficient privilege but silently leave it unchanged at reduced privilege. A deprivileged guest may believe it disabled interrupts even though no trap told the hypervisor to update its virtual interrupt state.

Three approaches solve this problem:

VMware used binary translation in 1999; Xen used paravirtualization in 2003. Intel VT-x arrived in 2005 and AMD-V in 2006. Ordinary guest applications can still run directly when kernel code needs translation or hypercalls.

Keep ring number separate from virtualization mode

With Intel VT-x, the hypervisor runs in VMX root operation and a guest in non-root operation. The guest kernel can use ring 0 there. Ring 0 in a guest is not control over the physical host.

Configured events cause a VM exit, saving vCPU state in a control structure such as the VMCS. The hypervisor handles the event and resumes through a VM entry. Many system calls can stay entirely inside the guest. ARM uses EL2 for the hypervisor; RISC-V provides a hypervisor extension.

If a guest executes HLT and that event is configured to exit, the hypervisor marks its vCPU waiting and schedules other work until a virtual interrupt arrives. The physical core need not sit idle. Exits and entries still cost time, typically hundreds to thousands of cycles, so batching and paravirtual interfaces remain valuable.

A guest application uses a guest virtual address. Its kernel translates that to a guest physical address, which is the guest's view of RAM. The hypervisor then maps that address to a host physical, or machine, address. Each guest can believe its RAM starts at zero while using different real frames.

Preserve the page offset

Use 4 KiB pages. The guest maps virtual page 2 to guest frame 7; the hypervisor maps guest frame 7 to machine frame 42. For address 0x2ABC:

1. Split it into page 2 and offset 0xABC.
2. Replace page 2 with guest frame 7, giving 0x7ABC.
3. Replace guest frame 7 with machine frame 42, hexadecimal 0x2A.
4. Keep the offset, giving machine address 0x2AABC, or 174,780 bytes from address zero.

Only the page number changes. Treating the guest physical address as a real RAM address would bypass the hypervisor's allocation and isolation decision.

Compare software and hardware translation

Shadow page tables map guest virtual pages directly to machine frames. The real MMU uses these tables, built by the hypervisor. The hypervisor write-protects guest page-table pages and intercepts guest table changes and switches so its shadows stay consistent. Frequent guest address-space changes therefore cause extra traps.

Nested paging, Intel EPT or AMD NPT, gives the MMU both stages. Hardware walks the guest tables and the guest-physical-to-machine tables. Guest page-table updates no longer all need software interception. The TLB caches the combined final translation.

A cache miss can still be expensive. With four levels at each stage, each of four guest-table entries needs four nested-table references plus the entry itself: 20 references. Translating the final guest physical data address adds four more, making 24 translation references before the data access. Large pages and page-walk caches reduce this worst-case uncached work.

Reclaim memory with limited guest knowledge

Memory overcommitment promises more total guest memory than the host has. Four 16 GiB VMs total 64 GiB on a 48 GiB host. This works only while their active needs fit; if they all become busy, reclamation or swapping becomes necessary.

A balloon driver inside a guest allocates pages at the hypervisor's request. The guest OS chooses what to free, while the hypervisor reclaims the machine frames behind balloon pages. This reduces the information gap: the guest knows which data matters, while the hypervisor sees allocations without all their meaning.

Page sharing, such as Linux KSM, merges identical pages into one read-only copy and splits it on a write. Hypervisor swapping provides another fallback, but can choose poorly because it lacks the guest's knowledge. A device passed to the VM also needs an IOMMU to translate and restrict its DMA addresses.

Sharing identical pages across customers has a privacy cost: copy-on-write timing can reveal whether another guest holds the same data. Cloud providers commonly disable that form of cross-customer sharing even though it saves memory.

A VM needs storage, networking and other devices as well as CPU time and RAM. The device interface affects compatibility, performance and the code an untrusted guest can reach. A fast interface and a small trusted interface are related design choices, but they are not the same property.

Choose how the guest reaches a device

A helper such as QEMU can emulate an Intel e1000 network card. The ordinary guest driver writes familiar registers; the helper implements the effect. Frequent register accesses can cause many exits and software operations.

A virtio driver instead exchanges requests through shared-memory queues. It can enqueue several operations and notify the backend once. VMware's vmxnet3 and Hyper-V's VMBus devices follow related virtualization-aware designs. A paravirtual device driver does not mean the entire guest kernel must use paravirtual CPU operations.

In device passthrough, the guest owns a physical device or an SR-IOV virtual function. The IOMMU confines DMA to the VM's memory. Passthrough reduces software mediation, but transferring device state to another host can make live migration difficult.

Treat virtual disks as managed state

A virtual disk is commonly a host file, such as qcow2, VMDK or VHDX, or a storage volume. Copy-on-write images let many guests share a base while recording separate changes. Snapshots can preserve disk state cheaply, and a complete running snapshot also needs the VM's execution and memory state. Cloning and migration rely on the hypervisor's ability to manage these representations.

Locate the boundary an attacker must cross

A compromised guest kernel already controls that VM, but reaching the host requires a VM escape or another boundary failure. The 2015 VENOM flaw affected QEMU's emulated floppy-disk controller and could let guest code execute in the host's QEMU process. Keeping device emulation in a separate sandboxed process limits the authority exposed through such a bug.

Shared CPU caches, branch predictors and memory buses also create side channels. Spectre and L1TF showed that isolation cannot be judged from page mappings alone. Mitigations include processor and microcode updates, clearing relevant state between guests and avoiding simultaneous placement of different customers on sibling hardware threads of one core.

Confidential computing, including AMD SEV-SNP and Intel TDX, protects guest memory from the hypervisor with hardware-held encryption keys and supports attestation of the launched environment. It extends the trust boundary against a compromised host or operator. It does not remove every side channel or every way a host can interrupt service. VM security still depends on the hypervisor, device interfaces, hardware and their configuration.

A Linux container is a set of processes using the host kernel with restricted views and privileges. It has no separate kernel or emulated computer. Namespaces give processes private instances of resources that would otherwise be global.

Match each namespace to its view

clone() with namespace flags creates a child in selected new namespaces. unshare() separates namespace context, and setns() joins an existing namespace through a descriptor such as one opened from /proc/1234/ns/net. PID reassociation affects later children without moving the caller. For time namespaces, unshare(CLONE_NEWTIME) also prepares a namespace for later children; setns() can move the caller into an existing time namespace. Time namespaces are created with unshare().

Thus unshare --pid --fork --mount-proc bash uses --fork to create the shell inside the new PID namespace. That shell is PID 1 there even if its host PID is 48213. PID 1 adopts orphaned processes; if it exits, the kernel kills the namespace's remaining processes. Processes in an ancestor namespace can see descendants, while processes inside cannot see outward into ancestor PID namespaces.

Separate files and networking

A mount namespace gives a private mount list. A runtime assembles a file tree and makes it the root using pivot_root. A container can use another Linux distribution's /etc and /usr while still calling the host kernel. chroot alone merely changes the starting point for path lookup; it is not a safe boundary against a privileged process.

A new network namespace has a loopback interface. A runtime can connect it through a veth pair, with one endpoint inside and another connected to a host bridge. Separate stacks let two containers both listen on port 80 without competing for one namespace's port.

Translate identities across a user namespace

A line in /proc/<pid>/uid_map lists the first inside ID, first outside ID and range length. The line 0 100000 65536 maps inside IDs 0–65535 onto host IDs 100000–165535. Inside UID 1000 therefore maps to 101000 outside. IDs without a usable mapping are normally displayed as the overflow UID 65534.

Root inside that namespace has capabilities over resources governed by it, not host-wide authority. Loading a kernel module or changing the real system clock still needs privilege in the initial user namespace. A UID mapping does not protect a privileged runtime socket handed into the container.

A private view also does not limit consumption. A process with its own PID and mount namespaces can still consume CPU and RAM. Cgroups limit amounts, while capability limits and seccomp restrict operations.

A control group, or cgroup, groups processes for resource accounting and control. Namespaces determine their view; cgroups govern quantities such as CPU time, memory and task count. Children inherit their parent's group, and the hierarchy prevents a child escaping an ancestor's limits.

Read the cgroup v2 files

Cgroup v2 uses one unified hierarchy, normally at /sys/fs/cgroup. A parent enables controllers for its children through cgroup.subtree_control, for example +cpu +memory +pids. Individual files in each child hold limits and counters.

Controllers include cpu, cpuset, memory, io, pids, hugetlb and rdma. They govern CPU time, placement on CPUs and memory nodes, ordinary memory, block I/O, task count, huge pages and RDMA resources. Cgroup-aware performance events support measurement, and cgroup.freeze pauses a group's execution.

Distinguish a CPU cap from a share

cpu.max contains quota and period in microseconds. 50000 100000 allows 50 ms of CPU execution per 100 ms, equivalent to half of one CPU. 200000 100000 allows two CPUs' worth of time, potentially spread across cores. max 100000 means no quota cap. Once the quota is consumed, the group waits for the next period even if a CPU is idle.

cpu.weight sets a relative share during competition, from 1 to 10,000 with default 100. For equally placed, runnable groups on four CPUs, weights 100 and 300 divide available time 1:3: approximately one CPU and three CPUs. If the heavier group goes idle, the other can use all four, unless another limit intervenes. A weight is not a reservation or hard ceiling.

Contain memory and task exhaustion

memory.high applies reclaim pressure and slows tasks above its threshold. memory.max imposes a hard boundary. The kernel first tries to reclaim memory, including cache and available swap. If it cannot recover enough, cgroup OOM handling kills an eligible process inside the affected group.

For memory.max of 512M, the limit is 536,870,912 bytes, or 512 MiB. A leaking process can force reclaim and eventually an OOM kill, but the victim is selected by the kernel and is not guaranteed to be that process. The group boundary contains the event instead of choosing an unrelated host process.

pids.max counts processes and threads. At a limit of 100, further creation fails with EAGAIN, containing a fork bomb's task growth. An I/O limit such as 8:0 rbps=10485760 in io.max caps reads from block device 8:0 at 10 MiB/s.

These files do not hide host processes or file names. Also, /proc/meminfo and tools reading it may report host RAM instead of memory.max; the displayed total alone does not prove a cgroup has no limit. Resource measurements must be read at the layer that enforces the boundary.

A useful container boundary combines views, resource controls and restricted authority. A private file tree alone does not stop dangerous system calls, and a CPU quota does not stop access to an exposed host file. The host kernel enforces every one of these controls.

Reduce the process's authority

A runtime combines namespaces, cgroups and a file-system image containing programs and libraries. It can strengthen the boundary by:

• Running the workload as non-root or mapping container IDs through a user namespace.
• Dropping Linux capabilities, especially broad powers such as CAP_SYS_ADMIN and CAP_SYS_MODULE.
• Applying seccomp to restrict unnecessary calls.
• Applying an SELinux or AppArmor policy.
• Making the root file system read-only where practical.
• Setting no_new_privs so setuid execution cannot raise privilege.

Each control covers a different route. A read-only root file system still allows permitted network operations; a seccomp filter still allows the calls on its list. The design should match the operations the application actually needs.

Identify shared-kernel risk

All ordinary containers share the host kernel. A reachable kernel bug can therefore compromise the host and other containers. In a VM, exploiting the guest kernel gives control of that guest; reaching the host requires a further hypervisor or device boundary failure.

Root inside a container is especially sensitive. Without a user namespace, it is UID 0 to the host kernel, constrained by capabilities, filters and mandatory policy. Privileged mode or access to the host root file system can remove those constraints. A root-owned runtime socket is also dangerous: its client can ask the runtime to perform privileged actions. Mapping container root to an unprivileged host ID does not make that socket safe or remove kernel vulnerabilities.

Choose a boundary for the workload

A Linux container needs a Linux kernel. Platforms on Windows and macOS therefore often provide a Linux VM beneath Linux containers. VMs boot an OS and allocate memory for each kernel; containers start ordinary processes and can pack workloads more densely.

Recognize intermediate designs

gVisor handles many container system calls in a user-space kernel, reducing direct contact with the host kernel. Kata Containers gives each sandbox, commonly a Kubernetes pod, a lightweight VM; containers in that sandbox share its guest kernel. Firecracker supplies microVMs with a minimal device model for platforms running functions or container workloads, including AWS Lambda.

Cloud systems often place containers inside VMs: a VM separates customers while containers organize one customer's applications. The guest kernels and the host kernel remain distinct, so a container escape inside a VM does not by itself prove a hypervisor escape.