I/O Devices and Storage: Disks, SSDs and RAID

An I/O device moves data into or out of a computer or keeps it persistently. A keyboard supplies input; a network card sends and receives packets; a disk keeps data after power is removed. Devices connect through buses, or interconnects. PCI Express (PCIe) connects high-performance devices such as NVMe SSDs. SATA and USB serve other storage and peripheral devices farther out in the hierarchy. Fast interconnects cost more to build and have tighter physical limits, so a computer combines several kinds.

The interface is the software's view

A controller can contain processors, memory, and firmware, the software running inside the device. The operating system controls it through an exposed hardware interface. Two controllers can have very different internals while supporting the same commands.

A simple device has three registers:

For a simple output operation, the kernel first checks status until the device is ready. It then writes the data, writes the command, and checks status until completion. Finally, it checks for an error. Writing the command before supplying its data could start an operation with the wrong contents. Repeatedly checking status is polling; having the CPU move the payload is programmed I/O (PIO).

Reach registers safely

Port-mapped I/O gives registers a separate address space. On x86, instructions such as in and out access those ports. The kernel controls which code may execute them. Memory-mapped I/O (MMIO) places registers in the physical address space, so loads and stores can reach them through mappings controlled by the kernel's page tables. PCIe devices normally use MMIO.

An MMIO address denotes a device operation, even though its access resembles a memory access. A read can clear a flag. A write can start a transfer. Status can change because the device finishes work, with no CPU store to that address. Ordinary cache behavior could hide those changes or combine accesses that the device needs separately.

Linux drivers therefore use accessors such as readl() and writel(), suitable mapping attributes, and required memory barriers. A volatile pointer can stop the compiler from removing or merging accesses, but it does not provide all ordering against ordinary memory or CPU reordering. For example, a command descriptor in RAM must be visible to the device before a doorbell register tells it to fetch that descriptor.

Keep device control in the kernel

An application normally calls read() or write(). The kernel checks permissions and performs the device protocol on its behalf. This prevents one program from issuing commands that interfere with another program's request. A user buffer is the application's data area; a command register is the controller's control point. They play different roles even when both have addresses.

A device can take much longer to finish a request than the CPU takes to issue it. The waiting strategy decides whether the CPU spends that interval checking the device or running other work. It does not change how quickly the device itself completes the request.

Poll or sleep

With polling, the CPU repeatedly reads status. A completion can be noticed quickly, but checks made while the device is still busy consume CPU time. With interrupts, a process waiting for a read becomes blocked and another ready process can run. When the device finishes, it signals the CPU. The kernel's interrupt handler, also called an interrupt service routine, checks completion, handles the result, and makes the waiting process ready. The scheduler decides when that process runs again.

Suppose process A starts a disk read that takes 10 ms. Polling occupies the CPU for that wait. With interrupt-driven I/O, process B can compute during the same 10 ms. A's device operation still lasts 10 ms; useful computation overlaps the wait.

Interrupts also take work: the CPU saves state, runs the handler, restores state, and may switch processes and refill caches. The cost is commonly a few microseconds. If a device responds in 2 µs, a brief polling loop can be cheaper than blocking and waking the process. Interrupts suit slow operations; short polling suits very fast ones.

Adapt to completion speed and load

A hybrid, or two-phase, strategy polls briefly, then sleeps for an interrupt if the request remains unfinished. Fast completions avoid a wake-up cost, while slow completions do not occupy the CPU for their entire duration. Linux supports polled completion for suitable fast storage requests.

Heavy interrupt traffic creates a different problem. If almost every CPU cycle handles newly arriving events, the code that consumes their data gets little time to run. This is livelock: the machine is busy but useful progress collapses. Linux's NAPI network mechanism limits interrupt work and switches to batch polling under load.

Interrupt coalescing groups several completions into one notification. At 100,000 completions per second and 5 µs of CPU work per interrupt, separate notifications cost:

That is half of one CPU core's time each second. Combining eight completions per interrupt gives 12,500 interrupts and 62.5 ms of CPU work per second, or 6.25% of a core. A request may wait briefly for its batch, so lower overhead comes with added notification latency.

Send notifications to the right core

PCIe devices commonly use MSI or MSI-X, interrupts delivered as messages to a special address. Multiple interrupt vectors let a device direct different queues' completions to different cores. An NVMe controller can pair per-core queues with suitable interrupt vectors, keeping submission and completion work close together. More vectors organize CPU work; they do not shorten a flash read or a disk rotation.

Waiting and copying are separate jobs. Interrupts can stop the CPU from spinning while a device works, but the CPU would still be busy if it had to move every word through a data register. Direct memory access (DMA) assigns that transfer to a hardware engine, usually inside the device controller.

Measure the work PIO leaves on the CPU

Imagine a 4-byte data register that takes 100 ns per access. Moving a 4 KiB block requires:

Those register operations occupy the CPU. Larger buffers require proportionally more accesses. DMA removes that word-by-word register copying; the CPU still prepares the request and handles its completion.

Follow a DMA read

1. The driver prepares a destination buffer and makes it available for device access.
2. It supplies the DMA address or addresses, byte count, direction, and read command.
3. The device reads the requested storage blocks and writes the bytes into memory.
4. The CPU runs other work while the device transfers data.
5. Completion is reported, commonly by an interrupt. The handler checks status and wakes the waiting process.

The DMA engine acts as a bus master, initiating memory transfers. Setup costs depend on the mapping, the platform, and the number of memory regions. DMA also consumes memory bandwidth shared with the CPU. It frees CPU instructions, not all resources.

Describe scattered memory

A buffer can be contiguous in a process's virtual address space while occupying separate physical pages. A scatter-gather list describes those regions as address-and-length pairs. A device can transfer one request across several regions without the CPU first copying them into one physically contiguous buffer.

For example, a three-page buffer can be backed by three nonadjacent pages. The driver creates descriptors for the device-visible regions and keeps the backing pages stable until completion. Pinning prevents those pages from being removed or reused while hardware may still access them. Unpinning early risks a later DMA write overwriting somebody else's data.

Use the device's address space

A device does not normally use an application's virtual pointer. The kernel's DMA API supplies bus addresses, which can differ from CPU physical addresses. An IOMMU translates device addresses to physical memory and restricts the memory a device may access. This isolation also helps a hypervisor assign a device to a virtual machine.

CPU caches add another coordination step. Before an outgoing DMA transfer, the device must see the CPU's latest bytes; after an incoming transfer, the CPU must see the device's updates. Some hardware maintains coherence automatically. Other platforms require cache synchronization through the DMA API. Address mapping, page lifetime, and cache visibility each solve a different part of making the transfer safe.

A device driver translates a general request into a controller's commands, register accesses, and completion handling. This lets a file system ask for a block without knowing whether the hardware uses SATA, NVMe, or USB. The driver knows the controller's specific protocol and quirks.

Give each layer a job

A Linux storage read usually passes through this chain:

Application → system call → file system → generic block layer → driver → controller → storage device.

The file system maps a file position to storage blocks. The block layer queues requests, merges adjacent ones, and can schedule their order. The driver submits the resulting operations. Raw block-device access can bypass the file system while still using the lower layers.

The standard interface makes new hardware usable without rewriting every application. It also hides some special features. Interfaces such as ioctl() or command pass-through let software request controls that do not fit an ordinary block read or write.

Separate device classes

Linux exposes block and character devices as special files, such as /dev/sda and /dev/nvme0n1. Drivers commonly run in kernel mode, so a driver bug can crash the kernel or corrupt memory. Driver isolation limits that reach in systems that support it. Linux's driver collection is a large share of its source tree; studies have also found driver code particularly error-prone.

Queue work across cores

Linux's blk-mq block layer uses per-CPU software queues and hardware dispatch queues. An I/O scheduler such as mq-deadline, bfq, or kyber can reorder work or manage fairness; none avoids an extra scheduling policy. Hard disks benefit from reducing seeks. Fast NVMe devices often use none, since they have no head to reposition. The selected scheduler appears in square brackets in /sys/block/<device>/queue/scheduler.

SATA with AHCI/NCQ supports a command queue of up to 32 outstanding commands per device. NVMe supports many submission and completion queues in memory: up to 65,535 I/O queues, with up to about 64,000 commands per queue. Actual controllers offer smaller limits. Per-core queue pairs reduce contention on shared submission paths.

Follow one NVMe read

A process calls read(fd, buf, 4096). For a cache miss, the file system finds the disk block and the block layer builds a request. The NVMe driver writes a descriptor into a submission queue, then writes its MMIO doorbell register. The controller fetches the command, transfers the data into memory by DMA, and posts a completion entry. An MSI-X interrupt leads the kernel to complete the request. In a buffered read, data reaches the application's buffer through the page cache, and read() returns 4096. A cache hit can satisfy the read before device I/O is needed.

A hard disk exposes an array of sectors numbered from zero. These numbers are logical block addresses (LBAs). A sector was traditionally 512 bytes; many drives now use 4,096-byte physical sectors. A 512e drive accepts 512-byte logical sectors over that larger physical format. A file system's 4 KiB block can therefore span several logical sectors.

Picture the moving parts

Magnetic platters rotate on a spindle. Each surface has concentric tracks divided into sectors and a read/write head on a moving arm. The heads move together. Tracks at the same arm position across surfaces form a cylinder. Common spindle speeds include 5,400 and 7,200 RPM, with 10,000 and 15,000 RPM in faster disk families.

Nearby LBAs are usually placed nearby, helping sequential reads avoid repeated seeks. A sector remapped to a spare can break that relationship. Software therefore sees a useful linear address space without knowing exact physical placement.

Outer tracks have more room for sectors. Zoned recording puts more sectors on those tracks, so more bytes pass under the head per rotation. Sequential throughput can fall as a read moves toward the center. Track skew shifts sector positions between neighboring tracks: after the head changes tracks, the next sequential sector arrives at the right time instead of having just passed.

Separate completion from durability

A drive cache can retain recently read data and accept writes. These buffers range from a few megabytes to hundreds of megabytes; read caching can retain the rest of a track already passing under the head. With write-back caching, a write can complete while its data remains in volatile cache. Write-through waits for the medium. A cache flush requests that cached writes reach durable storage; Force Unit Access (FUA) requests durable completion for the affected write.

Power-failure atomicity is another property. An atomic write unit changes completely or remains unchanged when power fails. Sector size alone supplies no such promise: the device must document its guarantee. A write crossing multiple atomic units can leave a torn write, with some new units and some old ones. Flushes address durability; atomic units address partial updates.

Read capacity labels correctly

Drive capacities use decimal SI prefixes. Binary units use powers of two:

A 4 TB drive therefore contains:

Both labels describe the same byte count. A tool that displays about 3.64 while labeling it “TB” has used binary arithmetic with a decimal-looking label; the conversion has not removed space.

Recognize an extra write constraint

Shingled magnetic recording (SMR) overlaps adjacent tracks to pack in more data. Rewriting one track can disturb its neighbor, making small random updates expensive. Large sequential writes in suitable zones work better. This is a magnetic disk with a write-management constraint, even though ordinary reads still involve the same heads and rotating platters.

A disk request spends time reaching the data before transferring it. For a simple request with no queueing or cache hit, divide service time into three parts:

The seek moves and settles the arm on the track. Rotational delay waits for the sector to reach the head. Transfer time covers the bytes passing under it. Queueing, when present, adds waiting before service starts.

Convert RPM into a wait

One rotation takes 60,000 ms divided by RPM. At 7,200 RPM:

For a random sector position, the average wait is half a revolution, about 4.17 ms. At 15,000 RPM, a revolution takes 4 ms and the average wait is 2 ms. A higher RPM reduces rotational delay; it does not directly reduce the arm's seek time.

Data sheets give an average seek time. Uniformly distributed requests have an average seek distance of about one third of the full stroke, but seek time is not proportional to distance. Acceleration and settling dominate short seeks; long seeks include more travel at full speed.

Calculate a small random read

Use a 7,200 RPM drive with an 8 ms average seek and 200 MB/s transfer rate. A 4 KiB request contains 4,096 bytes. First calculate the transfer in seconds:

Then add positioning and transfer:

One outstanding request at a time gives about:

The effective rate is about 4,096 bytes multiplied by 82 per second, or 0.34 MB/s. A 200 MB/s transfer specification describes moving already-positioned data; it cannot remove the roughly 12 ms spent positioning for each small random request.

Calculate a long sequential read

Now read 100 MB with one initial seek and rotational wait. Transfer time is 100/200 s, or 500 ms. Total time is about 512.17 ms, giving:

The effective rate is roughly 580 times the small-random rate. One positioning cost is spread over many more bytes. Large sequential transfers are dominated by transfer time; small random requests are dominated by positioning.

Use bytes consistently when mixing KiB with MB/s, and convert milliseconds to seconds before dividing bytes by time. IOPS suits small random operations; MB/s suits sequential streams. Read-ahead, nearby placement, and combining small writes into sequential batches improve performance by changing how often positioning is paid.

A disk scheduler chooses which pending request to serve next. Position gives it useful information about service cost. Reducing arm travel helps, but a policy must also consider how long individual requests wait.

Compare the policies

SSTF can repeatedly choose new nearby requests while a distant one waits. An OS that lacks exact geometry often approximates it using LBA distance, called nearest block first. SCAN-style fairness assumes the system can keep serving work as it sweeps; an overloaded device still cannot provide a fixed response time for every arrival.

Add travel one leg at a time

Take tracks 0–199, a head at 50 moving upward, and arrivals 82, 170, 43, 140, 24, 16, 190. Stop counting once all requests finish.

FCFS follows the list. Its travel is 32 + 88 + 127 + 97 + 116 + 8 + 174 = 642 tracks. SSTF visits 43, 24, 16, 82, 140, 170, 190, totaling 208 tracks.

SCAN visits the higher requests, continues to 199, then reverses to 16. Its distance is 149 + 183 = 332. LOOK turns at 190, giving 140 + 174 = 314. C-LOOK goes from 50 to 190, wraps to 16, and finishes at 43: 140 + 174 + 27 = 341 tracks. The wrap is real movement even though no request is served during it. C-SCAN would travel to 199, return to 0 without serving, then serve upward through 43.

The best total in this finite queue does not establish the best policy for a stream of future arrivals. SSTF's low travel here comes with a starvation risk under continued load.

Include rotation and firmware

Shortest positioning time first (SPTF) minimizes seek plus rotational delay. A farther track can win if its sector arrives sooner. Accurate choices need geometry and rotational position, so drive firmware is better placed to make them than the OS.

The OS can merge requests for adjacent blocks, then submit several commands through NCQ. Firmware reorders that batch using internal information. Anticipatory scheduling may briefly leave the disk idle to wait for a likely nearby follow-up request. Deadline scheduling limits postponement; Linux's mq-deadline tracks expiry and favors reads. SSD scheduling focuses on fairness and latency because there is no seek or rotation to save.

NAND flash stores information as electric charge in cells. The controller distinguishes charge levels to read the bits. A cell holding more bits needs more distinguishable levels, which packs in capacity but makes programming and error control harder.

Count the charge levels

For a cell with b bits, the number of levels is $2^b$. More bits per cell generally lowers cost per gigabyte, while programming becomes slower and erase-cycle endurance falls. These are properties of the cell organization; the whole SSD's controller and parallelism also affect performance.

Match each operation to its unit

Flash chips contain erase blocks, each holding many pages. A page is commonly 4–16 KiB, while an erase block can contain hundreds of pages and span hundreds of KiB to several MiB. A flash page, a virtual memory page, and a file system block are distinct units even when some sizes happen to match.

A read returns a page, commonly in tens of microseconds. A program writes an erased page and takes hundreds of microseconds. An erase resets a whole block to ones and takes milliseconds. Pages are programmed in the chip's required order, commonly from lower to higher pages. The controller must follow those ordering rules while placing data.

A used page cannot simply receive a replacement version. Its erase block must first be erased, destroying the contents of every page in that block. This is the central mismatch: writes are page-sized, while erases are block-sized.

See the cost of an in-place update

Imagine one block containing pages A, B, C, and D. To replace A with A′ while keeping the same physical placement:

1. Read B, C, and D into temporary storage.
2. Erase the whole block, including the old A.
3. Program A′, B, C, and D into the erased pages.

One page of new information has caused three reads, four programs, and an erase. Erasing only A is unavailable. An SSD avoids paying this cost on each host overwrite by writing the replacement elsewhere and changing its mapping.

Manage damage and retention

Every program/erase (P/E) cycle damages the cells slightly. Representative endurance ranges are about 50,000–100,000 cycles for SLC, 1,000–3,000 for TLC, and around 1,000 or fewer for QLC; actual ratings vary widely by chip generation and maker. Controllers retire blocks that become unreliable.

Charge also leaks over time, especially in worn cells. Reading or programming can disturb nearby cells. Strong error-correcting codes (ECC) repair many resulting bit errors, and the controller can refresh data whose error count is rising. Flash is persistent storage with asymmetric operations and finite endurance, so a fast page read alone says little about the cost of sustained overwrites.

The host reads and overwrites logical blocks. The SSD's flash translation layer (FTL) maps those addresses to physical flash pages and manages the erases hidden beneath them. Keeping logical page N permanently at physical page N would make frequent overwrites repeatedly erase the same blocks.

Write a new version elsewhere

A log-structured FTL programs new data into a free page and changes its mapping. Suppose LBA 100 maps to physical page 0 and LBA 101 maps to page 1. An overwrite of LBA 100 is programmed into page 2. The map becomes 100 → 2; page 0 becomes invalid. The host still asks for LBA 100 and gets the new version.

Invalid is a mapping state, not an immediate physical erase. The old bits remain until their erase block is reclaimed. Deleting or overwriting a file therefore leaves no dependable promise about when every old physical copy disappears. Reliable erasure uses a suitable drive sanitize operation or a correctly managed whole-drive encryption scheme with key destruction.

Reclaim space and count extra writes

Garbage collection (GC) chooses a block, copies its valid pages elsewhere, updates the map, and erases the old block. Invalid pages need no copy. A block with many invalid pages yields more free space per copied live page.

Write amplification (WA) measures that extra programming:

In a 64-page block with 48 valid and 16 invalid pages, GC copies 48 pages. Erasing the block creates 64 free pages but the copies consumed 48 pages elsewhere: the net gain is 16. Filling that gain with 16 host pages produces 64 flash-page programs for 16 host-page writes, so WA = 4. Higher amplification consumes bandwidth and erase-cycle endurance faster.

Over-provisioning reserves flash space the host cannot fill. More free space gives GC flexibility to choose cheaper blocks. TRIM, called Deallocate in NVMe and UNMAP in SCSI, tells the device which logical ranges no longer contain needed data. A deleted file's pages can then be treated as invalid instead of copied during GC.

Budget mapping memory and wear

A page-level map needs one entry per logical page. A 1 TiB device with 4 KiB pages has:

At 4 bytes per entry, the table takes 1 GiB. Block-level mapping reduces the table but makes small updates expensive. Hybrid mapping combines block maps with page-mapped log areas; another approach caches only active page-map entries.

Wear leveling spreads erases across blocks. New writes already move around, but blocks holding cold, unchanged data may otherwise avoid wear forever while hot regions wear out. Moving some cold data frees lightly used blocks to join the write pool. That migration can add writes, so the controller balances even wear against copying cost.

An SSD has no disk arm or rotating platter. It avoids millisecond positioning delays, but its speed still depends on request size, parallelism, write history, and the work its controller does in the background.

Compare latency and throughput

These are representative device classes, not guarantees for every drive. SATA SSD throughput is constrained by the 6 Gb/s link. NVMe uses PCIe and can keep many flash channels busy at once.

A program issuing one small read and waiting for it before issuing the next sees each request's latency. If latency is 80 µs, that serial pattern allows about 12,500 operations per second. A device advertised for a much higher IOPS rate needs many requests in flight, supplied by threads or asynchronous I/O. Queue depth creates opportunities for parallel work; it can also add waiting when the device is saturated.

Account for write history

Random and sequential writes remain different. Random updates can leave invalid pages scattered among live pages, increasing GC copies. Sequential workloads can invalidate whole regions together, making reclamation cheaper and keeping WA closer to one. An empty SSD may program directly into free space; a filled SSD must reclaim it. Steady-state performance includes that ongoing GC cost, while fresh-out-of-box tests may not.

Read endurance ratings as a write budget

TBW is rated total terabytes written by the host. DWPD expresses full-drive host writes per day over the warranty period. For a 1 TB device rated at 600 TBW over five years:

At 50 GB per day, 600 TB of host writes would accumulate after 12,000 days, about 33 years. That arithmetic is a workload comparison, not a lifetime forecast: controller failures, aging, and warranty limits still matter. High internal amplification wears cells faster even though the endurance rating counts host writes.

Match the device to the work

Hard disks offer low cost per terabyte for large, infrequently accessed data sets. SSDs suit latency-sensitive work. Tiered storage keeps hot data on SSDs and colder data on disks. Both can fail: disks have mechanical faults and bad sectors; SSDs have wear, controller faults, and firmware bugs. Redundancy and separate backups remain useful for both.

On SSDs, defragmentation adds writes with little seek-related benefit. TRIM helps the controller identify unused data. Current partitioning tools commonly align starts to 1 MiB boundaries, avoiding poorly aligned block requests.

RAID, a redundant array of inexpensive or independent disks, presents several physical disks as one logical block device. Internally, a mapping sends each logical block to data copies or to a stripe with redundancy. Applications and file systems can continue using the block interface without understanding that layout.

Compare three separate measures

Capacity is the usable space for user data. Reliability describes the failures a layout can tolerate. Performance includes both throughput across many operations and latency for one operation. A level can improve one measure while reducing another. More aggregate throughput also need not reduce the latency of a single block request.

Use this simple model for equal-sized disks:

S and R use the same rate units, such as MB/s. On a hard disk they differ greatly: the earlier timing example gave about 195 MB/s sequentially and 0.34 MB/s for small random reads. Model formulas assume enough concurrent work and no controller bottleneck; their purpose is to expose the costs of the layout.

State the failure model

A fail-stop disk works correctly until it fails completely and visibly. This makes basic RAID reasoning manageable: the array knows which disk is missing. Real disks can also have unreadable sectors or return incorrect bytes. Those partial and silent faults require integrity checks in addition to a layout that survives whole-disk loss.

A redundant array runs in degraded mode while failed disks remain within its tolerance. It still serves data, but surviving disks perform extra work or have less spare read capacity. A rebuild fills a replacement from surviving information. A hot spare is an idle installed disk that lets rebuilding start promptly. RAID-0 has no redundancy and no such degraded operating state after losing a member.

Keep multi-disk updates consistent

RAID can run in a hardware controller with its own processor and protected cache, in OS software such as Linux md, or within a file system such as ZFS. Each must handle a write interrupted between updates to different disks. Mirrors can disagree; data and parity can describe different versions. The parity case is the write hole.

Protected nonvolatile cache or a suitable journal can preserve pending updates for recovery. A write-intent bitmap records which areas need resynchronization after interruption. It does not retain the new data and is not by itself a complete parity write-hole solution when a disk is also missing.

RAID maintains availability through specified disk failures. A deletion or ransomware write reaches the array's copies through the normal interface. Fire or theft can remove the entire array. Separate backups supply recoverable versions or copies beyond those shared events.

RAID-0 stripes data across disks without keeping redundant information. With a one-block chunk and zero-based numbering, logical block A goes to:

For four disks, block 14 belongs to disk 2 at offset 3. Modulo chooses the disk; integer division counts complete stripe rows before that block.

Include chunk size in the mapping

A chunk is a run of consecutive blocks placed on one disk before moving to the next. Let its size be c blocks. First identify the chunk, then the disk and local offset:

For block 14, c = 2, and N = 4, the chunk is 7, the disk is 3, and the offset is 2. Using A modulo N directly would ignore the two-block chunks. Small chunks can spread a large request across more disks; larger chunks keep more nearby work on one disk. Common array chunks range from 64 to 512 KiB; Linux md commonly defaults to 512 KiB.

RAID-0 has capacity NB and ideal aggregate sequential and random rates NS and NR. A small single-disk request still takes about T. Any member failure loses some array data, so this layout suits reproducible scratch data when availability is unimportant.

Add mirrored copies

A two-disk RAID-1 mirror writes each block to both members. RAID-10 stripes across mirrored pairs. With four disks and one-block chunks across pairs, block A goes to pair A modulo 2 at offset floor(A/2), and both members store it.

RAID-10 uses half the raw capacity. It survives any single failure and can survive N/2 failures if each hits a different pair. Losing both members of one pair loses that pair's data even if all other pairs remain healthy.

Count useful work per disk

Each mirrored write updates two disks in parallel. Its latency is about T, set by the slower member, while aggregate random write throughput is NR/2. Reads can be balanced across all copies, giving NR for enough independent random reads.

The simple mirrored sequential-read model gives NS/2 because partners skip over one another's blocks. Implementations that assign whole read ranges to each copy can do better. Mirroring supplies more read-serving capacity; it does not make one disk finish an individual read in half the time.

A parity stripe stores data blocks and a redundancy block equal to their bitwise XOR. For each bit position, XOR is one when an odd number of inputs are one. XORing a value with itself cancels it. That cancellation lets the array solve for one missing block.

Recover a known missing block

For data 0011, 0101, and 1001:

If the second block is missing:

The array must know which block to reconstruct. A parity mismatch among apparently healthy blocks shows inconsistency but does not identify the bad member. A checksum can help establish which data is valid.

Update one block without reading the whole stripe

Reconstructing parity from the other data is additive parity. With five data disks, changing one block requires reading the other four data blocks, then combining them with the new value. Subtractive parity, or read-modify-write, removes the old block's contribution and inserts the new one:

Change 0101 to 1100 in the stripe above:

This requires two reads, old data and old parity, followed by two writes, new data and new parity. The read phase must finish before the write phase, giving about 2T latency. A full-stripe write has every new data block, so it computes new parity directly and needs no old-block reads.

Distribute the parity work

RAID-4's parity disk performs a read and write for every small update, creating a fixed bottleneck. RAID-5 spreads parity across members, allowing unrelated stripes' updates to run concurrently. It still pays four physical I/Os per logical small write. Random reads can use all disks at NR; large sequential writes reach about (N − 1)S.

RAID-6 adds independent P and Q redundancy, commonly using a Reed–Solomon code for Q. Two copies of the same XOR would give only one equation. Updating one data block by read-modify-write reads old data, P, and Q, then writes all three: six I/Os.

For four 4 TB disks with R = 1 MB/s, RAID-4 and RAID-5 provide 12 TB and random write rates of 0.5 and 1 MB/s. RAID-6 provides 8 TB and about 0.67 MB/s. In degraded RAID-5, a missing data block needs N − 1 surviving-block reads. In RAID-6 with one missing data block, P reconstruction needs N − 2 reads and omits Q; two missing blocks require both independent equations.

Start a RAID choice with the workload and the losses it must survive. Capacity alone leaves out write cost, degraded performance, and the time needed to restore redundancy. A larger array has more disks contributing capacity, but also more members and sectors exposed during recovery.

Match the layout to the workload

RAID-0 suits scratch data that can be recreated and does not need disk-failure tolerance. RAID-10 suits many small random reads and writes when giving up half the capacity is acceptable. Parity RAID uses space efficiently and works well for large full-stripe writes. RAID-6 adds protection against any two failures, useful when rebuilding large drives takes a long time.

With four disks, RAID-10 and RAID-6 each provide half the raw capacity. RAID-10 needs two physical writes per small logical write; RAID-6 read-modify-write needs six I/Os. RAID-6 survives every two-disk failure pattern, while RAID-10 requires those failures to hit different pairs. Neither choice dominates every measure.

Calculate a rebuild lower bound

A mirror rebuild copies its surviving partner. A full parity-array rebuild reads surviving data and reconstructs missing blocks. RAID-6's P equation can reconstruct one missing data block without Q; actual recovery reads depend on layout and implementation. Full-device rebuilds can involve extensive reads across the survivors.

Writing a replacement 20 TB disk at 200 MB/s takes at least:

That is about 27.8 hours with no competing load or extra bottleneck. Normal I/O, seeks, controller work, and recovery throttling can make it longer. Adding a hot spare shortens the delay before rebuilding begins; it does not remove the copying time.

Examine the exposure during recovery

A degraded RAID-5 array has used its only disk-failure allowance. A further disk loss defeats recovery. An unreadable sector on a survivor can also leave one stripe with two unknown blocks: the missing member's block and the unreadable survivor's block. Other stripes may remain recoverable.

As an illustrative calculation, rebuilding four 4 TB RAID-5 members reads three full surviving disks:

If errors are modeled at one per 10¹⁴ bits, the expected count is 0.96. This is an expected count under that rate model, not a 96% failure probability. A data-sheet error-rate limit is not a measured prediction for a particular rebuild. Enterprise specifications often give a stronger limit, such as one per 10¹⁵ bits.

Increasing RAID-5 from four to eight disks raises usable capacity from 75% to 87.5% of raw space while increasing the surviving data read during recovery. Shared age and wear also weaken an assumption of independent failures. A write-heavy database may favor RAID-10; a large sequential archive may favor RAID-6. Schedule scrubbing to uncover bad sectors while redundancy remains, and retain separate backups.

The fail-stop model identifies a whole disk as missing. A fail-partial device mostly works while some blocks are unreadable or incorrect. Such faults can stay hidden in rarely accessed data until another failure makes those blocks necessary for a rebuild.

Separate a reported read error from wrong bytes

A latent sector error (LSE) means the drive cannot read a sector and reports an error. Physical damage or bit errors beyond internal ECC's correction strength can cause it. Because the location is known, a redundant system can read a mirror or reconstruct that block from parity. Rewriting recovered data lets the drive retire or remap the damaged location.

Silent corruption means a read returns incorrect bytes without reporting an error. Faulty firmware, a controller or bus fault, or a misdirected write can produce it. The command's successful status says the transfer completed; it is not an end-to-end validation of the contents.

Distinguish address and version errors

A misdirected write can overwrite an unrelated block while leaving the intended destination stale. A lost write leaves the intended block's old version in place. An address identifier helps catch the first fault; an independently stored expectation of the new version helps catch the second.

The drive's internal ECC corrects many errors in stored bits. It cannot check all faults elsewhere in the path, such as a value corrupted before the drive computes its code or after the drive returns it. Protection confined to the medium leaves other components unchecked.

Understand why scale matters

One field study followed 1.53 million production disks for 32 months. LSEs affected about 8.5% of nearline drives and 1.9% of enterprise-class drives. A separate corruption study of the same population over 41 months found checksum mismatches on about 0.86% and 0.065%, respectively. These are historical observations over different study periods, not annual probabilities or current guarantees. At a large fleet scale, even uncommon per-drive faults become regular operating events.

Use redundancy with verification

A normal two-way mirror read may choose one copy without comparing it to the other. It can return corrupt bytes if no checksum is checked. If the copies are compared and disagree, two copies alone do not say which is right. A valid checksum gives a basis for selecting a good version.

During a RAID-5 rebuild, an LSE on a survivor can remove the last information needed for its stripe. The array already has one missing block, and one XOR equation cannot recover two unknowns. Finding and repairing latent faults while all members remain available preserves options for the next failure.

A checksum summarizes a data block in a much smaller value. On reading, recompute it and compare it with the stored expectation. A mismatch detects inconsistency. Different inputs can share a value, called a collision, so matching checksums provide strong evidence whose strength depends on the chosen function and error pattern.

Try XOR and addition

For bytes 0x36, 0x5A, 0xC1, and 0x0F, bitwise XOR gives 0xA2. In binary the calculation is:

Flip the lowest bit in both first bytes, making 0x37 and 0x5B. The two changes cancel under XOR, leaving 0xA2. A cheap operation can have a predictable blind spot.

An 8-bit addition checksum adds the byte values modulo 256:

The paired flip above changes the sum to 0x62, so addition catches that error. But reordering bytes leaves both simple addition and XOR unchanged.

Add sensitivity to position

A Fletcher checksum keeps two sums, initialized to zero. For each byte d, update the first sum and then add its new value into the second:

The result is (97, 122). Swapping the first two bytes gives (97, 158). The first sum ignores order, but the second weights positions through the running accumulation. Fletcher detects single-bit errors and many burst errors at low cost, though collisions remain possible.

Follow a short CRC division

A cyclic redundancy check (CRC) treats bits as polynomial coefficients and divides using XOR. With generator 1011, representing x³ + x + 1, a three-bit remainder protects the data. For data 1101, append three zeros:

1. Start with 1101000 and XOR with 1011000, leaving 0110000.
2. Align the generator to the leading one and XOR with 0101100, leaving 0011100.
3. XOR with 0010110, leaving 0001010.
4. XOR with 0001011, leaving 0000001.

The remainder is 001, so store 1101001. Dividing that codeword gives zero; changing it to 1111001 gives remainder 110. An n-bit CRC detects every burst of length at most n bits. CPU support makes CRC practical for storage and network traffic, including CRC32C in ext4 metadata and Btrfs.

Cryptographic hashes such as SHA-256 also resist deliberate collision attempts. A hash used against tampering needs a trusted expected value; replacing both data and its untrusted hash defeats comparison. OpenZFS normally uses Fletcher for ordinary data and SHA-256 for deduplication. Eight checksum bytes per 4 KiB block cost about 0.2% space. Detection alone supplies no replacement bytes: repair needs redundancy or an error-correcting code.

A checksum function decides which changes are detectable. The storage design decides what value it is compared against, which address it belongs to, and whether a good replacement exists. A strong checksum attached to the wrong version can still accept stale data.

Choose a place for the expectation

ZFS uses checksums in parent block pointers, creating a verification tree up to its root. Protection information can also carry the expected address. T10's reference tag uses the lower 32 bits of the logical block address alongside a 16-bit data CRC and an application tag. An address check can reject correct bytes returned from the wrong location.

Separate wrong location from wrong version

Suppose a read of disk 3, block 1,207 returns a valid data/checksum pair labeled for another address. The checksum alone agrees, but the address identifier reveals the misdirection.

A lost write is harder: the old contents can have a valid checksum and the correct address. Read-after-write verification compares a fresh read with the expected new contents, adding roughly another I/O. It must observe the stored result, not merely the same volatile cached copy. An expected checksum stored separately, such as in a committed parent pointer, can instead reject the stale child version. The separate expectation must itself describe the intended update.

Verify before returning or repairing

On a read, obtain the block and its expected checksum, recompute, and compare. If the first copy fails, read another copy or reconstruct the block from redundancy. Verify that result before returning it. If it passes, return the good data and rewrite the damaged copy. This is self-healing. If no valid copy can be found, return an error instead of unchecked bytes.

For example, a mirrored ZFS pool reads disk A and detects a checksum mismatch. Disk B's copy passes. The system returns B's data, repairs A, and records A's error. Ordinary mirroring without checksum verification could return A's bad bytes without noticing.

Look for errors before a failure forces recovery

Scrubbing periodically reads and verifies stored data, including blocks applications rarely touch. With redundancy available, the system can repair faults before another member fails. In the historical LSE study, scrubbing found more than 60% of the errors.

Verification consumes CPU time, checksum space, and sometimes extra I/O. Eight bytes per 4 KiB cost about 0.2%; eight per 512 data bytes cost about 1.6%. Scrub reads compete with foreground requests, so quiet periods are useful. The end-to-end argument puts the final check near the file system or application, covering more of the path than a drive's medium-only ECC.