Operating Systems Detailed Course#

Table of Contents#

1. Kernel Architectures
2. Process State Machine & Lifecycle
3. Thread Models & User vs Kernel Threads
4. Advanced CPU Scheduling & Linux CFS
5. Synchronization Internals & Futexes
6. Deadlocks & Banker's Algorithm
7. Multi-Level Paging & TLB Shootdown
8. Copy-On-Write (COW) & Page Swapping
9. File System Internals & Journaling
10. I/O Multiplexing: epoll & io_uring
11. Linux Container Internals
12. Core OS Interview Questions

1. Kernel Architectures#

Operating systems differ fundamentally in how much code runs in privileged kernel mode:

Monolithic Kernels (Linux, FreeBSD): All core services — scheduling, virtual memory, file systems, IPC, and device drivers — run within Ring 0. Pros: Maximum performance due to zero context-switch overhead between subsystem function calls. Cons: A bug or crash in any third-party device driver can bring down the entire system.
Microkernels (Mach, seL4, QNX): Only the bare minimum runs in Ring 0: basic thread scheduling, low-level memory mapping, and IPC. Drivers, file systems, and network stacks run as isolated user-space servers. Pros: Extreme fault-tolerance and formal verification capabilities. Cons: High IPC messaging and context-switch overhead across subsystem boundaries.
Hybrid Kernels (macOS XNU, Windows NT): Pragmatic compromise combining microkernel-like message passing structures with performance-critical subsystems running in kernel address space.

2. Process State Machine & Lifecycle#

From creation to termination, a process transitions through a deterministic state machine:

New (Created): Process is being instantiated by fork() or clone(); PCB is initialized.
Ready: Loaded into RAM and waiting in the run-queue for CPU core assignment by the scheduler.
Running: Instructions are actively being executed on a hardware CPU core.
Waiting (Blocked): Paused awaiting an external event (e.g. disk read, socket packet, timer, or lock).
Terminated (Zombie): Process has called exit(), but its return code remains in the PCB until the parent process calls wait() / waitpid().

Orphan vs Zombie: An orphan process is one whose parent died before it; it is adopted by PID 1 (systemd or init). A zombie process is dead but waiting for its parent to read its exit status so its PCB can be freed.

3. Thread Models & User vs Kernel Threads#

How application threads map to kernel execution contexts:

1:1 Model (Kernel Threads): Each application thread maps directly to an OS thread (POSIX pthread on Linux via clone(CLONE_VM | CLONE_FS | ...)). The OS handles scheduling across all CPU cores. High fidelity, but thread creation and stack memory (typically 1-8 MB) are relatively heavy.
N:1 Model (Green Threads / Fiber): Many user-space threads multiplexed on one kernel thread. Ultra-fast switching with tiny stacks, but cannot utilize multi-core CPUs and one blocking syscall blocks the entire process.
M:N Model (Go Goroutines, Erlang BEAM): M application coroutines scheduled dynamically across N OS kernel threads by a runtime scheduler. Delivers the best of both worlds: sub-microsecond context switches, tiny 2 KB initial stacks, and multi-core parallelism.

4. Advanced CPU Scheduling & Linux CFS#

The Linux Completely Fair Scheduler (CFS) models an ideal multi-tasking CPU where every task receives an equal share of compute time proportional to its nice priority:

Virtual Runtime (vruntime): Represents the normalized amount of CPU time a task has consumed. A higher-priority task (lower nice value) accumulates vruntime at a slower rate.
Red-Black Tree: The CFS runqueue is organized as a self-balancing binary search tree keyed by vruntime. The scheduler always picks the leftmost node (the task with the lowest accumulated vruntime) in O(log N) time (and O(1) cached lookup).

5. Synchronization Internals & Futexes#

Naive locks either waste CPU cycles spinning in user mode or waste hundreds of nanoseconds in syscalls. Linux solved this with Futexes (Fast Userspace Mutexes):

Fast Path (Uncontended): The lock is acquired entirely in user space with a single atomic hardware instruction (e.g. compare-and-swap / CMPXCHG) with zero system call overhead.
Slow Path (Contended): If the lock is already held, the thread issues the futex() syscall to put itself to sleep in the kernel's wait-queue until the holder unlocks it.

6. Deadlocks & Banker's Algorithm#

Dijkstra's Banker's Algorithm evaluates resource allocation requests by simulating whether granting the request leaves the system in a Safe State (a state where a sequence exists allowing all processes to eventually complete):

// Safety Algorithm check
Available[m]: Available instances of resource type m
Max[n][m]: Max demand of process n
Allocation[n][m]: Currently allocated instances
Need[n][m] = Max[n][m] - Allocation[n][m]

// A state is safe if there exists a sequence 
// such that for each Pi, Need[i] <= Available + Sum(Allocation[j]) for j < i

7. Multi-Level Paging & TLB Shootdown#

A flat 64-bit page table would require millions of gigabytes just to store page mappings. Modern OSes use hierarchical 4-Level or 5-Level Paging (PML4 / PML5 on x86-64):

The virtual address is partitioned into offsets pointing through: PGD (Page Global Directory) → P4D → PUD (Page Upper Directory) → PMD (Page Middle Directory) → PTE (Page Table Entry) → Physical Page Offset.
Tables for unused address regions never need to be allocated in RAM, keeping the memory overhead tiny.
TLB Shootdown: When a page mapping is modified or unmapped on a multi-core machine, the initiating CPU must send an Inter-Processor Interrupt (IPI) to all other cores to flush their local TLB entries for that address.

8. Copy-On-Write (COW) & Page Swapping#

When fork() is called, copying gigabytes of memory from parent to child would be prohibitively slow. Instead, the OS uses Copy-On-Write (COW):

Both parent and child page tables point to the exact same physical frames.
The OS marks all shared pages as Read-Only in both page tables.
When either process attempts to write to a page, a CPU memory protection fault is raised.
The OS kernel catches the fault, allocates a new physical frame, copies only that single 4 KB page, marks it writable, and points the writing process's page table entry to the new duplicate.

9. File System Internals & Journaling#

If power is cut while writing a file, metadata and data can fall out of sync, destroying the file system. Journaling (ext4, NTFS, XFS) ensures crash-consistency:

Journal Write: The intended metadata updates are sequentially written to a dedicated on-disk circular log (the journal) along with a checksum.
Commit: A commit block is written to disk, marking the transaction as sealed and durable.
Checkpointing: The updates are written to their permanent locations in the inode and data block tables.
Recovery: On reboot after a crash, the OS replays the committed journal transactions in seconds without needing a full-disk fsck scan.

10. I/O Multiplexing: epoll & io_uring#

Scalable networking requires managing tens of thousands of connections per server (the C10K / C1000K problem):

select() / poll() (O(N)): The application passes an array of thousands of file descriptors into the kernel on every call; the kernel scans every single one, yielding poor performance under high connection counts.
epoll (Linux) / kqueue (BSD/macOS) (O(1)): The kernel maintains an interest list in a red-black tree and an event-ready list. When a socket receives network data, a hardware interrupt triggers a callback placing the FD directly on the ready list.
io_uring (Modern Linux): Shared ring-buffers between user space and kernel space for true asynchronous submission and completion of I/O operations with zero syscall overhead per event.

11. Linux Container Internals#

Containers are not virtual machines; there is no hypervisor or guest operating system. A container is simply a standard Linux process isolated using two primary kernel primitives:

Linux Namespaces (What a process can see):
- PID: Isolates process IDs (container process sees itself as PID 1).
- NET: Isolates network interfaces, routing tables, and IP addresses.
- MNT: Isolates file system mount points.
- IPC: Isolates POSIX message queues and shared memory.
- UTS: Isolates hostname and domain name.
- USER: Maps container root (UID 0) to an unprivileged host UID.
Control Groups (cgroups v2) (What a process can use): Enforces resource boundaries and throttling for CPU shares, memory limits, disk I/O IOPS, and network bandwidth.