Linux Detailed Course#

Table of Contents#

1. 1. The Linux Boot Sequence
2. 2. Filesystem Hierarchy & Virtual FS
3. 3. systemd & Unit Architecture
4. 4. Disk Partitioning, Filesystems & LVM
5. 5. Memory, Page Cache & OOM Killer
6. 6. Kernel Modules & sysctl Tuning
7. 7. Network Stack, iptables & nftables
8. 8. Security, SSH Hardening & Capabilities
9. 9. Performance Diagnostics & eBPF
10. 10. Production Troubleshooting Runbook

1. The Linux Boot Sequence#

From power-on to a multi-user shell:

1. UEFI / BIOS: Hardware Power-On Self-Test (POST) queries non-volatile RAM and executes the primary bootloader from the EFI System Partition (ESP).
2. GRUB2 (Grand Unified Bootloader): Loads the compiled Linux kernel binary (vmlinuz) and the Initial RAM Disk (initramfs / initrd) into memory.
3. Kernel Initialization: Mounts initramfs as a temporary root filesystem, detects core hardware, loads device drivers, and mounts the real root filesystem (/).
4. Init (PID 1): The kernel starts user space by launching /sbin/init (typically symlinked to systemd), which brings up targets, mounts, and background services in parallel.

2. Filesystem Hierarchy & Virtual FS#

Linux organizes files by purpose according to the Filesystem Hierarchy Standard (FHS):

• /etc: Host-specific system configuration files (e.g. /etc/hosts, /etc/fstab).
• /var: Variable data files that persist across boots (logs, mail spools, web root /var/www).
• /proc: Virtual in-memory filesystem exposing live kernel statistics and per-process metadata (e.g. /proc/cpuinfo, /proc/<PID>/cmdline).
• /sys: Unified device model exposing kernel tunable parameters and hardware buses (sysfs).
• /dev: Device nodes representing physical or virtual hardware (e.g. /dev/nvme0n1, /dev/urandom).

3. systemd & Unit Architecture#

Modern Linux distributions use systemd as their init system and service manager. Writing custom services:

# /etc/systemd/system/techtoday-api.service
[Unit]
Description=TechToday Backend Service
After=network.target

[Service]
Type=simple
User=ec2-user
WorkingDirectory=/var/www/techtoday
ExecStart=/usr/bin/python3 -m app.server
Restart=always
RestartSec=5s
Environment=PORT=8000
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload           # Reload unit configuration
sudo systemctl enable --now techtoday-api # Enable on boot and start immediately
sudo systemctl status techtoday-api    # Check health and recent log output
journalctl -u techtoday-api -f         # Follow live logs with systemd-journald

4. Disk Partitioning, Filesystems & LVM#

Storage management breaks into three abstraction tiers under Logical Volume Management (LVM):

1. Physical Volumes (PV): Underlying raw disks or partitions (e.g. /dev/sdb1).
2. Volume Groups (VG): Pools of combined physical storage chunks.
3. Logical Volumes (LV): Virtual partitions created from a VG that can be resized live without unmounting the filesystem.

lsblk                                # Display block storage device hierarchy
mkfs.ext4 -L DataVol /dev/vg0/lv_app # Format logical volume with ext4
# Add to /etc/fstab for persistent mounting on reboot:
# /dev/vg0/lv_app  /data  ext4  defaults,noatime  0  2
mount -a                             # Mount all filesystems in fstab

5. Memory, Page Cache & OOM Killer#

Linux aggressively uses free RAM for the Page Cache to accelerate disk reads. When memory runs critically low:

• The kernel scans page lists and writes dirty pages to disk via background flush threads.
• If memory pressure cannot be relieved, the Out-Of-Memory (OOM) Killer activates. It calculates a "badness score" based on memory footprint and /proc/<PID>/oom_score_adj, sending SIGKILL (9) to the highest-scoring candidate to protect the system.

6. Kernel Modules & sysctl Tuning#

Kernel code can be dynamically loaded at runtime without rebooting:

lsmod                       # List currently loaded kernel modules
modinfo wireguard           # Inspect module details and dependencies
sudo modprobe overlay       # Load overlayfs module dynamically

# Tune live kernel parameters via sysctl
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
sudo sysctl -w fs.file-max=2097152
# Persist permanently in /etc/sysctl.d/99-custom.conf:
sudo sysctl --system

7. Network Stack, iptables & nftables#

Linux packet filtering uses the kernel's Netfilter hooks:

• PREROUTING → INPUT (local process) or FORWARD (routing) → OUTPUT → POSTROUTING.
• iptables / nftables: Filter by port, IP, interface, and connection state.
• Network Namespaces: Isolates network interfaces, allowing containers (like Docker) to have their own virtual Ethernet pairs (veth) and routing tables.

8. Security, SSH Hardening & Capabilities#

Enterprise server hardening checklist:

1. Disable Password Authentication: Enforce SSH keys only in /etc/ssh/sshd_config (PasswordAuthentication no).
2. Disable Root SSH Login: PermitRootLogin no.
3. Linux Capabilities: Instead of granting full root privileges, grant fine-grained capabilities (e.g. setcap 'cap_net_bind_service=+ep' /usr/bin/node to bind ports < 1024 without running as root).
4. Mandatory Access Control (MAC): SELinux (RHEL/CentOS) and AppArmor (Debian/Ubuntu) enforce strict security profiles on binaries regardless of file permissions.

9. Performance Diagnostics & eBPF#

Troubleshooting high latency and crashing programs:

# Trace system calls of a live running process
strace -p <PID> -T -e trace=network,file

# List open files, sockets, and pipes held by a process
lsof -p <PID>
lsof -i :8000                 # Find which process holds port 8000

# High-resolution hardware profiling with Linux perf
perf top                      # Real-time CPU function hotspot analysis

10. Production Troubleshooting Runbook#

Next Steps: Review core OS kernel concepts in the OS Detailed Course, or test fundamentals in the Linux Crash Course. Browse the entire catalog at All OS Courses.