Networking Detailed Course#

Table of Contents#

1. 1. Autonomous Systems & BGP Routing
2. 2. TCP Congestion Control: CUBIC & BBR
3. 3. Flow Control & Sliding Window Scaling
4. 4. Linux Socket Lifecycle & Ring Buffers
5. 5. Zero-Copy I/O: sendfile & splice
6. 6. QUIC Protocol Architecture & Multiplexing
7. 7. Anycast Routing & Edge CDNs
8. 8. DDoS Mitigation & SYN Cookies
9. 9. High-Throughput Linux sysctl Tuning
10. 10. Top Networking Engineering Questions

1. Autonomous Systems & BGP Routing#

The global Internet is not a single mesh; it is a network of networks called Autonomous Systems (ASes) operated by ISPs, cloud giants (AWS, Google, Cloudflare), and telecommunications providers, each identified by an Autonomous System Number (ASN).

• Border Gateway Protocol (BGP): The routing protocol that glues the Internet together. BGP is a path-vector protocol where routers advertise the list of ASes a packet must traverse to reach a destination prefix.
• Peering vs Transit: In peering, two networks exchange traffic between their respective customers for free. In transit, a smaller network pays an upstream Tier-1 ISP for access to the entire global routing table.

2. TCP Congestion Control: CUBIC & BBR#

While Flow Control prevents a fast sender from overwhelming a slow receiver, Congestion Control prevents senders from overwhelming the shared intermediate routers on the network:

1. Loss-Based (TCP Reno / CUBIC): Assumes packet loss indicates network congestion. Sends data exponentially (Slow Start), then linearly (Congestion Avoidance). When a packet drops, CUBIC slashes the Congestion Window (cwnd) by a multiplicative factor and uses a cubic growth curve. Weakness: Suffers from Bufferbloat on modern high-speed links with massive router queues.
2. Model-Based (Google BBR - Bottleneck Bandwidth and RTT): Rather than waiting for packet drops, BBR measures the maximum delivery rate (bandwidth) and minimum round-trip time (RTprop). It paces packet transmission exactly at the pipe's capacity, eliminating queue build-up and latency spikes.

3. Flow Control & Sliding Window Scaling#

TCP Flow Control coordinates delivery speeds using the Receive Window (rwnd):

• The receiver advertises how many free bytes remain in its kernel socket buffer (SO_RCVBUF).
• The sender never transmits more unacknowledged bytes than min(cwnd, rwnd).
• TCP Window Scale Option (RFC 1323): The original 16-bit TCP header only allowed a maximum window size of 64 KB (insufficient for gigabit bandwidth-delay products). Window Scaling adds a shift multiplier allowing windows up to 1 GB.

4. Linux Socket Lifecycle & Ring Buffers#

Tracing data from a physical Ethernet wire into an application process:

1. NIC & DMA: Network Interface Card receives optical/electrical pulses, reconstructs the Ethernet frame, and writes it directly into host RAM ring buffers via Direct Memory Access (DMA).
2. Hard IRQ & SoftIRQ (NAPI): The NIC raises a hardware interrupt to the CPU. The kernel schedules a software interrupt (NAPI softirq) to poll the ring buffer in batches, avoiding interrupt storms.
3. TCP Stack: Kernel strips the Ethernet header, validates IP checksum, decodes TCP sequence numbers, reassembles packets, and deposits data into the socket's receive queue.
4. Application Read: Process calls read() or recv(), copying data from kernel space into the user application buffer.

5. Zero-Copy I/O: sendfile & splice#

Traditional file serving requires 4 context switches and 4 data copies (Disk → Kernel → User space → Kernel → NIC). Linux sendfile() achieves Zero-Copy:

// Traditional approach: 4 copies, 4 context switches
read(file_fd, buffer, len);
write(socket_fd, buffer, len);

// Zero-Copy approach: 2 context switches, 0 user-space copies
sendfile(socket_fd, file_fd, NULL, len);
// The kernel streams data directly from page cache to the NIC DMA buffer!

6. QUIC Protocol Architecture & Multiplexing#

QUIC (RFC 9000) reinvents transport layer semantics by embedding reliable multiplexing directly on top of UDP:

• Independent Stream Frames: In HTTP/2 over TCP, if one packet drops, all multiplexed streams stall until that single packet is retransmitted. In QUIC, each stream is tracked independently; a dropped packet on Stream A never delays Streams B or C.
• Built-in TLS 1.3: Encryption is not a layer on top of QUIC; the transport handshake and cryptographic key exchange are unified into a single 0-RTT / 1-RTT round-trip.
• Connection IDs: Connections are identified by a 64-bit Connection ID rather than an IP:Port 4-tuple. If your phone switches from home Wi-Fi to cellular LTE, existing file transfers and calls continue without dropping.

7. Anycast Routing & Edge CDNs#

Traditional hosting uses Unicast (one IP address belongs to exactly one physical server). Modern networks (Cloudflare, Google, AWS CloudFront) use BGP Anycast:

• The exact same IP address (e.g. 1.1.1.1) is advertised by BGP routers in 300+ data centers around the world simultaneously.
• When a user in Tokyo sends a packet, global BGP routing sends it to the Tokyo POP. When a user in London sends a packet to the same IP, it routes to London.
• Minimizes latency, optimizes DNS resolution times, and naturally disperses DDoS attacks across the globe.

8. DDoS Mitigation & SYN Cookies#

In a SYN Flood Attack, an attacker sends millions of spoofed TCP SYN packets without ever sending the final ACK, overflowing the kernel's connection backlog and freezing the server.

SYN Cookies (Linux solution): When the backlog queue fills, the kernel stops allocating memory for incoming connections. Instead, it encodes the client's IP, port, and a cryptographic timestamp into the initial 32-bit sequence number (Y). When the client responds with ACK (containing Y + 1), the server validates the signature and establishes the socket without ever having held state.

9. High-Throughput Linux sysctl Tuning#

# /etc/sysctl.d/99-networking.conf
# Increase maximum listen queue backlog for high-concurrency servers
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Enable BBR congestion control
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Expand TCP socket buffer sizes for 10GbE+ links
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Accelerate socket recycling
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

10. Top Networking Engineering Questions#

Next Steps: Review quick mental models in the Networking Crash Course, or explore OS kernel internals in the OS Detailed Course. Return to the TechToday Homepage.