How the OS Manages Networking: the Network Stack

Few basic operating systems textbooks dwell at any length on networking; where they do, they usually talk about the structure and format of popular layer 2,3, and 4 network protocols. Some intrepid textbooks discuss this topic as the basis of OS facilities like remote or network file systems, or in the context of a "distributed systems" chapter.

This omission is funny because a network stack is an almost perfect operating systems principles topic, bringing together many of the issues otherwise studied in the context of an OS. One must remember, of course, that a network is nothing more than a collection of machines who have agreed to speak the same language; the network exists at their pleasure and only as a set of distributed state among their OS network stacks: the data structures and functions that make up the part of the kernel responsible for creating, delivering, and receiving packets.

Teaching OS From (Almost) One Concept

How an operating system manages networking is an almost perfect example of the many roles and responsibilities of a OS kernel. There are user management aspects, including utilities that use parts of the system call API to read and write important network--related state. There are major virtual constructs like interfaces; this set also includes concepts like sockets that can be treated much like files (along with the kernel state necessary to keep track of them and to which process they belong). This topic also includes great examples of how the kernel manages and talks to hardware like network interfaces: how packets are received, accepted (or discarded) by the hardware, and then an interrupt is issued to the kernel (interrupt handling, interrupt context) to copy the packets and later deliver them (top half/ bottom half) to the target process (the appropriate socket). This activity must be carried out concurrently with other user level and kernel level execution, bringing into play discussions of concurrency and locking primitives.

The User Level: Network Interfaces

What network "hardware" does the machine have? lspci

The ifconfig command. Now replaced with the //ip// command.

[locasto@csl ~]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:15:C5:EE:84:6A  
          inet addr:136.159.5.22  Bcast:136.159.5.255  Mask:255.255.255.0
          inet6 addr: fe80::215:c5ff:feee:846a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:51900841 errors:0 dropped:0 overruns:0 frame:0
          TX packets:55472494 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:26853120447 (25.0 GiB)  TX bytes:31429849417 (29.2 GiB)
          Interrupt:16 Memory:f8000000-f8012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:6705419 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6705419 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1508894646 (1.4 GiB)  TX bytes:1508894646 (1.4 GiB)

vmnet1    Link encap:Ethernet  HWaddr 00:50:56:C0:00:01  
          inet addr:172.16.104.1  Bcast:172.16.104.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vmnet8    Link encap:Ethernet  HWaddr 00:50:56:C0:00:08  
          inet addr:192.168.160.1  Bcast:192.168.160.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[locasto@csl ~]$

The User Level: Bound Network Ports and Network State

The netstat command. The arp command. The route command.

The System Call API

strace filter

the nc command

man 2 socket

socket / bind / listen / accept

send / recv / sendto / recvfrom, etc.

read / write

select / poll

Kernel Network State Configuration: sysctl

[locasto@csl ~]$ sysctl -a net.* | wc
...
error: permission denied on key 'kernel.cad_pid'
error: permission denied on key 'kernel.usermodehelper.bset'
error: permission denied on key 'kernel.usermodehelper.inheritable'
    938    2918   34266
[locasto@csl ~]$

Kernel Code

The Linux kernel (and many other OS kernels) have a number of responsibilities related to networking. They attempt to:

(device drivers) communicate with and use a particular NIC or network hardware to physically communicate with a network, whether this is wired Ethernet or 802.11 or Bluetooth or ...
(sockets) maintain a number of virtual communications endpoints
maintain connectivity with layer 2 and layer 3 (arp, ICMP, keepalive, etc.)
provide "advanced" features like NAT
(netfilter) provide the ability to act as a firewall
(forwarding) provide the ability to act as a gateway or router

Documentation: http://lxr.cpsc.ucalgary.ca/lxr/#linux/Documentation/networking/

http://lxr.cpsc.ucalgary.ca/lxr/#linux/net/

http://lxr.cpsc.ucalgary.ca/lxr/#linux/net/packet/af_packet.c

sys_recv: http://lxr.cpsc.ucalgary.ca/lxr/#linux+v2.6.32/net/socket.c#L1765