Addressing and routing

Created

The internet layer moves a packet from one host to another across many networks, using IP addresses to name hosts and routing tables to pick each next hop. An IP address has a network part, shared by every host on one link, and a host part. Ports extend an address to one process. On a link, ARP (IPv4) or NDP (IPv6) turns the next hop’s IP address into a MAC address. ICMP carries errors and diagnostics, and each router picks the next hop for a packet from its routing table, filled by hand or by routing protocols such as BGP.

Examples come from four Alpine containers on two Docker networks, with a container acting as the router between them:

The lab. Network lan1, 10.20.1.0/24 and fd00:20:1::/64, holds client at 10.20.1.10 and fd00:20:1::10, peer at 10.20.1.11 and fd00:20:1::11, and Docker’s gateway at 10.20.1.1. The container router joins lan1 at 10.20.1.254 and lan2 at 10.20.2.254. Network lan2, 10.20.2.0/24, holds server at 10.20.2.20.

IPv4 addresses

  • An IPv4 address is 32 bits, written as four decimal bytes: 10.20.1.10. About 4.3 billion addresses exist, and IANA handed its last free blocks to the regional registries in 2011.
  • The prefix length, written /24, says how many leading bits are the network ID. The rest are the host ID. 10.20.1.10/24 has network ID 10.20.1.0 and host ID 10.
  • A netmask writes the same thing as an address: /24 is 255.255.255.0.
  • Two hosts are on the same link when their network IDs match. They exchange frames directly; anything else goes to a router.
  • In each network, the all-zeros host ID names the network itself and the all-ones host ID is its broadcast address, so a /24 holds 254 hosts.
RangeUse
10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16Private networks; not routed on the internet
127.0.0.0/8Loopback, the host itself
169.254.0.0/16Link-local, self-assigned when no other address is configured
100.64.0.0/10Shared address space for carrier NAT
224.0.0.0/4Multicast
255.255.255.255Broadcast on the local link

Subnets

Subnetting moves the boundary between network and host bits to split one network into smaller ones. CIDR (Classless Inter-Domain Routing) allows any prefix length, replacing the old classes A, B, and C, which allowed only /8, /16, and /24. One address, 10.20.1.70, in three prefix lengths (busybox ipcalc):

PrefixNetmaskNetworkBroadcastHosts
/24255.255.255.010.20.1.010.20.1.255254
/26255.255.255.19210.20.1.6410.20.1.12762
/30255.255.255.25210.20.1.6810.20.1.712
  • Each extra prefix bit halves the network: a /24 splits into two /25s, four /26s, and so on.
  • A network address is the address with its host bits cleared; for /26, the last byte rounds down to a multiple of 64.
  • Aggregation works the other way: routes for 10.20.1.0/24 and 10.20.2.0/24 can be announced together as 10.20.0.0/22 when all four /24s in it go the same way. It keeps routing tables small.

Ports and socket addresses

  • A port is a 16-bit number, 0 to 65535, that names an endpoint inside a host. TCP and UDP each have their own ports.
  • A socket address is an IP address and a port: 10.20.2.20:8080.
  • A TCP connection is identified by the protocol and both socket addresses. ss -tn on the client showed 10.20.1.10:34443 connected to 10.20.2.20:8080.
  • The server listened on *:8080, every address of the host. The client’s port, 34443, was picked by the kernel from its temporary range, 32768 to 60999 by default on Linux.
  • IANA divides ports into well-known (0–1023, such as 80 and 443), registered (1024–49151), and dynamic (49152–65535).

Network address translation

NATNAT Network address translation: rewriting the IP addresses, and often the ports, of packets as they pass a router, such as to share one public address among many hosts. lets many hosts share one public IPv4 address. A router rewrites the source address of outgoing packets, and usually the source port, to its own public address, records the mapping, and rewrites replies back. Hosts behind it use private addresses. A NAT also blocks connections that start from outside unless a mapping exists, such as Docker’s -p 8080:80. It conserves IPv4 addresses, but breaks the end-to-end model: a host’s own address isn’t the one the other side sees.

Delivery: unicast, broadcast, multicast, anycast

  • Unicast: one sender to one receiver. Almost all traffic.
  • Broadcast: one sender to every host on the link. IPv4 only, and routers don’t forward it.
  • Multicast: one sender to every host that joined a group, such as 224.0.0.1, all hosts on the link.
  • Anycast: one address announced from several places; routing delivers to the nearest. Public DNS resolvers, such as 1.1.1.1, use it.

The client pinged the broadcast address 10.20.1.255 and the multicast group 224.0.0.1. Captures on peer and router showed both arrive, addressed to the MAC ff:ff:ff:ff:ff:ff and to 01:00:5e:00:00:01; a multicast MAC is 01:00:5e followed by the group’s low 23 bits. Neither host answered: Linux ignores echo requests sent to broadcast and multicast addresses by default (icmp_echo_ignore_broadcasts is 1), so a single packet can’t make every host reply at once.

From IP to MAC: ARP

A frame on a link is addressed by MAC address, a 48-bit hardware address such as 3e:f6:60:8d:df:99. Before sending to an IP address on its own link, or to the router for anything else, a host needs that address’s MAC. ARP (Address Resolution Protocol) asks for it:

FrameFromToSays
ARP request, 42 bytesclient’s MACff:ff:ff:ff:ff:ff, every hostwho-has 10.20.1.11 tell 10.20.1.10
ARP reply, 42 bytespeer’s MACclient’s MAC10.20.1.11 is-at 86:c3:a4:f7:ed:b7
  • Every host on the link received the request; only the owner of 10.20.1.11 replied.
  • The client stored the answer in its neighbor cache: ip neigh was empty before the ping and showed 10.20.1.11 dev eth0 lladdr 86:c3:a4:f7:ed:b7 REACHABLE after.
  • ARP has no authentication. Any host can answer, which is how ARP spoofing redirects traffic on a shared link.
  • A MAC address matters only on one link. A router builds a new frame for each hop, with new MAC addresses, while the IP addresses stay the same end to end.

IPv6 addresses

An IPv6 address is 128 bits, written as eight groups of four hexadecimal digits. Two rules shorten it: drop leading zeros in each group, and replace the longest run of all-zero groups with ::, once per address.

Written outShort form
fd00:0020:0001:0000:0000:0000:0000:0010fd00:20:1::10
fe80:0000:0000:0000:3cf6:60ff:fe8d:df99fe80::3cf6:60ff:fe8d:df99
0000:0000:0000:0000:0000:0000:0000:0001::1

A network is almost always a /64: the first 64 bits are the network ID and the last 64 the interface ID. Address categories, by prefix:

PrefixCategoryExample in the lab
2000::/3Global unicast, routed on the internetnone
fc00::/7Unique local (ULA), private like 10.0.0.0/8; fd00::/8 in practicefd00:20:1::10
fe80::/10Link-local, valid on one link only, on every interfacefe80::3cf6:60ff:fe8d:df99
ff00::/8Multicast; ff02::1 is all nodes on the linkff02::1
::1/128Loopback::1
  • Every IPv6 interface has a link-local address, with or without any other. The client’s was built from its MAC 3e:f6:60:8d:df:99: flip the seventh bit of the first byte (3e becomes 3c) and insert ff:fe in the middle. Hosts on real networks often use random interface IDs instead, so the MAC doesn’t leak.
  • A link-local address needs the interface to be named, such as fe80::…%eth0, because every link uses the same prefix.
  • IPv6 has no broadcast. Neighbor Discovery (NDP), part of ICMPv6, replaces ARP: the client asked for fd00:20:1::11 with a neighbor solicitation sent to the solicited-node multicast group ff02::1:ff00:11, built from the address’s last 24 bits, so only hosts whose addresses end that way process it.
  • Unlike IPv4 broadcast, an echo request to ff02::1 was answered: peer and router both replied, because Linux answers IPv6 multicast echo requests by default.

What IPv6 changes from IPv4:

  • Enough addresses for every device to have a global one, so NAT isn’t needed.
  • A fixed 40-byte header with no checksum, against IPv4’s 20 to 60 bytes.
  • Routers never fragment; only the sender does, after learning the path MTU.
  • Hosts can configure their own addresses from a router’s advertised prefix (SLAAC, stateless address autoconfiguration), with no DHCP server.
  • Multicast replaces broadcast, so a request bothers only the hosts that might care.

ICMP

ICMP (Internet Control Message Protocol) carries error reports and diagnostics for IP. Each message has a type and a code, and an error quotes the start of the packet that caused it, so the sender can match it to a connection. ICMPv6 does the same for IPv6 and also carries NDP. Messages the lab produced:

MessageType, codeCaused by
Echo request, echo reply8 and 0ping
Time exceeded11, 0A packet’s TTL reached 0 at the router: ping -t 1 10.20.2.20
Destination unreachable, port unreachable3, 3A UDP datagram to port 9999 on server, where nothing listened
Destination unreachable, fragmentation needed3, 4A 1,500-byte packet that mustn’t be fragmented, for a link with an MTU of 1,400
  • TTL (time to live; hop limit in IPv6) is decremented by each router, and a packet reaching 0 is dropped. Replies from server had TTL 63, not 64: one router on the way.
  • traceroute sends packets with TTL 1, 2, 3, and so on, and lists who returns each “time exceeded”. It showed 10.20.1.254, then 10.20.2.20.
  • Path MTU discovery: senders set “don’t fragment”, and a router with a smaller next link returns “fragmentation needed” with its MTU instead of forwarding. With the router’s lan2 MTUMTU Maximum transmission unit: the largest payload one frame on a link can carry, 1,500 bytes on standard Ethernet. set to 1,400:

Path MTU discovery. The client sends 1,500 bytes marked don’t fragment to the router, whose lan2 MTU is 1,400. The router answers with ICMP fragmentation needed, MTU 1400. The client sends 1,400 bytes, which the router forwards to the server.

  • The first ping -M do -s 1472 got Frag needed and DF set (mtu = 1400) from 10.20.1.254. The kernel cached the result: ip route get 10.20.2.20 then showed mtu 1400 for about 10 minutes, the next large ping failed locally with Message too large, and a 1,400-byte one went through.
  • A firewall that drops all ICMP breaks this. Large packets then vanish silently, and connections hang after the handshake, a failure called a PMTU black hole.

Routing

A host or router sends each packet by its routing table: a list of prefixes, each with a next hop or a directly attached link. The most specific matching prefix wins, called longest prefix match. The client’s table, with one extra route added for the test:

RouteMeaning
default via 10.20.1.1Anything else goes to Docker’s gateway
10.20.0.0/16 via 10.20.1.1Added for the test
10.20.1.0/24 dev eth0lan1 is attached directly: send frames straight to the host
10.20.2.0/24 via 10.20.1.254lan2 is behind router

ip route get showed the choice for each destination:

DestinationMatching prefixesChosen
10.20.1.11/0, /16, /24 on eth0Direct, on eth0
10.20.2.99/0, /16, /24 via routervia 10.20.1.254
10.20.9.9/0, /16via 10.20.1.1, from the /16
1.1.1.1/0 onlyvia 10.20.1.1, the default route
  • The router forwards packets because IP forwarding is on (net.ipv4.ip_forward=1); a host with it off drops packets not addressed to it.
  • Server needed its own route back, 10.20.1.0/24 via 10.20.2.254. Each direction is routed separately, and the return path can differ.
  • Routing protocols fill tables automatically and react to failures. Inside one organization, interior gateway protocols such as OSPF and IS-IS find shortest paths across its routers.
  • Between organizations, BGP (Border Gateway Protocol) connects autonomous systems (AS), networks under one routing policy, each with a number. A BGP router tells its neighbors which prefixes it can reach and the list of ASes on the way, the AS path. Each AS picks routes by its own policy and business relationships, not by shortest distance. The internet’s global routing table is the result of tens of thousands of ASes exchanging BGP routes.

In a program

Go’s net/netip package parses and compares addresses: netip.ParsePrefix("10.20.1.70/26") gives a prefix, .Masked() gives the network 10.20.1.64/26, .Contains(addr) tests membership, and .Is4(), .IsPrivate(), .IsLinkLocalUnicast(), and .IsMulticast() name the address’s category. net.JoinHostPort writes socket addresses, adding brackets for IPv6, as in [fd00:20:1::10]:8080. A link-local address needs its zone, fe80::…%eth0, to be dialed. net.Interfaces lists the host’s interfaces with their MAC addresses and MTUs, and Addrs lists each one’s addresses. The kernel does ARP, NDP, routing, and path MTU discovery; a program only sees their results, such as an EMSGSIZE error for a datagram larger than the path MTU. Networking basics covers the layers these sit in.

References: RFC 791: Internet Protocol, RFC 4632: CIDR, RFC 1918: Private address space, RFC 6890: Special-purpose IP address registries, RFC 6335: Port number procedures, RFC 3022: Traditional NAT, RFC 1112: IP multicast, RFC 826: ARP, RFC 8200: IPv6, RFC 4291: IPv6 addressing architecture, RFC 5952: IPv6 address text representation, RFC 4193: Unique local IPv6 addresses, RFC 4861: Neighbor Discovery, RFC 4862: SLAAC, RFC 7217: Stable private interface IDs, RFC 792: ICMP, RFC 4443: ICMPv6, RFC 1191: Path MTU discovery, RFC 4271: BGP-4, Linux IP sysctl, Go net/netip