Last week discussed how Ethernet CRCs don't cover what we think they cover. Surely the TCP checksum, as an end to end mechanism, provides at least a modicum of protection?
Unfortunately in today's Internet, no it doesn't. The TCP checksum is no longer end to end.
Our mental model has the client sending packets all the way to the server at the other end. In reality there is a huge variety of gear which spoofs the TCP protocol or interposes itself in the connection, and the checksum is routinely discarded and regenerated before making it to the server. We'll look at some examples.
Load Balancers
In a typical datacenter, the servers sit behind a load balancer. The simplest such equipment distributes sessions without modifying the packets, but the market demands more sophisticated features like:
- SSL offload, allowing the servers to handle unencrypted sessions
- Optimized TCP Window for clients on broadband, dialup, or wireless.
- Allow jumbo frames within the data center, and normal sized frames outside.
All of these features require the load balancer to be the endpoint of the TCP session from the client, and initiate an entirely separate TCP connection to the server. The two connections are linked in that when one side is slow the other will be flow controlled, but are otherwise independent. The TCP checksum inserted by the client is verified by the load balancer, then discarded. It doesn't go all the way to the server it is communicating with.
WAN Optimization
High speed wide area networks are expensive. If a one-time purchase of a magic box at each end can reduce the monthly cost of the connection between them, then there is an economic benefit to buying the magic box. WAN optimization products reduce the amount of traffic over the WAN using a number of techniques, most notably compression and deduplication.
The box watches the traffic sent by clients and modifies the data sent across the WAN. At the other end it restores the data to what was originally sent. In some modes it will carry the TCP checksums across the WAN to re-insert into the reconstructed packets. However the gear typically also offers features to tune TCP performance and window behavior (especially for satellite links), and these result in modifying the packets with calculation of new checksums.
NAT
NAT involves overwriting the IP addresses and TCP/UDP port numbers, which impacts the TCP checksum. However NAT is somewhat special: it is purely overwriting data, not expanding it. Therefore it can keep the existing checksum from the packet, subtract the old value from it, and add the new. As this is less expensive than to recalculate the checksum afresh, most NAT implementations do so.
Thus though the checksum is modified, most NAT implementations don't have the potential to corrupt data and then calculate a fresh checksum over the corruption. Yay!
HTTP Proxies
Organizations set up HTTP proxies to enforce security policies, cache content, and a host of other reasons. Transparent HTTP proxies are setup without cooperation from the client machine, simply grabbing HTTP sessions which flow by on the network. For the web to work the proxy has to modify the data sent by the client, if only to add an X-Forwarded-For header. Because the proxy expands the amount of data sent, it ends up repacking data in frames. Therefore proxies generally calculate a new checksum, they can't just update it the way NAT boxes do.
Conclusion
The Internet has evolved considerably since TCP was invented, and our mental model no longer matches reality. Many connections end up repeatedly recalculating their checksum via a proxy at their own site, a WAN optimizer somewhere along the way, and a load balancer at the far end. The checksum has essentially been turned into a hop by hop mechanism.
This isn't necessarily bad, and it has allowed the Internet to expand to its current reach. The point of this post and the earlier musing is simply that the network cannot be depended on to ensure the integrity of data. End to end protection means including integrity checks within the payload.
footnote: this blog contains articles on a range of topics. If you want more posts like this, I suggest the Ethernet label.