Network Fundamentals Key to Understanding and Solving IP Mysteries
By Patrick Hunter
This is a pretend scenario recounted for dramatic purposes. The IP addresses, devices, and events in question are fictitious and any similarity to real IPs or devices, living or dead, is purely coincidental.
There was an interesting network issue in the IP world that came up the other day, and it got me thinking. Like we discussed in multiple articles in the past, in any IP routing domain, there are very likely many different possible paths from one source to one destination. That’s a pretty intuitive thing to figure out as well, once you’ve been around any sort of IP networks. In fact, one of the biggest strengths of IP networks is the ability for a network to have multiple paths to a destination and the flexibility to determine the best one under different conditions, just like the routing discussion in the Fall 2018 issue of Broadband Library (https://tinyurl.com/sks3hrj). Take a look at Figure 1. It is pretty representative of an asymmetrical routing example. The two hosts on each end of the communication don’t much care about the path the packets take to get where they’re going so long as they get there. But, there are some issues that come with such flexibility that might not be immediately apparent, until you start to see trouble. That’s where our case begins…
One day, in the not too distant past, we discovered that there appeared to be some strange traffic originating from one of our internal routers to very unlikely IP addresses out in the Internet. This discovery was possible because there are lots of great security controls at work in our network to help us monitor unusual communications or trends in network traffic. In this case, we had an Internet firewall that was telling us about this unusual traffic, so we decided to investigate.
Now, it’s important to understand that Internet firewalls perform a number of different functions. Their primary purpose is to guard the internal network from malicious traffic. But, blocking all traffic to and from the Internet wouldn’t really serve our purposes, so certain traffic is allowed to go through. Generally, traffic that originates out on the Internet headed toward our internal networks is blocked automatically (because we don’t want just anyone to be able to talk to our network). But, if traffic originates from within our network headed toward the Internet, that’s a different case. A good example is a user in our network accessing a web page somewhere else. Since we are the ones initiating the conversation, our firewall allows the traffic to go out, but it also performs one other important function; that is to allow the return traffic (the web page response to our original request) to come back in to the internal network. The way it does this is to keep track of all of the internally-originated “conversations” so that it can tell which traffic is genuinely a response to a legitimate request. That ability to track and allow these already-established conversations is the definition of a “stateful” firewall. So, with that in mind, back to our investigation.
Our first clue was that we saw traffic originating from our router in the diagram. The thing that made this so strange was that there was nothing configured on this router telling it to send unsolicited traffic out to the Internet. It made no sense whatsoever. So, we dove in to learn more.
The first place we looked was the configuration for the router. Perhaps someone inadvertently misconfigured the device, although it was a pretty unlikely possibility. Not surprisingly, everything on the router appeared to meet our configuration specifications and security compliance. So, we knew that we’d need to dig in even deeper. Our next step was to “sniff” the traffic as it was on its way to the Internet proper. Perhaps if we could see what type of payload was in the IP packets, we could better determine what was happening. This led us to our second clue. As it turns out, we saw that these packets were in fact Internet Control Message Protocol (ICMP) echo responses. In other words, these packets were responses to pings coming from out on the Internet, not actually traffic initiated by our router! For us, that made no sense because our firewall told us that these packets were the first transmissions in the conversations that the firewall tracks. Also, we already knew that traffic initiated from out on the Internet from strange addresses wouldn’t even be allowed into our network through the firewall. So, we really had a mystery on our hands: How could the firewall think that there was traffic initiated by our router toward the Internet, but the traffic looked like it was really a response to a ping?
The answer would lie in the source and destination IPs in those packets. Upon closer inspection, we noticed that the source IP address of these echo responses was not an internal interface, like we thought. It was an external-facing IP address from our router! So, the mystery gets deeper. Why would traffic sourced from an external interface destined for the Internet come across the internal network and hit our firewall? Why wouldn’t it just go out the external interface? We had to look at ICMP echo requests and responses to figure that out. The rules of ICMP say that when an IP interface receives an echo request, the source and destination IP addresses should be switched and a response created and sent out as in Figure 3. Well, it turns out that was happening correctly here, but something wasn’t quite right.
Looking at Figure 2, we can see that our router on the left-hand side of the diagram actually has two different connections of note. One is its interface connecting to the “internal” network, and the other is the interface connecting to the Internet. This is a bit of an unusual use case in that this router performs some VPN/firewall functions of its own to allow external virtual private network (VPN) connections for us. So, the only purpose for the external-facing connection is to allow those VPN tunnels to establish, not for general Internet connectivity. That was the big break in the case that we needed to find!
Normally, the interface through which the request came would also be the interface from which the reply is sent. But, in the case of our VPN router, it is programmed (like all of our other routers) to send Internet-bound traffic across the internal network to our Internet firewall to be sent out to the remote destination on the Internet. So, the requests coming in from the Internet to that router weren’t being replied to in the same direction. Since the Internet firewall didn’t even know about the original requests, it just saw that our router was for some reason generating those replies, thus demonstrating the strangeness of the traffic and our subsequent investigation.
This particular sleuthing was quite interesting, and we definitely affirmed that network fundamentals are the key to understanding and solving IP mysteries like this one. Being able to know based on a firm understanding of protocols and network device behavior is still the IP detective’s best tool to closing the case and going home happy.
Patrick Hunter — “Hunter”
Director, IT Enterprise Network and Telecom,
Charter Communications
hunter.hunter@charter.com
Hunter has been employed with Charter since 2000 and has held numerous positions, from Installer, System Technician, Technical Operations management, Sales Engineer, and Network Engineer. His responsibilities include providing IP connectivity to all users in Charter’s approximately 4,000 facilities, including executive and regional offices, technical centers, call centers, stores, headends, hubsites, and data centers. Mr. Hunter has served on the SCTE Gateway Chapter Board of Directors since 2005. He spends his spare time mentoring, teaching, and speaking on IP and Ethernet networks as well as careers in the network field.
shutterstock
