Challenges of Managing IP Networks
By Patrick Hunter
On the heels of another great Cable-Tec Expo and an emphasis toward the topic of operations, I thought I’d contribute to the conversation by talking about some of the operational commitments and challenges that are a part of managing a modern IP network. As we have all painstakingly learned, networks continue to be living, breathing manifestations of our desire to create resilient and robust connectivity from a client or subscriber to pretty much everywhere in the world that we can reach. Our mandate is not only to build these networks, but ensure that they continue to operate with the highest level of availability possible. In order to find success in these tasks, we need to find ways to meet our goals while maintaining operational and budgetary efficiency.
The first task that requires our attention is the implementation of our networks. They haven’t always been in existence and a specific need brought them to life. The genesis of this need is one or more specific requirements. Gathering all of the requirements with the goal of designing a network is not the simplest of tasks, and finding out what the customer or client wants is very often a significant exercise. We have taken our lumps when we have made assumptions about “what’s best” for the user, and in some cases have made multi-million dollar mistakes as we followed our “expert” assumptions. In order to truly build the “best” network, an engineering team is required to understand not just what questions to ask of the customer, but how to ask those questions. As it turns out, while many of our users aren’t professional network engineers, they’re also not imbeciles incapable of explaining what they want their connectivity to do or how they want it to perform. So long as we listen to what is said and offer our guidance as trusted advisors to our customer, we stand a good chance of building what is really needed.
In the case of our service provider networks, we don’t simply poll our legions of subscribers, but rather listen to market studies and analyses delivered from our marketing and product departments. They really exist from our perspective to shed light on what the end users want today and what they might want tomorrow. They are the “customer” to us as we design our networks to maximize our companies’ returns on investment and make sure to deliver a service that brings in and retains scores of customers. For those of us tasked with building our internal, or enterprise networks (recall from the summer, https://tinyurl.com/y5d5x284, and fall, https://tinyurl.com/yycd9qog, 2016 issues of Broadband Library that there is a meaningful difference between these two networks, and it matters), our customers are the various business units inside the company. They may be a billing department, technical operations team, dispatchers, accountants, software developers, and a host of other groups. The functionality and performance desired by those business units matters very much and a sincere requirements gathering conversation must take place in order to deliver what is desired while ensuring that best practices with regard to scalability and security are followed.
So, that takes care of gathering requirements so that we may design and build a network. This is very much what I like to call the “service delivery” aspect of our work. We deliver a specifically-designed service to a customer so they might consume it happily for years to come. But there is certainly more to this story. The second and equally important part of our job is to maintain these networks. This is often the least glamorous part of the job, but is arguably the most critical to our long-term success. Referring back to the spring 2019 issue, https://tinyurl.com/y4rjftat, one might recall that IP networks need a little TLC, just like our outside plant networks. They require 24×7 monitoring and collection of telemetry data so that our teams are constantly in touch with the health of the operation. Like any other piece of computer hardware, the routers, switches, firewalls, and load-balancing appliances require software that needs to be updated in order to keep pace with the rapidly-changing environment. The physical appliances themselves need to be kept as clean, dry, and dust-free as possible in order to operate reliably for as long as they are able. Keeping up with all of the monitoring, updates, and physical care is almost always a full time job and when we let years pass with the budget cuts that take away the ability to care and feed our networks, we always end up paying in the long run, either in equipment refresh costs, lost customers due to poor network performance, or both. We all know that it’s a matter of when the equipment will fail, not if it will fail.
The third critical part of operating a network is to triage and troubleshoot when network impairments or outages come. And, boy, do they come. Making sure we reduce the likelihood is important, but so is ensuring that we have the sharpest engineers available to do the troubleshooting in order to minimize our mean time to repair (MTTR).
This brings me to the lesson that I’ve been able to learn in the last few years of network management. Balancing the needs of the business with the costs of building and maintaining the right networks is a delicate art. The greatest success can be had when the right people are put in the right positions to build and maintain the networks. To suggest that it is summed up by making sure to hire the best engineers is a gross oversimplification of the concept. Indeed, what I am saying is that one must put the most experienced and seasoned engineers on the tasks that require the most experience and knowledge. Having a certified network expert with decades of experience push the newest version of code on network devices thousands of times a year is by far not the best way to manage a network operation. Maximizing the time spent by the top tier engineers on tasks that really need an extra level of scrutiny and judgment is critically important.
In the context of service delivery and network maintenance, each “project” should be managed in a manner which only includes effort by the top engineers on the tasks that merit that level of knowledge and experience. For an engineer with the highest credentials, having that person handle every aspect of creating a design, bill of materials, quoting (don’t forget the costs), and implementation of the network solution is not the most effective operational model. Allowing the best and brightest to focus on reviewing the design and verifying the completed solution after implementation is the most efficient manner in which to operate. This is very different from the way many network service delivery projects are handled.
Having the newer, fresher faces focus on the very repeatable, low level, low risk tasks allows us to most cost-effectively get the work of maintenance or physical device implementation (hanging switches at a specific location, for example) completed in a manner that scales to any size network efficiently. Accomplishing that is key to network operational efficiency.
Keeping our eyes on the goal of building and maintaining the most advanced and reliable networks is critically important. It is one of our greatest differentiators over our competition. Being able to accomplish that goal by the most cost-effective means requires that we think about how the members of our teams are spending their time day after day so that we maximize their potential while minimizing unnecessary waste or inefficiency.
Patrick Hunter — “Hunter”
Director, IT Enterprise Network and Telecom,
Charter Communications
hunter.hunter@charter.com
Hunter has been employed with Charter since 2000 and has held numerous positions, from Installer, System Technician, Technical Operations management, Sales Engineer, and Network Engineer. His responsibilities include providing IP connectivity to all users in Charter’s approximately 4,000 facilities, including executive and regional offices, technical centers, call centers, stores, headends, hubsites, and data centers. Mr. Hunter has served on the SCTE Gateway Chapter Board of Directors since 2005. He spends his spare time mentoring, teaching, and speaking on IP and Ethernet networks as well as careers in the network field.
shutterstock
