The Proactive Culture of PNM
By Jason Rupe
Picking up the pace: PNM DevOps
The proactive network maintenance (PNM) community has a culture of being meta-proactive by quickly improving itself. Take for example how the PNM working group (PNM-WG) at CableLabs and SCTE’s Network Operations Subcommittee Working Group 7 (NOS WG7) on PNM have been coordinating for a few years now. But this year, that was no longer enough; as Larry Wolcott, the chair for NOS WG7 says: “With the CableLabs and SCTE merger, we’ve been able to streamline the PNM working groups with DevOps to accelerate adoption in the field. Bringing together the art and science sooner in the process can dramatically reduce the time for cable operators to realize the benefits of PNM.” Full stop mic drop!
Lucky me, as lead for the CableLabs side of this effort, I can tell you how the industry has and will continue to benefit from this new purpose-driven synergy. A path is laid in steps, so let me take you along on the journey to demonstrate how PNM data translates to efficient, impactful action on network, services, and customer experiences. Get ready to act on this opportunity, too!
Identifying the problem
We have been able to identify the types of impairments in spectrum data and receive modulation error ratio (RxMER) per subcarrier data for some time. PNM data tells us the service is impacted, and how severely, so we can tell if the customer is impacted too. But only a trained expert can really translate what the data show into the type of impairment, and then into what it says about the network condition. While we have had some tools that translate the data patterns into impairment types for a while, that isn’t enough. Even an educated guess that a cluster of customers is impacted is not enough. It isn’t sufficient to put a red x on the map. Somebody has to fix the problem, and they must be well prepared to fix fast and fully.
Now we are improving our capabilities by applying machine learning (ML) techniques to create new anomaly detection (AD) methods. Brady Volpe just reported in Broadband Library’s Summer 2021 issue how machine learning has come to PNM [1]. CableLabs has had its own version to share with the industry, reported in an IEEE paper [2]. Early results are promising: We can find more anomalies, select the right ones to be addressed, and overall be more efficient with our experts’ time as a result.
Some of these impairments are difficult to identify: a wave is not just a wave. For example, the PNM community has developed several methods for identifying the difference between a standing wave (aka amplitude ripple) and a water wave, the latter occurring when water enters the coax. Some of these methods are reported in an SCTE Cable-Tec Expo paper from this year [3]. A standing wave, occurring in an echo cavity caused by impedance mismatches at discrete points in the network, looks repetitive like the spectrum plot at the top of Figure 1. But a water wave, formed by water intrusion into the coax, looks more random and often attenuates more toward higher frequencies, like the spectrum plot at the bottom of Figure 1. But also, sometimes a resonant peak or a suckout can look a lot like a standing wave appearing in just a part of the spectrum plot. All these different impairment types are caused by different failure modes in the outside plant. So, depending on the impairment type, and how many customers are experiencing the same problem, the technician will be looking for different evidence at different locations in the network. Different types of problems might also impact the cost of the repair, which technician gets assigned to fix the problem, and how urgent it is to fix it.
Thus, it is important that we keep building better tools that help us get better at identifying the problem before deciding to roll a truck.
Figure 1. Top: A plot of full band capture data from a cable modem indicating a standing wave in the network. Bottom: A plot of full band capture data from a cable modem indicating a water wave in a subscriber drop.
Translating impairments into action
The best cable technicians and experts know how to look at an impairment and know where in the network to go first to find and fix it. This step can be costly if not done efficiently. Sharing this expertise has been an ongoing challenge for PNM, but the industry is solving it.
CableLabs created a data labeling tool allowing experts to label impairments in spectrum and RxMER per subcarrier data, among other types of data. Figure 2 shows the version we’re using today to label impairments in spectrum data, for use in training AD solutions, engineers, and technicians alike.
Figure 2. The data labeling tool developed by CableLabs for use by the cable industry.
Connecting the Dots: Data to Operations to Repair
CableLabs built ProOps, available to the community, for turning network data (PNM and otherwise) into action (proactively, if you will) [4]. The community just mapped some of the needed paths, as will be revealed in an SCTE Operational Practice soon, and we plan to map many more paths with our work on a forthcoming repair matrix project.
The repair matrix will link clearly and confidently the PNM data indicators to the plant conditions that are indicated, to the impact to the customer, to the action that is needed to fix it, including the most likely locations to search. See Figure 3 for a small example.
If you are familiar with failure modes and effects analysis (FMEA) or failure modes, effects, and criticality analysis (FMECA), you will know what we’re doing. By connecting the knowledge in this proven way, we can quickly share the knowledge and apply it to further improving PNM.
Figure 3. An example part of the planned repair matrix.
Picturing a more proactive future
With a DevOps cycle approach to PNM, and a repair matrix map for where we are going, the PNM community is moving fast toward an efficient future for cable. By applying new concepts and technologies to our operations problems, we are all working toward improving network access reliability, enabling cost effective operations in HFC, and blazing the path toward network convergence and 10G.
Since you are a member of this community, we look forward to connecting the dots with you!
References
[1] Brady Volpe, “Machine Learning and PNM: Yes, it’s real and yes, it’s deployed and working!,” Broadband Library, Summer 2021.
[2] Jingjie Zhu, Karthik Sundaresan, Jason Rupe, “Proactive Network Maintenance using Fast, Accurate Anomaly Localization and Classification on 1-D Data Series,” 2020 IEEE International Conference on Prognostics and Health Management (ICPHM), 2020.
[3] Kathy Fox, Jason Rupe, Tom Williams, Jay Zhu, Ron Hranac, Nathan Zedan, James Kolcun, Larry Wolcott, “Water Can Run, But It Can’t Hide: PNM Finds Soaked Cables,” SCTE Cable-Tec Expo 2021 Fall Technical Forum, 2021.
[4] Jason Rupe, “Kickstarting Proactive Network Maintenance with the Proactive Operations Platform and Example Application,” SCTE Cable-Tec Expo 2019 Fall Technical Forum, 2019.
Jason W. Rupe, Ph.D.
Principal Architect, CableLabs
Dr. Rupe has held titles including senior technical staff and director at USWEST, Qwest, Polar Star Consulting, and Tenica. He was the Managing Editor of IEEE Transactions on Reliability and co-chair of IEEE Blockchain, and is a VP for the IEEE Reliability Society and associate board member of the Rocky Mountain Chapter of SCTE. He is a CableLabs Principal Architect working on proactive network maintenance and operations convergence.
