Protecting Our Network Heroes
By Jim Walsh
Keeping customers connected and technicians safe
The COVID-19 pandemic created the perfect storm for broadband providers around the world. Offices and schools were emptied out as people seemingly overnight began working and learning from home. Bandwidth demand, especially upstream, increased dramatically as did the criticality of maintaining a reliable broadband connection. Minor service glitches that would have previously been ignored now disrupt critical Zoom calls prompting a flood of customer complaints. Maintaining cable networks and especially installing and repairing services in customer homes became high-risk activities. Some technicians, especially those in the higher-risk age category, temporarily sat out of the job market until more was understood about the risks increasing the load on less experienced techs and contractors. Strict health regulations turned many established technical operations processes on their head forcing rapid innovation and adaptation of new processes to keep customers connected while keeping everyone safe. Fortunately, the cable industry was well-prepared to deal with this perfect storm through careful bandwidth planning and the agility to innovate and adapt on the fly to a rapidly changing business environment.
It would be impossible to list all of the individual adaptations that cable operators have made to reduce risk while installing and maintaining critical broadband services, but there are a few recurring themes:
- Spend more time up front remotely to minimize time in the field later
- Leverage experienced techs to get the most from the newer ones
- Adapt processes around the new environment
Spend more time up front remotely to minimize time in the field later
Access to customer homes and businesses was cut off or severely restricted in much of the world early in the pandemic. Technicians didn’t want to enter a stranger’s home any more than subscribers wanted an unknown technician entering their homes, so operators increased their focus on minimizing home visits.
One path to minimize home dispatches is the capability to remotely discern whether the cause of a customer complaint is located from the tap down versus in the hardline plant. Pre-pandemic, for many operators the automatic customer call response was to dispatch a tech to their home or business to investigate. This process was fine for cases where the problem lived between customers’ CPE and the tap, but home dispatches for hardline issues result in the worst-case scenario for current times — an unnecessary home visit.
Other consequences of incorrect dispatch decisions:
- Increased exposure risk for tech and customer
- Annoyed customer, tech visited but problem still present
- Often an unnecessary CPE or drop replacement
- Delay fixing real problem — increased MTTR
All of these impacts are multiplied when multiple trucks are rolled to multiple homes impacted by the same plant problem. Operators have been well aware of this problem but the pain that it caused was less than the resources required to address it until the pandemic hit. Most operators have a patchwork of tools and systems that, in the hands of their top techs, are used to triage home versus plant with modest accuracy albeit not in a scalable process. Fast forward to March 2020, and some small to medium operators began pulling a handful of their top technicians from the field and training them to perform this pre-dispatch triage. While it was painful to lose these folks from the field, this was a necessary step to minimize risk by reducing unnecessary home dispatches. Longer term, operators are seeking out automated solutions to free up these experts by providing higher accuracy in home versus plant triage and making it more widely applicable.
Discerning between pipe and application problems has also increased in importance with the rapid rise in VPN usage during the pandemic. While VPN services are typically provided by a subscriber’s employer, broadband service providers are often guilty until proven innocent when VPN problems emerge. Virtual tools to differentiate pipe from application are critical to avoid chasing application issues that a service provider cannot fix.
Leverage experienced techs to get the most from the newer ones
In addition to helping with remote dispatch, senior technicians are also being leveraged to support newer techs in the field via remote instrument access. In one use model, an experienced technician can remotely access a newer technician’s meter from afar to help them understand test results or even remotely control the meter and not require a second visit later to retest or properly repair an issue.
Other typical remote instrument access use models:
- Tech enters home and connects instrument, exits, and drives all testing from outside home minimizing time spent indoors. This is especially common when repairs are required on the home exterior wiring including the drop.
- Leave behind instrument — If measurements are frequently needed from inside of a previously-staffed service provider’s facility, an instrument can be wall-powered and accessed remotely removing the need for a person to enter the vacated facility.
Adapt processes around the new environment
New service installations are the elephant in the room regarding technician and subscriber risk. Different operators have used different mixes of alternative install processes largely depending on local health regulations:
- Self-Install — If the customer has a hot drop, CPE and cables are shipped to the customer with installation instructions. If questions or problems emerge, the customer is encouraged to call customer care. As this model gains in popularity large operators and vendors are developing systems to auto-detect self-installs and poll CCAP/CPE to certify install robustness and create birth certificates.
- Assisted Self-Install — Technician will roll a truck to the customer premises, pressure test the drop and home from outside, and check to ensure that services are good at the ground block. All needed CPE/cables are left on the customer’s doorstep, the tech then remotely walks them through the installation from outside, sometimes using augmented reality applications. This process allows protection against homes/drops being ingress sources that can take out an entire node while still minimizing technician risk.
- Business as Usual — Where local health regulations allow, some operators are sending technicians into customers’ homes to perform installations. Certain aspects of the installation may be abbreviated to reduce risk such as not walking the entire house to ensure Wi-Fi coverage, etc.
Repair processes have adapted to better enable technicians to get in, fix problems right the first time, and get out fast when home entry is required. Scripted testing is one example — a tech simply picks the appropriate work order on their instrument and steps through the specified tests. Closed loop verification assures that all required tests were performed and passed before they leave. Pressure testing has also gained adoption to quickly find and fix shielding weaknesses in the home, often resulting in a 10 minute fix instead of a 60 minute rewire job. It is also viewed as cheap insurance against return visits by detecting hidden issues not currently impacting services but that could create ingress issues later.
We can’t forget about maintenance technicians. While they don’t frequently enter homes, they do have significant exposure when working the outside plant in crowded urban environments. Upstream and downstream PNM tools help them remotely localize issue root causes and minimize time spent outside their trucks finding issues. Similarly, plant leakage technologies are seeing a resurgence beyond typical FCC compliance uses. Leaks are automatically detected and pinpointed as techs drive between jobs, and time in the field finding and fixing these leakage and ingress points is minimized via directional antennas. Leakage tools are also used in conjunction with PNM systems including overlay of leaks on PNM maps to more accurately dispatch and in the field with directional antennas to accelerate find and fix processes.
Where we go from here
The current pandemic will eventually be in our rearview mirror, but the consensus is that life will not revert to exactly what we previously knew. Working from home will continue at least part time for many employees based on statements from many major corporations. Videoconferencing has become a vital part of everyday life for telemedicine, technical support and even social uses meaning the increase in both bandwidth consumption and criticality of services are here to stay. Service providers who adopted remote test tools and processes out of necessity continue to see their value for both risk reduction and cost reduction reasons even as their original drivers fade. This has certainly been a challenging time for broadband service providers, but the lessons learned and actions taken to keep customers connected while protecting our network heroes will continue to pay dividends well into the future.
Jim Walsh,
Solutions Marketing Manager,
Viavi Solutions
Jim has over 25 years experience in telecommunications and semiconductor industries serving in primarily engineering, product line management, and marketing roles. He is currently a Solutions Marketing Manager at Viavi Solutions focusing on HFC and fiber broadband service delivery with previous experience at Intel and Delphi. Jim received both undergraduate and MBA degrees from Purdue University, holds seven U.S. patents, and is a six-sigma black belt.
Shutterstock
