So, You Want to Build a Lab…
By Kraig Neese –
Of all the things we do as operators, regardless of how similar or divergent we are, we all have the goal to test our products and devices to meet our needs. The things we test could be a new platform, a new feature, a new way to configure an existing function to enhance reliability or improve performance, or maybe all of the above. DOCSIS® labs are not anything new and have been around since more than a few of us were old enough to work in the cable industry. Technology has changed and flipped more times than we can count, and our labs have evolved to keep up. If they did not, they fell into irrelevancy and provided little value leaving operators open to preventable casualties in operating a DOCSIS network.
Over the years, I have had opportunities ranging from building rolling test rigs to certify newly installed CMTS platforms going into production, to building a full-fledged environment for vetting DOCSIS-based products being prepared for consumer use. For those who have come before and built a lab or inherited a lab or just operated in a lab, the complexity can be overwhelming. One must be a project manager who is also an RF engineer, an installer tech, an amateur electrician, a network engineer, a facility manager, equipment installer, interdepartmental communicator extraordinaire, and about a dozen other roles. The numerous hats that lab administrators must wear require more than a few racks to hang them on. I have been fortunate enough to have incredible mentors and peers by my side to guide me in pointing out the good, the bad, and the ugly.
Building the lab environment
During the big iron days before remote PHY, we had to build out coax to the I-CMTS and M-CMTS units so RF would start and stop with the CMTS itself or from downstream edge QAM modulators. Cable modem banks, arrays, shelves, etc., had to exist with a few hundred modems, MTA devices, and DSG set-top boxes at a minimum to give us operators a chance of mimicking the real world for product testing. Each of those modems needs a dedicated Ethernet connection for traffic generation, some or all MTA devices need phone lines run to a call generator, addressable power outlets for each device to allow for remote power cycling, set-top boxes tied to screens or better yet a remote viewing platform, a provisioning back office, and the know-how of many people to make it all work.
Then to complicate things more, throw in the need to make the labs remotely accessible and controllable so you do not need hands all the time to move patch cables. The use of RF switches can become your best friend or a very complicated enemy if you let things get unruly. To mimic real world issues like ingress or improperly placed carriers on top of other channels, the ability to insert an RF source like a QAM carrier or a noise generator becomes a must. How else can you test the effectiveness of partial-mode or OFDM/A resiliency without designed impairments? Each of these items I mentioned above also comes with its own world of proper documentation, software, networking, IP addressing, and a few thousand other things needed to keep the wheels on track.
Adapting to remote PHY
Just when we had the basics working and I-CMTS/M-CMTS figured out, we got to modify our labs again to accommodate remote PHY. We had to augment our labs to have a slice of DAA setup so the RPD nodes and R-PHY cores could talk to each other. This augmentation came with all the other accoutrements needed for remote PHY like PTP clocks, aggregation layers, and additional provisioning systems, etc. The RPD nodes need fiber and coax run to them with the ability to tie their RF into the cable modem banks, preferably remotely controlled using an RF switch. If you are fortunate enough, you might be able to prune down some of the coax and equipment needed from the I-CMTS/M-CMTS days still needing some support, at least until remote PHY pushes them all out.
Remote PHY added a little more pavement to the automation environment for provisioning RPD nodes and configuring cores, all which comes with the need for a new type of testing. Having the lab environment flexible enough to accommodate this testing can prove to be very beneficial for teams needing the environment to test quickly and effectively. It can also prove to be a sanity pill for the lab administrators getting the request to accommodate such testing. Painting yourself into a corner in the lab is no fun to get out of. Many of us switched to remote PHY and DAA but still used some kind of big iron box as the principal core and maybe additional cores for other services. There is a good chance many of us are at this point in the process.
The shift to vCMTS
However, the days of big, dedicated iron are limited. Over the past few years, we in the industry have been moving closer and closer to the next phase of the DOCSIS evolution, making way for the latest DOCSIS core transition known as virtualized CMTS or “vCMTS.” That also brings us to the next evolution of our lab needs. We still have our legacy equipment because, until it is all out of the production network, we must ensure it functions with new code, firmware, and configurations. Likely breathing the same air as their big iron cousins, the vCMTS servers are getting installed nearby using less space, less power, less cabling, and with more service group density.
The less complex physical installation is a welcome relief but introduces the need to put testing automation into overdrive to allow for zero touch provisioning of vCMTS clusters. There will be new ways to provision RPD nodes to the vCMTS core and additional cores to build services, while new abilities allow us to make changes to existing provisioning elements without ever touching the CLI. The vCMTS brings a whole new operating model to the table with software and hardware being truly decoupled. There will no longer be the swapping of cards and switching over of supervisors. Those complicated tests performed on big iron boxes will be replaced with containers and pod testing.
Collaboration and new roles
New groups will have to come into the fold of testing that may have never been involved with DOCSIS. Server admins might find themselves alongside DOCSIS engineers vetting the same platform. There will be additional equipment or repurposing of existing equipment if the need remains to support QAM-based video on the plant. This evolution feels very familiar, but also somewhat less complex as we get to reuse much of the precious efforts and functions we have built into the labs already.
The lab is never finished, but it is also done at the same time, like the lower floors of a skyscraper being finished before the top floors can stack. Those new upper floors could be furnished with simulated RPDs and modems using applications providing our usual services. The ability to stress test in a lab environment that feels like a fully loaded real world CMTS is more obtainable than in the past. Combined with limited physical nodes and modems, a tester could load up a cluster with tens of thousands of simulated modems and dozens upon dozens of simulated RPD nodes. This level of testing has generally been impractical to perform physically because it requires a vast amount of space, power, and cooling infrastructure.
The future of lab testing
Many of the labs out there are supporting hundreds, if not thousands of physical modems, the ability to run traffic to each modem, the ability to impair modems in the RF domain, the ability to simulate fiber cuts by failing over to longer redundant paths, making phone calls to prove land lines still work, back office systems can interact with the labs to vet tooling, and most importantly create the confidence needed to put a new product or feature in front of our customers.
As more and more vCMTS clusters find their way into more and more labs with their less complex installs, the labs will still be a complicated joy to maintain. The never-ending quest to evolve the lab to the next phase just received a new main quest, and the form this quest takes on will take many of us into retirement. There is a long and bright future for us coax diehards who will be there to shepherd our herds of modems into greener pastures ahead.
Acknowledging the lab’s value
The next time you have a chance, take a moment to appreciate the marvel that is a well-built and functional lab, give a nod to those that have built and maintained them to race to the next cliff of product testing or feature release. Labs can be loved and hated all in one moment, but they are our best companion in our battle against anomalies we all must chase down that come with all new, and sometimes old products our customers consume. Bring on the AI, they tell me that’s the easy button!
Kraig Neese
Cable Access Engineer,
Cox Communications, Inc.
Kraig Neese is a Cable Access Engineer at Cox Communications. He has spent the last 25 years in the cable industry with Cox Communications. He spent his first 17.5 years in the Phoenix market, where he was involved in the various evolutions of DOCSIS platforms from DOCSIS 1.0 to the first launches of 3.0 and 3.1. He joined the Atlanta Access Engineering Team in 2018 and has been focused on supporting remote PHY on an operations and design level.
Images provided by Shutterstock
