The Testing Dilemma: Getting it Right vs. Right Now
By David Ririe
In preparing to write this article, I spoke with one of our senior engineering leaders here at Cox to share a few ideas. As part of that conversation, I asked if there was any one thing that was really keeping him up at night. His response was, “The pressure to get all of our technology launches done right, versus right now.”
This statement struck me as a good place to start that could take many paths, so it became my starting premise. There is a saying that many are familiar with — “You can have it good, fast, and/or cheap. Pick two.” Let’s talk about some of those challenges we face in testing our new platforms and devices and how we might launch them both the “right way” and at the “right time.”
Our engineering group here at Cox is responsible for all of the design and testing of both DOCSIS and PON networking platforms. In the last four years, we have launched three completely new platforms (not counting upgrades!), two versions of DOCSIS with up to 32 bonded SC-QAM channels, a residential GPON platform for FTTH, and dragged IPv6 along for the ride!
The fun part is there are no signs of things slowing down. That’s what keeps me up at night. We are preparing to launch OFDM for D3.1, remote-PHY devices, evaluate new xPON platforms, and full duplex DOCSIS sits on the horizon. All of this new work, and much of what we install, must maintain some level of backwards compatibility with our current network technologies. How do we possibly maintain test plans, labs, and QA environments to account for this growth, not to mention our sanity?
We will not be getting any relief on launch timelines for these new platforms and upgrades, so how do we ensure they all work as intended going forward? We must work as an industry, through both our common vendors and standards bodies, to share the burden and hold each other accountable, as the new technologies come to fruition.
We can start with the writing of the specification itself. Whether it is IEEE, SCTE, or a CableLabs specification, there is an old saying that applies: “Don’t complain about the food if you didn’t help make it!” That was a saying in my house growing up, and my kids hear it from me now. In the case of industry standards, this applies whether you are an operator, test equipment vendor, or OEM. Make sure that you provide input and feedback to the specifications, as they are originated and after they are implemented.
Several years ago, I attended a large kickoff for the D3.1 specification in Denver. There was good representation from the equipment vendors and silicon providers, but only two operators showed up. This seemed liked a missed opportunity to me, as participating in the creation of the specification will most likely yield a few things you need down the road.
I like to use an analogy to help other engineering or product teams understand how the network moves forward when we look at DOCSIS-specific upgrades. I describe the whole infrastructure as a three-legged stool. We use the CPE, the CCAP/CMTS platform, and our spectrum plan to represent the three legs of the stool. These must all stand underneath our network to enable our full product set and deliver the results we desire. If I pull out a leg of that stool, or if one is shorter than the others, we have an unstable platform that doesn’t meet our needs or function as expected. That is particularly relevant when it comes to the testing and QA abilities of the various teams in our engineering and operations groups.
We must leverage our network equipment and CPE vendor partners, not only to make sure that they use testbeds that represent all three legs of the stool, but that they can also turn around fixes and features at a sustainable velocity. All too often, our QA and engineering teams are left finding issues that could have been addressed upstream — if only we had gotten alignment in our vendor testbeds. This, of course, needs to be balanced with protection of intellectual property and adequate NDAs, but we must do a better job of sharing the validation burden.
Another area we should examine as operators and vendors is our internal organizations. How do we break down the traditional silos across our engineering, operations, and QA functions? Our network platforms are not only increasing in complexity, but the demands of backwards compatibility and speed to market force us to look across all of our organizations for support. I can no longer test a new piece of CPE in a static lab environment when I know that it must work across multiple platforms and specifications. How do I test a new remote-PHY node without a test plan from multiple organizations, including outside plant teams, access engineers, and a network test for L2 compatibility with a new Ethernet switch? Our traditional tests don’t look so traditional anymore.
As we work with vendors to meet these needs, efforts like the interoperability testing done by CableLabs and others become increasingly important. We have to work across platforms and technologies that have not previously needed to share information and/or physical network boundaries. What lives in the headend today will live in the field tomorrow, and vice versa. We must represent our organizations and companies to help guide those test plans being developed and work towards common interfaces wherever and whenever possible. We must work together on driving common testing standards that can be replicated in both operator and vendor environments.
Looking into the future, we will be addressing new interactions between our test equipment vendors and software automation. We cannot continue to manually test all the pieces and iterations of network equipment without more intelligence in our software and hardware testing platforms. As we move into even more future-state networks that rely on virtualization and common computing platforms, our testing needs will change again. They will need to become more software-focused and standards-driven, requiring more interoperability than ever before.
These are just a few areas we will need to look at as an industry if we are to be successful and timely in our delivery moving forward.
There are plenty of other areas where we need to improve if we are to get better at not only delivering “right now,” but also the “right way.” Don’t let new opportunities to collaborate pass you by, participate in the specification and standards bodies, and work with our equipment vendors to build meaningful test beds and plans. If we, as an industry, don’t step up in these and other areas, we won’t be delivering the “right way” very often.
David Ririe
Sr. Director, Access Engineering
Cox Communications, Inc.
david.ririe@cox.com
David Ririe serves as a Senior Director in the Access Engineering group for Cox Communications in Atlanta, GA. He has worked in various engineering roles for Cox Communications starting as a Network Engineer in the Omaha, NE market 14 years ago. He has lead the team through a number of transitions and upgrades on both the DOCSIS and PON technology platforms in that time. He began his telecom career in the U.S. Air Force working on various communications and data networking platforms in the air and on the ground.
Credit: Shutterstock
