3
In sum, what to know:
–Telecom testing is shifting from point-in-time verification to continuous validation. As networks become increasingly cloud-native, multi-technology and automated, operators are moving toward continuous, end-to-end testing that combines active and passive elements and real-world performance monitoring.
–AI is transforming both networks and the way they are tested. Testing autonomous AI systems requires new approaches, because AI models continuously learn and adapt. But humans have to be able to trust the decisions and actions of AI, which means AI asking permission, showing its work, and AI-driven changes being able to be validated before and after they are made.
–User experience and real-world performance are the ultimate measures of success. Averages and peak speeds can mask local performance issues and statistical outliers that have real impacts on user experience. Increasingly, operators want telecom testing to focus on consistency, reliability and actual customer experience across complex network environments.
Telecom testing: Real-world experience versus averages
The telecom industry’s approach to testing and validation is undergoing a fundamental shift as artificial intelligence, cloud-native architectures and ever-more-complex network environments push the limits of traditional methods of assuring performance.
That was the central theme which emerged from discussions at the recent Test and Measurement Forum, where analysts, operators, vendors and researchers argued that point-in-time verification of networks is becoming less and less reflective of both telecom networks and the user experience. Instead, continuous validation, real-world performance measurements and new approaches to testing AI-driven systems are becoming essential.
Sylwia Kechiche, vice president of industry analysis at Opensignal, identified three forces reshaping telecom testing: Growing network complexity, including the emergence of Non-Terrestrial Networks (NTNs); rising demand from AI-powered applications; and the increasing use of AI inside networks themselves.
Operators are now managing converged environments that are complex webs of software and hardware which draw from multiple technology ecosystems, combine terrestrial and NTNs, and consist of multiple generations and types of wireless, Wi-Fi and wired network technologies. The vendor ecosystems have expanded to include hyperscalers and a variety of software providers. At the same time, AI workloads are changing traffic patterns, placing new demands on latency, uplink performance and coverage.
The result is a test and measurement challenge that extends well beyond traditional performance benchmarks. However, users still experience the network in ways that are highly dependent on their location.
“When we think about connectivity … quite often we think about national averages, coverage, population reach, and so on. But that’s not how users experience a network,” Kechiche said. “We experience it wherever we are.”
Several speakers emphasized that user experience—not peak performance metrics—is becoming the most important measure of network quality.
Michael Thelander, president and founder of Signals Research Group, noted that average measurements often hide a long tail of less-than-ideal network conditions that actually end up having the greatest impact on customers. Latency spikes, for example, can significantly degrade gaming, video conferencing and emerging AI applications—even when average latency appears acceptable.
“You can have a small percentage of data points where the latency is extremely high, and that can impact the user experience, even though your average result may be quite good,” Thelander said.
There is also a persistent gap between laboratory performance and real-world deployments. Monisha Ghosh, professor of electrical engineering at the University of Notre Dame, said studies of deployed 5G networks since the technology first emerged, have repeatedly shown that theoretical capabilities and the telecom industry’s aspirations for a new “G”, do not always translate into practical performance.
While technological maturity and increased familiarity has helped 5G network deployments to improve, much depends on local radio conditions–and ideal radio performance conditions often occur less frequently than models assume. That’s an important factor to take into consideration as the industry pushes forward with 6G planning and design: What actually came to fruition in 5G, and what did not play out as planned?
“As we look forward to 6G, I think it’s really important to understand which of [the 5G promised metrics] were met,” Ghosh said.
AI: A two-fold impact on network testing
A major focus of the forum was the growing role of AI in network operations, and the challenge of validating systems that continuously learn, adapt and change.
Speakers repeatedly drew a distinction between automation and autonomy. Traditional automated systems can be tested using repeatable scripts and predictable outcomes—and repeatability of results has long been a cornerstone of network testing. AI-driven systems, however, should be expected to evolve over time and produce different outputs under similar conditions.
Per Kangru, head of technology at Viavi Solutions, said testing autonomous systems requires a fundamentally different approach.
“AI systems will learn and evolve, and therefore, as we test and evaluate them … we’re not just going to give them the same test every day of the week and every month and every year—instead, we have to evolve that test,” Kangru said.
To address this challenge, industry participants increasingly view digital twins as a critical validation tool. Additionally, one of the biggest areas of debate around AI is, how can trust be built with autonomous AI systems if their decision-making capabilities are so far beyond human-scale? Multiple speakers concluded that AI will only gain widespread acceptance in telecom operations if its recommendations can be validated and explained—which means that testing has to keep up with AI capabilities.
Ross Cassan, senior director for service assurance at Spirent Communications, now part of Keysight Technologies, said operators increasingly want AI systems that can reduce the overwhelming volume of network data and alerts, while still providing transparency into how conclusions are reached.
“We can do a lot of automation. We can kick off workflows. We can, ultimately, find the problem—but if the customer doesn’t trust the result that the system is creating, there’s not a lot of value there,” Cassan said.
Mohamed Nabih, vice senior manager for end-to-end performance and capacity at Rakuten Mobile, echoed that concern, saying operators must balance rapid deployment of AI tools with rigorous validation processes.
“We are trying to balance between the fast deployment of new automation/AI systems with the reality or the reliability of validation, especially from our SMEs,” said Nabih, adding: “We try to make sure that whatever outputs coming from the automation and AI systems we are deploying in operations, are clearly linked to what the SMEs expect and make sure that the results are strong. That can help the operational team to take [a] good, informed decision-making process. Because at the end … if the results are separated from the reality, separated from the team needs, or [have] a lot of hallucination, then the trust of this operational tool drops, and at the end it [does] more harm than good.”
Persistent and new testing challenges
Test and Measurement Forum also highlighted the growing complexity of multi-vendor environments, particularly as Open RAN deployments expand. Paul McKibbin, senior product manager at Calnex Solutions, said interoperability remains one of the industry’s biggest challenges.
“Every new hardware and software combination becomes a new integration challenge, and often that can be a long and a manual process,” McKibbin said.
If there was one consistent hope for AI to make real and practical improvements in telecom networks during the course of the forum discussion, it often centered around the idea that AI may be able to handle the scale and complexity of fine-tuning interoperability within open networks.
Beyond current deployments, speakers urged the industry to begin validating technologies that will underpin future networks, including early identification of potential RF issues and the impact of post-quantum cryptography. Nirlay Kundu, head of technology standards at IMDEA Networks, argued that quantum-safe migration is no longer a distant concern and pointed out a number of regulatory and procurement requirements which will begin to take effect this year.
Meanwhile, a 6G-focused panel stressed that lessons from 5G should shape future wireless technology development. And there are plenty of lessons to be learned from 5G, in terms of the device impacts of adding new, higher bands to cellular networks; in anticipating potential receiver and interference issues from new operations in already-crowded regions of the radio spectrum; and in general, that the more deployment and feature options that a G offers, the more of a challenge and expense that interoperability and testing becomes.
Across the event, one message remained consistent: As telecom networks become more software-driven, distributed and intelligent, testing itself must evolve. The future of network assurance will depend not only on testing performance metrics, but also on validating decisions, behaviors and outcomes—particularly as AI becomes responsible for actively managing increasingly complex systems.
Download the full Test and Measurement Market Pulse report for free from RCRTech. Sessions from Test and Measurement Forum are available for on-demand viewing here.
