13
As AI becomes the public face of business, organizations must validate performance, security, and cost efficiency at scale. Comprehensive testing under realistic workloads is essential to ensure reliable, secure, and economically sustainable customer-facing AI systems.
Generative AI chatbots, recommendation engines, and agents are rapidly becoming the face of the enterprise, taking on the frontline duty of answering pre-sales questions, handling support queries, and helping get users up to speed on products. This is freeing up staff, who can step in only when necessary to tackle more difficult situations.
As businesses embrace this shift, they must ask a critical question: are these systems truly ready for the demands of customer service?
The answer is readiness, and readiness must be validated through comprehensive, at-scale testing across three interconnected dimensions: performance, security, and cost-efficiency. Performance encompasses not just the speed of responses and actions, but their suitability for the tasks entrusted to them, with answers and responses needing to be consistently accurate.
Front-facing AI will need to manage thousands of complex and nuanced interactions every second. It must be fast, reliable, and secure. For conversational AI applications in particular, “fast” has a precise meaning. Users are sensitive to two latency metrics in particular: Time to First Token (TTFT), the delay before the model begins streaming a response, and Time to Last Token (TTLT), the total time to response completion.
These degrade non-linearly under load. A system delivering sub-second TTFT at low concurrency can exhibit multi-second delays when hundreds of parallel sessions compete for GPU compute and KV-cache memory. Testing must measure these metrics across percentile distributions, median performance alone is misleading; P95 and P99 tail latency is what users actually experience during peak usage.
Higher accuracy often comes with the use of larger, more resource-intensive models. This is especially true for applications that need to analyze large amounts of data or need to be useful over long conversations, making the LLM context window a second critical factor. As context-window size increases, so does the cost per token, and this matters even more as AI inference applications scale.
AI inference typically accounts for 80–90% of total compute spend over a model’s production lifecycle, making right-sizing decisions consequential for both economics and user experience. Rigorous load testing under realistic prompt profiles is the only reliable mechanism for identifying where resources are genuinely needed and where they are wasted.
Optimizing performance and cost
There are many techniques that can reduce computational overhead but still maintain broadly the same level of response quality. One possibility is to explore performance on the target application using different off-the-shelf models to see if smaller variants work. If custom training is possible, distillation of a well-known model that uses prompts focused on the target application can deliver a more optimized implementation with a much lower cost per token.
There are many hardware infrastructure choices that will further determine performance and effective cost. Organizations will need to not just right-size the AI capability in terms of the number of nodes needed to handle expected demand. They will also need to decide on the best memory and mass-storage organization for the workload. For example, an input prompt that contains a large number of tokens will stress the memory in different ways to prompts that are much shorter, but which need to access external data for retrieval augmented generation.
Choices are not just architectural as there is a security element as well. Implementers need to anticipate the methods by which cybercriminals might launch attacks on core business systems through these agents. Security researchers have demonstrated a range of attacks that can subvert the large language models (LLMs) that power generative AI. These techniques enable the extraction of sensitive data, taking control of core IT systems, or simply degrading service levels. Failure to meet these expectations will lead to the same negative outcomes as poor human service: lost revenue, damaged trust, and compliance risks.
Considering common attack vectors
Among the common attack vectors is prompt injection. This technique works by hiding malicious instructions in seemingly benign queries causing the model to ignore rules set during finet-uning and alignment. The attack is analogous to the much older technique of SQL injection, in which commands to a backend database are embedded in messages processed by front-end software. Though the instructions for prompt injection use natural language, hackers often use changes in language or the use of special characters to trigger an unwanted response.
Another common form of attack is to overflow the context window. This can push the LLM into a mode where it ignores its safety rules. At this point, an attacker may find it easier to gain access to internal data and controls. More subtle approaches use sequences of queries designed to capture data on the LLM’s training and the neuron weights it has stored. This information may be used in a subsequent intrusion attempt or simply used to steal intellectual property (IP).
At the other end of the scale, attackers may simply try to flood the system with requests. This form of distributed denial of service (DDoS) may be used to simply crowd out legitimate users, making the system unusable, but it can also be used to disguise targeted attacks on the underlying systems.
Testing implementations and guardrails
Firewalls and guardrail controls act as the defenses against these attacks. When configured correctly, they can block most attack attempts, but “correctly configured” is harder than it sounds. A guardrail enforcing token quotas, for example, can detect a single actor issuing thousands of rapid requests and throttle them while legitimate users on the same infrastructure continue uninterrupted. But that same guardrail, if its thresholds are set too aggressively, will generate false positives, blocking real customer queries during a traffic surge and creating the very service degradation it was designed to prevent. Monitoring systems can flag anomalies in real time so that teams can refine rules as new patterns emerge. The critical point is that these defenses must be validated, not assumed, through testing that runs adversarial and legitimate traffic simultaneously at production scale.
Validating that the LLM implementation and its defenses are production-ready, requires at-scale testing that exercises all three dimensions, performance, security, and inference infrastructure efficiency simultaneously. On the performance side, testing must measure TTFT and TTLT across concurrency levels that reflect real-world peak load, not controlled lab conditions. Bottlenecks that are invisible at low concurrency – such as GPU saturation, KV-cache pressure, scheduler queuing – emerge only when hundreds or thousands of sessions run in parallel, and the resulting latency spikes can violate service-level commitments that looked comfortably achievable in development.
On the security side, guardrail validation must run adversarial traffic simultaneously with legitimate traffic: the goal is to confirm that security controls hold without introducing latency penalties or false-positives that block real users. On the cost side, testing under realistic prompt profiles reveals where GPU memory, compute, and context-window allocation can be safely reduced without degrading the user experience – information that cannot be derived from generic benchmarks or small-scale pilots.
Test systems for AI inference at scale
Scale testing places a focus on the scalability of the hardware and software used to implement the tests. It is not enough to distribute just test packets. They need to emulate prompts and multi-turn conversations that reflect real-world usage and known attack scenarios. One approach is to use open source or script real-user interactions, which require dedicated expertise and expense to orchestrate, maintain, and scale.
The solution is to employ sophisticated test systems designed specifically for the purpose of AI inference testing. Enabling organizations to emulate hundreds or thousands of concurrent user sessions, operating at the prompt frequency and realistic multi-turn, multi-modal under interactions needed to stress-test the system.
Such a test harness should be designed to ease the input of custom prompts at scale to exercise the specific AI applications and their guardrails, which generic benchmarking prompts will miss. The ability to customize multi-turn conversations, vary prompt lengths, configure context depths, and diverse prompt profiles is critical to identifying and optimizing bottlenecks. This, combined with dynamic modeling of massive concurrency and bursty, spiky traffic patterns (not just steady-state ramps), is critical because production AI traffic is inherently unpredictable.
Token rates and response latencies
Finally, the resulting analysis should show how token rates and response latencies change under concurrency spikes and other variables like context depth. Also correlating this analysis with the AI inference infrastructure metrics show the pressure on GPU and memory as well as caches and queues, to reveal important bottlenecks.
With AI, attacks and use-cases can change rapidly. This evolution is matched by frequent advances in LLM capabilities. Swapping out an old model for a more up-to-date replacement can improve both response quality and efficiency. That is why it is becoming equally important to continue testing regularly. Utilizing test automation to integrate into a continuous validation pipeline and modern DevOps workflow, ensures testing is an integral part of the development and deployment lifecycle.
Continuous testing practice also allows for benchmarking of user experience (latencies), responses to spot potential problems emerging that might be caused by factors such as distributional shifts and cost (token usage). It can also identify further room for AI inference infrastructure (compute, memory, security and networking) and configuration optimization. For example, A/B testing of different strategies for subsystems can be highly advantageous.
Such tests might focus on key-value cache size and organization. Others may evaluate mixture-of-experts models against unified LLMs. A test infrastructure that stores all responses and test results for easy analysis makes comparisons easier to benchmark against earlier implementation choices.
With LLMs becoming the public face of business, organizations need to ensure the technology can withstand not just peaks in usage but sustained cyber-attacks. An integrated and purpose-built solution that allows testing at scale is the way to deliver the assurance needed that these agentic era conversational interfaces and AI applications deliver on the customer experience promise and say nothing out of place.
Sashi Jeyaretnam is the Senior Director of Security Product Management for VIAVI. She has over two decades of experience in networking and cybersecurity technologies and has been instrumental in driving the introduction of market-leading application performance and cybersecurity test solutions for on-premises, cloud, and hybrid networks. Sashi regularly speaks at security events and webinars on the importance of taking a proactive, measured approach to mitigating cybersecurity risks and validating the performance of cloud, distributed security architectures, and AI infrastructure. Prior to VIAVI, Sashi led Product Management at Spirent.
