Every few years, testing gets rediscovered. I’ve watched this happen more than once. Years ago, when I worked at BZ Media, the old owner of SD Times, we ran a testing conference and magazine, sold them off, sat out a five-year non-compete on the word “testing” itself, and then walked back into a test conference expecting to find a changed industry. Instead we found the same vendors telling the same stories with the same tools. Nothing had moved.
That is not the world we’re in anymore. In Session 3 of our four-part SD Times Live! Supercast on AI in Testing, “The AI-Augmented Tester: Tools, Skills, and Practices,” we talked with Adam Auerbach, Head of Applied AI for North America at EPAM Systems, about what’s actually changing on the ground. Then LeapWork’s VP of Developer Relations, Donovan Brady, walked through why Playwright has become the default browser automation framework, and what happens when AI starts writing the tests but nobody has time to read them.
Here’s what stuck with me.
AI doesn’t change what good testing looks like. It raises the stakes for not having it.
Auerbach has been doing this a long time. Manual tester, automation architect, QA transformation lead at Capital One, and now the person EPAM sends in to help enterprises figure out where AI actually fits in the software delivery life cycle. His first point cut against a lot of the AI-testing hype I’ve been hearing all year: the fundamentals haven’t changed. You still need traceability, quality gates, real test data, and real environments. What’s changed, he said, is the cost of skipping them.
“I constantly will see organizations who have leaned into development using AI, and then they have significant production issues because testing hasn’t kept up,” Auerbach said. Developers are shipping faster. Testing that was already a bottleneck before AI is now the thing standing between “we moved fast” and “we broke something in production.”
Auerbach laid out a maturity model that’s worth internalizing if you’re trying to figure out where to start:
- Level 1, augmented: AI helps write test cases, generate test data, and produce scripts. This is the entry point, and it’s also where most organizations still are.
- Level 2, triage and self-healing: Once you have a suite of automated tests, AI helps triage failures and, in some cases, feed fixes back into the code itself. EPAM’s open-source Report Portal, which now has MCP support, is built for exactly this.
- Level 3, spec-driven and agentic: Here the testing pyramid starts to flip. EPAM’s own no-code tool, AgenticQA, sends agents directly to a website or mobile app to evaluate what they see and make a judgment call, rather than executing a fixed script.
One thing Auerbach was clear about: the debate over whether the same model that writes your code should also test it is the wrong debate. “It’s less about the model and more about how are you giving the model context about your application, your business, the tests that you’ve run in the past, where you have defects,” he said. A prompt with no institutional memory behind it will always miss things, regardless of which model wrote the code.
And on the tooling shift itself, Auerbach didn’t hedge: Selenium was the pitch for a decade and a half. Today, that pitch belongs to Playwright, largely because of how well it works with AI. Which set up the rest of the hour perfectly.
Why Playwright, and why now
Donovan Brady opened with a number that’s hard to argue with: 77 million npm downloads a week, a mostly linear climb since Playwright launched in 2020 that turned exponential in roughly the last year. On GitHub, Playwright now sits at nearly 100,000 stars, ahead of Selenium’s 34,000 and Cypress’s 50,000, despite both having a significant head start.
Brady’s framing was useful: Selenium and Playwright were never really solving the same problem. Selenium answered “how do I control a browser?” Playwright asked “how do I help developers ship with confidence?” Selenium was built for a web made of static HTML pages that changed every few months. It assumed browsers behaved consistently and that synchronization issues could be handled with a well-placed sleep timer. Once single-page apps, asynchronous API calls, and continuous deployment became the norm, those assumptions broke all at once, and testers were left writing what Brady called “spaghetti code” just to guess when a page was actually ready to interact with.
Playwright, built by former Puppeteer engineers with a clean slate, addressed that directly: automatic waiting, direct browser protocol access instead of a WebDriver translation layer, isolated browser contexts so one test’s leftover cart items don’t corrupt the next test’s checkout run, and dramatically better network mocking and debugging tools, including screenshots, video, and time-travel style trace review.
But the real inflection point, Brady argued, is AI. Playwright is still code, and most QA professionals aren’t developers. Once AI got good enough to write robust Playwright scripts on someone’s behalf, that barrier came down, and adoption took off.
The enterprise gap: green doesn’t mean safe
This is where the session got most useful for anyone running a testing organization rather than just writing tests. Brady’s core warning: a passing test tells you a change worked in isolation. It doesn’t tell you it works across your actual enterprise, spanning SAP, Citrix, legacy mainframe systems, Salesforce, and everything else sitting alongside your homegrown applications. “That test that the developer wrote might pass in isolation,” Brady said. “But what does it look like when it’s plugged into the entire web process of all of our applications?”
He also shared something LeapWork found in its own house: when the team ran Playwright with AI generating the tests, coverage numbers looked great, 100% passing, full coverage. Digging in, they found the AI had learned to write tests that passed, not tests that actually validated behavior. Nobody had time to manually review hundreds of AI-generated tests, so nobody caught it. Even Playwright itself recommends human verification of AI-written tests, a recommendation that quietly breaks down the moment AI is generating hundreds of tests a day.
Brady’s answer to that gap is LeapWork’s new product, LeapWork Play, now in early access. AI handles authoring, but the resulting tests run deterministically, with no token cost at runtime, evidence-linked reporting, audit logging, and reusable components teams can share instead of rebuilding tests app by app. It also supports MCP, so it can run alongside AI code generation tools like Claude Code, validating changes as they’re made rather than after the fact.
If you want to see it, you can sign up now at leapwork.ai and get free access with 1,000 tokens before the product goes fully live on September 30th.
Catch up, and save the date
If you missed Session 1 (context, agents, and MCP in distributed testing) or Session 2 (cutting through AI fatigue to focus on outcomes), or if you want to watch this one, Session 3, in full, including the full conversations with Adam Auerbach and Donovan Brady, you can watch the complete Aug. 6 session here.
The fourth and final episode of this Supercast series will air November 5th. If AI is already reshaping how your team writes, triages, and validates tests, register now for the Nov. 5 session to save your spot.
You can also find Adam Auerbach’s ongoing writing and commentary on applied AI in software delivery on LinkedIn.
