J Studios/DigitalVision via Getty ImagesZDNET’s key takeaways
- AI models absorb human biases from training data.
- Customers can be as liable as AI vendors themselves for AI-led mistakes.
- Experts advise knowing your full stack and planning ahead.
Large language models are sold on the promise of efficiency, faster decision-making, and a higher benefit-to-cost ratio compared to headcount expansion and employee onboarding in several scenarios. But what if all that promised efficiency has serious drawbacks — and comes with a secret accountability price tag?
Gartner predicts that AI spending across corporations and enterprises will exceed $2.5 trillion by the end of 2026. At the same time, companies including OpenAI, Replika, Clearview, and Character.ai have been fined millions of US dollars — often for mistakes their models make due to biased training data. And the chain of liability does not end with the AI vendor.
Also: If your AI-generated code becomes faulty, who faces the most liability exposure?
In Moffatt v. Air Canada, the British Columbia Civil Resolution Tribunal held Air Canada fully responsible for damages when the company’s AI chatbot incorrectly told a passenger they were eligible for a refund. Air Canada argued that the chatbot was a third-party integration, but the court nevertheless held the airline liable for deploying it in the first place.
When AI makes the wrong call, who’s held responsible — and how can businesses protect themselves? ZDNET looked into three incidents from different industries and spoke to experts about who they impact most, who deals with the fallout, and what organizations should know.
When model training goes bad
In January, the National Transportation Safety Board (NTSB) began investigating Waymo robotaxis for illegally blowing past a school bus stop sign in Austin, Texas, even when the bus clearly had the stop arms extended and the red lights flashing. When similar incidents were reported in Atlanta, the National Highway Traffic Safety Administration (NHTSA) opened a probe into Waymo’s self-driving algorithms. This led Waymo to recall over 3,000 vehicles while they worked on fixes.
In January, the NTSB launched a separate investigation into Waymo’s algorithms after a fresh violation occurred in the wake of a patch that the company claimed should have solved the issue.
Missy Cummings, Director of the Mason Autonomy and Robotics Center and a former safety advisor for the NHTSA, told ZDNET that a specific change in Waymo’s engineering model caused the string of traffic violations. The company recently switched to the Waymo Foundational Model, an end-to-end (E2E) learning model that imitates real-world driving patterns on top of existing Gemini models. It does not prioritize traffic rules as thoroughly as previous Waymo algorithms. A system trained on real-world driving footage is only as good as the data it’s actually seen, which makes it more likely to absorb reckless and unsafe practices — like blazing past school bus stop signs.
Also: 12 rules of agentic AI for successful enterprise transformation
“E2E models learn from small clips of video data about spatial relationships and actions of entities in the scene. Such models require a huge amount of data to be comprehensive,” Cummings said. Moreover, there’s not enough publicly available research on what amount of data is actually sufficient to ensure safe outcomes.
“These are not isolated cases,” Cummings told ZDNET over email. “They are the predictable outcome of attempts to scale when the technology is being held together with bandaids.”
Her research also references similar issues affecting self-driving vehicles operated by competitors like Cruise and Zoox. Waymo’s proprietary model has also been reproduced under an open-source license so that it can be studied and reused by other self-driving fleets. While the open-source version hasn’t been deployed publicly by another company, it is likely to influence more businesses in the autonomous vehicle industry.
In fact, Cummings told ZDNET that the problem isn’t necessarily confined to the autonomous vehicle industry.
“I am seeing more medical technology cases unfortunately starting to follow the same path,” she noted.
Avoid a systemic failure — before it’s too late
In 2019, Ziad Obermeyer, a professor at UC Berkeley’s School of Public Health, published a study that found Optum’s Impact Pro risk assessment algorithm systematically categorized Black patients as healthier than white patients, even when their medical data reflected the same level of illness. At the time, many hospitals were using the algorithm to determine if a patient should be enrolled in extra care programs. The algorithm’s bias directly impacted whether patients received the care they needed.
Obermeyer noted the algorithm presented “evidence of racial bias” because it used “health costs as a proxy for health needs.” Instead of predicting illness directly, the model predicted healthcare expenses based on a patient’s past medical costs.
Also: These companies are actually upskilling their workers for AI – here’s how they do it
Obermeyer found that among Black patients sharing the same burden of disease, annual medical expenses ran lower than those of white patients by $1,800. However, this was due to the lack of equal access to healthcare, not an indicator of actual medical need. If the algorithm had been working properly, the percentage share of black patients selected for enrollment in the extra care programs would have risen from 17.7% to 46.5%.
This issue wasn’t specific to Optum — it was common among leading healthcare risk scoring algorithms at the time.
“Cost is strongly correlated with illness, easy to measure, and available at enormous scale,” Obermeyer told ZDNET. Moreover, “population health management programs were also intended partly to reduce avoidable spending, so identifying patients likely to incur high costs seemed like an obvious thing to do.”
But Obermeyer noted that responsibility in a healthcare system is always shared and doesn’t belong to any single entity or organization.
“Many hospitals bought the tools,” he said. “Many doctors had the opportunity to overrule the tools but did not. So this is more like a systemic failure.”
Also: 45% of professionals use shadow AI tools – here’s how to manage the risks
Optum’s press relations office did not respond to ZDNET’s requests for comment. But a new rule under Section 1557 of the Affordable Care Act, which went into effect in May 2025, makes the liability chain quite explicit. Where there are allegations of discrimination, it categorically refuses to let hospitals, clinics, private practitioners, and health insurers defer responsibility to the tools or platforms they used to inform patient care.
McDermott Will & Schulte, a law firm specializing in healthcare litigation, recommends that hospitals and clinics regularly audit any tools or algorithms used to aid in their clinical decision-making. It also suggests maintaining a detailed paper trail of these audits to justify that reasonable efforts were made to ensure non-discrimination.
“Physicians should approach AI with the same diligence as any clinical tool,” Deepika Shrivastava, COO of The Doctors Company, said in a 2025 interview on Medical Economics. She also added that doctors should carefully document all AI use with the explicit understanding that it may be scrutinized in court.
Know what’s in your stack — or risk legal exposure
Mobley v. Workday, an ongoing class action lawsuit filed in 2023, alleges that Workday’s AI-based applicant-screening platform disproportionately rejected candidates based on age, disability, and race.
According to Workday’s own internal numbers, the AI platform screened at least 1.1 billion candidates — all of whom could be eligible for compensation depending on how the lawsuit unfolds.
Also: 3 surveys deliver the same uncomfortable truth about adopting agentic AI
Unsurprisingly, Workday maintains that it is not liable for decisions made by employers using its tech.
“Our customers maintain full control of their hiring processes, and our tools are designed with human oversight at their core,” a Workday spokesperson told ZDNET.
But the court finds it more complex than that. Tabitha Weinstein, former executive director of HR for the Maryland Department of Public Safety, explained to ZDNET that a 2025 ruling in the case treats an AI vendor as an extension of the employer that hired it. That means liability does not stop with Workday — creating an uninsured risk for employers who use AI platforms in HR, Weinstein said.
“Ask an HR team which AI tools touch their hiring, when those tools were last checked for adverse impact, and who can overrule the machine. A lot of them can’t answer one of the three,” she noted.
For companies looking to avoid blowback, the fixes aren’t hard to implement. Weinstein said HR leaders should be keenly aware of every AI platform in their stack, read up on a vendor’s own bias audit before deploying (while making their own adverse impact calculations), and always keep human recruiters in the loop with the authority to override these platforms.
She added that the Workday class action has an even greater impact through legal precedent than it does through the sheer number of people affected.
“If a neutral-looking algorithm rejects older or minority applicants at a higher rate, that’s a legal problem on its own,” she said.
What to do if your business is using AI
These three incidents affect different demographics and fall under the jurisdiction of different agencies or authorities, but they all point to the same underlying dilemma.
When a company uses AI to automate parts of its data handling or decision-making, leaders may assume that the vendor is responsible for guaranteeing the safety, privacy, and responsible use of its product. But in many examples of AI missing the mark, businesses themselves have been as exposed to liability as the AI vendor.
Also: 13% of professionals are confident using AI – how top companies solve this skills gap
Vendors don’t set out to build unsafe systems, but experimental technology and procedural gaps lead to serious compliance risks that affect clients and users downstream. If you’re a decision maker at an organization implementing a new AI or LLM-based workflow, you need to fully review and audit the model training and testing data from your preferred vendor (wherever possible), as well as any terms regarding the storage and distribution of your data through their systems.
AI vendors offer generalized assurances that don’t always hold up to legal scrutiny, especially when it comes to federal regulations on discrimination or corporate malfeasance. Ideally, you should have a clear accountability plan in place before you let an AI platform access your systems and workflows, not after an incident has already happened.
