LLM Slop Links Ignite a Surge In AI Cyberattacks

Large language model hallucinations are becoming a profitable side business for cybercriminals.

They prod the AI models to answer prompts with non-existent but plausible-looking URLs, registering the domain names that recur often. Then the criminals set up malicious websites on the fake domains.

In June, the Santa Clara, Calif.-based security firm Palo Alto Networks reported that it had prompted LLMs to generate 2.1 million URLs that resemble the names of major brands. Of those, more than 13,000 pointed to domains already registered and confirmed malicious, while roughly 250,000 commonly-hallucinated domain names remained unregistered—sitting available for scammers to claim.

Last year, other security specialists detected a similar threat, from scammers who prompted LLMs to hallucinate plausible URLs for existing software packages. These AI “slop” links took users to counterfeit packages with scammers’ code—ready to be added to a project with a click.

Scammers who engage in this “slop squatting” have plenty to work with: last year, a study by Joseph Spracklen—a PhD student at the University of Texas, San Antonio—and colleagues found that nearly 20 percent of LLM-recommended software packages were fictitious.

Seth Michael Larson, who coined the term “slop squatting” last year, is a security developer-in-residence at the Python Software Foundation, which oversees the programming language that is used worldwide by some 23 million people. Larson spoke with IEEE Spectrum about “slop squatting,” “phantom squatting,” and how security work is changing in response.

Slop Squatting in LLM-Generated URLs

What’s your definition of “slop squatting”?

Seth Michael Larson: It’s targeting a pattern in what an LLM is recommending. For lots of users asking for a particular resource, the model will answer with the same hallucination some percentage of the time. Users implicitly will take what an AI tells them to be authoritative. They’ll assume some security trust decisions have happened. But they haven’t.

What made you decide to call this slop squatting?

Larson: That term betrays my feelings about these systems and their marketing. We’re not giving enough warnings to users that the output is not something you should just blindly accept. People need to understand: The trust boundary between this information and your computer is you.

What percentage of malicious packages are sitting on slop-squatted URLs?

Larson: As people operating a package repository, we don’t map a name back to see if a LLM generated it. We could, but it’s expensive to do, and where it comes from matters less to us than taking it down.

But we can do the correlation. Say a[n erroneous] URL didn’t exist before, and now suddenly, in the past two years, its traffic is huge. If the graph went up two years ago, it’s probably due to LLMs.

Slop squatting is aimed at coders looking for packages. But phantom squatting targets anyone looking for advice about stores, banks, etc. So is phantom squatting a bigger problem for society?

Larson: Probably. Due to the fact that it targets users who are not necessarily expected to have any sort of security knowledge.

The cybersecurity firm Palo Alto Networks recently studied websites at domain names whose URLs are frequently hallucinated by AI chatbots, discovering at some of these sites a range of malware and other malicious code. Source: Palo Alto Networks

Protecting Yourself from AI Phishing

What should AI users do to protect themselves?

Larson: My recommendation with any AI output is: You should not be trusting this. As in any engineering field, you can’t just go off of vibes. You have to have a way to verify. How do we know that this result is correct? Have that in your mind when you’re using these tools.

That said, I don’t look to individual action for this. I think it is more about infrastructure. People can’t hold everything in their head all at once. In medicine, engineering, lots of areas, experts handle the hard cases for people. If you don’t do that, society just doesn’t function.

I try to push people in security work to actively help users. Don’t cross your fingers and just hope users are doing the right thing. You have to set up infrastructure so that users can’t not do the right thing. We can’t expect every single person on the planet to try to keep up with this rate of change. We need to be there to help them through that transition.

So what new security infrastructure needs to be put in place?

Larson: We haven’t been thinking about authentication in relation to LLMs, which make it so cheap and easy to create perfect phishing websites for pennies. We’re just at the very beginning of phishless or phish-resistant authentication technologies. These are things like WebAuthn. Those are promising. But there’s still this long, long tail of adoption ahead.

Right now, even things like password managers are a completely foreign concept to 90 percent of people. It’s still difficult to get people to even use a two-factor authentication method like TOTP [a one-time password] or even a text SMS code. People really don’t like the technology because it makes using digital services harder. So we just haven’t gotten there. Hopefully safety plans will get accelerated now.

How have these new uses of LLMs for phishing changed your work day?

Larson: The work of getting a vulnerability report and triaging it and potentially fixing it and rolling out fixes to everyone—for that, the numbers are three to four times what they were just a year ago. On a medium-sized Python project, you might have processes set up to deal with a vulnerability eight or ten times a year. And now suddenly you’re dealing with 40 or 50 or 60. You have to think, oh boy, we need to fix some of these processes here. It has been a really quick ramp up. Almost every open-source Python project is dealing with this at some scale. It’s a lot of extra pressure.

Evolving Security Processes with LLMs

So, how should security processes change?

Larson: It would be great if security work was more like other sorts of open-source contribution, where basically anyone can participate who is interested and has the ability and the time. Right now this isn’t possible. Security has involved secrecy. But if a publicly-available LLM can discover a vulnerability in an open-source codebase, the value of secrecy is very diminished. It means anyone who has $5 and access to a public model could have found that vulnerability.

So as LLMs make it easy for people to scam, they might also make it easier for people to be a white hat?

Larson: Maybe. This is all very new. I know the Linux kernel is experimenting with this idea, and at Python we’re also thinking about what this means for security teams. Right now in security work you’re in, or you’re out. There’s no in between. There’s no step-by-step where you do the work one time and then decide whether you like it, whether you want to do more of it. Maybe we can give you the first rung on a ladder towards these types of contributions.

Will AI agents make this problem worse? After all, an agent could accept a URL and download a malicious package without its human user even knowing.

Larson: I’ve never used an agent. And the number of times I’ve asked an LLM to generate code for me is probably in the single digits. I’m not exactly a fan. I would definitely say to be careful with agents. I know that that’s not going to jibe with the pace people are expected to deliver code at these days. But you need to always have output that you evaluate. Keep yourself as the driver.

Maybe we can learn some things from other engineering focuses. For example in [rail and airplane] transit, there is the practice of saying out loud the thing you are doing, to keep yourself in the loop. In software now we need practices that keep you engaged, keep you thinking critically about what’s actually happening, and not just clicking the accept button.