“Help! There are too many [LLM bug reports, blog posts about LLM bug reports, books, treatises, codices, scrolls, papyri, cuneiform tablets]! How do I choose which to read?”
—Many people, presumably
Stop there! If you are reading this, ask yourself how you got here. Did Substack’s algorithm recommend this article for you? Did a juicy thumbnail provide a welcome distraction from a mundane task? Maybe you know me personally and feel you have an obligation (you do)? Are you already regretting your decision to click?
The maintainers of many of the most important open source software repositories in the world are “drowning” in bug reports.1 Daniel Stenberg, who runs curl, has documented a rising tide of such reports,2 generated in part by well-meaning users equipped with the latest LLMs. These reports look entirely plausible, and a minority of them actually highlight real vulnerabilities. But most are essentially worthless. Actually, they might be worse than worthless, since the only way to know whether a report reports something real is to do most of the work of validating it by hand. The cost of producing bug reports has diminished, while the cost of validating them has remained constant. Thus, this flood of LLM generated reports diverts expert maintainers who could be spending their time and attention on reports with a higher relative signal.
This is an instructive microcosm of a wider LLM-fueled dynamic. With the ascendance of LLMs, the cost of producing credible–looking work across many domains has plummeted. Recently, I prompted Claude Code to do some research on a relatively advanced idea I was mulling in the AI alignment space (representational similarity analysis over LLaMA activations for prompted deceptive intent detection). It spat out, in LaTeX, a whole paper, complete with data from experiments that it had actually run, p-values, equations, figures, a literature review, and a bibliography (which mostly included real papers). It should come as no surprise then that the submission volume to academic journals has risen 42% since the introduction of ChatGPT, while writing quality has declined.3 Indeed, my paper was pretty bad (no doubt in part because of the quality of the idea I gave to it), but it looked very credible and cost me almost nothing to produce. I think it would have taken a domain expert around 2–3 minutes to work out that it was slop, and quite a bit longer to describe its main flaws in detail.
This time cost will surely rise.
The cost of producing credible-looking papers, credible-looking cover letters, credible-looking code, credible-looking blog posts, credible-looking bug reports, credible-looking mathematical proofs, and credible-looking risk analyses is heading to 0. So the supply will continue to skyrocket.
In essence, we are now great at generating stuff, but much less great at figuring out whether that stuff is actually any good.
I am battling with this problem even as I write this. I use Claude to help me editorialize and think through my ideas—relatively little shame in that. But as I navigate Claude’s outputs, I am spending a lot of my time not really ‘collaborating’ but trying to work out which of the “strengths” of my writing that it has picked out are merely sycophantic rehearsals of my ideas, and which of the “weaknesses” highlight genuine flaws.
Here, I argue that credibility cost collapses have historical precedent. I suggest that when they occur, we tend to invent new sociotechnical gating mechanisms/institutions that help us work out how to allocate our attention. I then talk about what the gating mechanism for credible slop might look like, and what it should avoid.
Hidden gates, cost collapse, and credibility signaling institutions
When things are hard to make, the mere existence of the thing is evidence that someone has invested a great deal of time and money (which hopefully correlates with relevant expertise) into creating it, and thus it is likely credible and worthy of one’s attention. For several centuries before Gutenberg, making one book took a scribe a full year and a herd of animals’ worth of skin to make. Then, you needed a patron in order to buy one, and to read the thing you needed to know Latin.
When books were scarce, nobody took time to wonder whether one was worth their attention. Scarcity was the gate. Of course, a “scarcity gate” does not guarantee credibility—it is an imperfect filter. Furthermore, scarcity often brings with it the politics of access which restricts the ability to participate in the production and dissemination of information. Ideally, a thing would be scarce purely because one requires expert skill and knowledge to produce it—but, as in the book case above, this is often confounded by wealth, social circumstances, or access to education.
But then the cost of producing things decreases. The printing press replaces the scribe; cheap paper replaces vellum; literacy spreads; things start being written in modern rather than ancient languages; computer science becomes the most popular undergraduate degree. The playing field is leveled, and leveled in a powerfully democratic way; socioeconomic barriers to production and consumption of information fall away.
With this newfound abundance, the scarcity gate stops working and so comes the need for new ways to work out what is actually worth our attention. New socio-institutional gates have to be built. The classic example is the journal: For a century and a half after the arrival of Gutenberg’s press there was a major concern among intellectuals at the newfound surplus of available printed-word documents. Conrad Gessner, in 1545, in the preface of his Bibliotheca universalis lamented the “confusing and harmful abundance of books.” Barnaby Rich, a writer and sea captain, grumbled in 1613 that “one of the diseases of this age is the multiplicity of books.” The historian Ann Blair called this the problem of “too much to know,” the sense that there were now more books than anyone could read in a lifetime and no obvious way to tell the worthwhile from the dross (Too Much to Know, 2010).
Later, in the 19th century with the birth of industrialized printing, we got yet more complaints. See the following quote from Schopenhauer on “the immense number of bad books” available at the time:
…these rank weeds of literature, which deprive the wheat of nourishment and choke it. Thus they use up all the time, money, and attention of the public which by right belong to good books and their noble aims, while they themselves are written merely for the purpose of bringing in money or for procuring posts and positions. They are, therefore, not merely useless but positively harmful.4
Back in the 17th century the socio-institutional solution of curated journals emerged to save the day. In the space of two months in 1665, Denis de Sallo launched the Journal des sçavans in Paris and Henry Oldenburg launched the Philosophical Transactions of the Royal Society in London. What made these important was not that they stored knowledge but that someone now stood at the door and decided what got through it. Oldenburg solicited, selected, and vouched for, so that appearing in it was itself a signal. It was no longer costly to write, but it was costly to get one’s writing past Oldenburg and into the journal. Readers of the journal, insofar as they trusted Oldenburg’s judgment, were then confident of the quality of the material to which they were allocating their attention.
This is one type of gate, but we have created many more—we peer review, we certify speakers with degrees, we count how often they cite each other, we invite people whose work we know and/or like to speak at events, we check follower counts, we count how often websites reference each other, etc. We know these proxies are imperfect (see Didier Raoult’s h-index) but we use them because we need some way of deciding who/what to pay attention to.
AI is a truly novel technology in its radical generality, and thus one should certainly take care in reaching for historical analogies. But, insofar as today’s models can be understood as dropping the cost of producing credible looking media, I think it is helpful to think about how we have dealt with such circumstances previously. The appearance of credibility has been severed from real credibility many times, precisely when it is no longer costly to look credible, and (admittedly sometimes after a period of chaos and strife) the response tends to be to build an institution to make that appearance expensive again.
The question then becomes what the next gate(s) might possibly look like. When it costs nothing to produce credible-looking work across most disciplines, what can remain expensive and be charged for that is a satisfactory proxy for something worth our time? I think there are more good bug reports, good blog posts, and good web apps being developed now than ever before, but the issue is that there are also vastly more bad ones—we need a mechanism for telling them apart.
How to not throw the baby out with the bath slop
So what do we do? Previously, proxies were invented to figure out whether something was worth one’s scarce time and attention, prior to consumption.
The digital approach has, thus far, been to use popularity-contest style proxies. PageRank, Google’s original algorithm, used the number of other web pages that point at a given web page to rank their relevancy. Similarly, many of the recommendation algorithms you use daily, from Substack to Amazon, rely heavily on what people are currently viewing, engaging with, and buying. In other words, we allocate people’s attention to things that other people are already attending to. But the logic of these measures, like the ones discussed above, have a perverse feature: They do not really tell us whether something is worth our attention. Instead, they tell us how much attention this thing has already received, and we treat the second as a proxy for the first. Thus, your attention becomes both the input into the mechanism and the output. Whether or not this blog post appears in your feed is a function of how many people have clicked it before, so attention accrues attention, creating a classic winner-take-all type dynamic. Worse, the moment you have a sorting infrastructure whose currency is attention, the platform that owns the infrastructure has the proxy (engagement, ad revenue etc.) as the incentive and not the target (providing content that is worth people’s time). This is a dynamic that Tim O’Reilly, Ilan Strauss, and I have studied before in our work on algorithmic attention rents.5
The point is that AI did not break a working gate. In fact, in some ways, AI has helped; I have talked elsewhere about how ad-free LLMs are currently better search tools than many traditional search engines.6
In the context of credible-looking-slop though, AI is a dam buster. Domains that were previously reliant on human-judgment-based gating such as academic journals, open source software repositories, are getting flooded. And attention-algorithmic digital search and recommendation platforms are sagging under the combination of the slop strain and their own feedback loops. How many distinctly AI-y articles have you clicked on lately on Substack? I clicked into YouTube’s “shorts” on a logged-out computer the other day and was staggered by the unbridled slop it served up. If you, like me, have been forced to engage with LinkedIn’s feed since ChatGPT’s ascendancy late 2022, I offer you my sincerest condolences.
One candidate solution is that we lean harder on the human-centric institutional gates that we already have: reputations, followings, h-indexes, knowing someone who organizes really cool unconferences, etc. This certainly feels like the most likely direction of travel. However, it carries the cost of entrenching incumbents: Your papers only get read if you are at Harvard; your open source contributions only get accepted if you are already well known in the community; your blog posts only get seen if you are featured by someone with a platform. Central to the appeal of cheaper production is the democratization of contribution—if you are smart and have a good idea for an app or for some alignment research, you can get Claude to help you prototype it without having to learn the entire modern internet stack. The issue is that if genuinely good ideas never get seen because the only stuff people think is worth their time comes with a recognizable affiliation, we destroy that democratization. The baby goes out with the slop.
The second obvious candidate solution is to call for more AI. Every gate thus far has been a proxy—scarcity, the credential, the citation, etc.—that doesn’t directly measure the quality of the content. Rather, it measures something easier to capture that, hopefully, correlates with the quality of the content. What a LLM-based gating system seems to offer, for the first time, is a gate that can actually “read” all the content. One could envision a future where we all encode our preferences in personal-reviewer type models, which then actually go through the films, books and journal articles we are selecting from in order to provide personalized, reliable recommendations. The signal, in such a world, comes home to the object and stays cheap.
Unfortunately, this response seems to miss two important points. The first is a turtles-all-the-way-down problem: The gate and the thing it gates are drawn from the same well. The second is a problem of incentives.
A detector built out of frontier model capabilities may always inherit frontier model blind spots. If AI is capable of convincing itself that the slop it’s generating is the baby, then, if they are the same models, it may be enough to convince the reviewer too. Of course, it is not that LLMs can only ever emit credible looking content—they conduct real mathematics,7 write real code, submit real bug reports. But these are currently few of the total cases (the baby) among a lot of false positives. AI will get better, and eventually perhaps all of the bug reports it submits will be real, all of the proofs it generates will be correct, etc. This problem might dissolve as the systems get more intelligent. But we don’t know when/if AI systems will get to this point, and even when/if they do, presumably it will be quite a bit after that point before we trust them with doing all the stuff—building our planes, creating our medications, designing our policies, etc.
The second thing this response misses is incentives: What happens if we have two such super intelligent machines aimed at deceiving each other? Will an employer’s verification AI be able to see through the ruse of the applicant’s application AI? What about a deviant academic, who sets his AI to work writing a paper optimized for receiving citations? Will the journal’s editorial AI’s be able to catch subtle massaging of data or p-hacking?
We have developed truly sci-fi technology for generating content, but our infrastructure for evaluating its outputs, for curating them, and generally for exercising taste at scale has lagged behind. Maybe the answer lies somewhere between the two avenues I’ve suggested thus far. We have LLM reviewers filter the bug reports, perform some diagnostics, before passing to the human maintainers. But even this risks the identification problems I discussed above.
So I don’t have a clean gate idea to sell you on, I wish I did. Maybe ask Claude?
