Like many online platforms, Discord has safety systems in place to automatically detect and flag abusive, illegal, and otherwise harmful content. This system bugged out in a very odd way over the past few months, leading to thousands of users being incorrectly banned. The situation provides a good chance to consider how online platforms automatically detect harmful images.
As The Verge reports, Discord users have been complaining online about being mistakenly banned for posting totally benign, inoffensive images that include grids in them. This means images of chessboards, game textures, and even things like screenshots of video game inventories.
Per The Verge, around 8,000 Discord users have been banned since May for sharing inoffensive images that Discord’s built-in content moderation tools had flagged. Discord itself said on X, formerly Twitter, that around 200 users had been banned over this past weekend. The company added that all affected users had been reinstated.
“Our systems flag content by matching it against known harmful material,” Discord explains. “This kind of similarity matching can produce false positives, which is why a member of our Trust & Safety team always reviews flagged content before any action is taken.”
However, as Discord continues, and as was obviously the case, this typical system failed. The system is supposed to prevent users from uploading new content while their accounts are being reviewed by a real person, rather than outright banning them.
“We had a bug,” Discord says of the mistaken bans. The company says the bug prevented cleared accounts from being unbanned correctly, and this issue has affected around 8,200 users since May.
“We know that’s not a satisfying explanation if this was your account,” Discord says. “We should have caught this sooner.”
The company is working on changing its safeguards to ensure this type of “quiet” ban cannot happen again and that its safety system doesn’t punish users who don’t violate the platform’s rules.
Perceptual Hashing and its Use in Identifying Images
As mentioned above, many platforms utilize automatic tools designed to detect harmful content, including ones that rely on similarity matching. As Reddit user u/itsFolf writes, flagging systems like this often utilize what’s called “perceptual hashing.”
Perceptual hashing functions typically convert an image to grayscale first, and then downscale it to a much lower resolution, as UK communications regulator Ofcom explains. A system then divides images into sections of squares and then assigns each square a unit value based on its brightness relative to its neighboring squares. While this may sound very simplistic, a given input and a specific hash function will always produce the same output. This means that any known image can be given a unique identifier, allowing an offensive image to be identified infinitely without human intervention.
There are limitations, but hashing has proven generally effective at identifying precise duplicates, which can certainly prevent the spread of harmful content. Ofcom specifically notes that perceptual hashing is among the two most effective means of detecting, deleting, and preventing the spread of CSAM online, alongside metadata analysis. There are multiple hash databases specifically designed to identify and delete CSAM, which platforms can use in their own safety systems.
This technology underpins things like Microsoft’s PhotoDNA, which is used specifically to combat the spread of child sexual abuse material (CSAM) online. It is also used in some technologies that detect instances of copyright violations online, such as on Dropbox. When Apple announced plans to scan iPhone photo libraries in 2021 to search for CSAM, that approach was also based on perceptual hashing.
Image credits: Header image created using an asset licensed via Depositphotos.com. Illustrative perceptual hashing images by Ofcom.
