In May 2019, Melissa Polinsky, director of Apple’s global investigations and child safety team, faced investigators working on the UK’s inquiry into child sexual abuse. During two hours of questioning, Polinsky admitted Apple employed just six people on its global team responsible for investigating child abuse images. Polinsky also said the technology Apple used to scan for existing child abuse images online was “effective.”
Fast-forward two years, and Apple’s work to tackle child sexual abuse material has fallen off the rails. On September 3 the company made a rare public U-turn as it paused plans to introduce a system that looks for known child sexual abuse materials, or CSAM, on the iPhones and iPads of people in the US. “We have decided to take additional time over the coming months to collect input and make improvements before releasing these critically important child safety features,” Apple said in a statement, citing the “feedback” it had received.
So what does Apple do next? It’s unlikely the company can win over or please everyone with what follows—and the fallout from its plans have created an almighty mess. The technical complexities of Apple’s proposals have reduced some public discussions to blunt, for-or-against statements, and explosive language has, in some instances, polarized the debate. The fallout comes as the European Commission prepares child protection legislation that could make it mandatory for technology companies to scan for CSAM.
“The move [for Apple] to do some kind of content review was long overdue,” says Victoria Baines, a cybersecurity expert who has worked at both Facebook and Europol on child safety investigations. Technology companies are required by US law to report any CSAM they find online to the National Center for Missing and Exploited Children (NCMEC), a US nonprofit child-safety organization, but Apple has historically lagged behind its competitors.
In 2020, the NCMEC received 21.7 million CSAM reports, up from 16.9 million in 2019. Facebook topped the 2020 list—making 20.3 million reports last year. Google made 546,704; Dropbox 20,928; Twitter 65,062, Microsoft 96,776; and Snapchat 144,095. Apple made just 265 CSAM reports to NCMEC in 2020.
There are multiple “logical” reasons for the discrepancies, Baines says. Not all technology companies are equal. Facebook, for instance, is built on sharing and connecting with new people. Apple’s main focus is on its hardware, and most people use the company’s services to communicate with people they already know. Or, to put it more bluntly, nobody can search iMessage for children they can send sexually explicit messages to. Another issue at play here is detection. The number of reports a company sends to NCMEC can be based on how much effort it puts into finding CSAM. Better detection tools can also mean more abusive material is found. And some tech companies have done more than others to root out CSAM.
Detecting existing child sexual abuse materials primarily involves scanning what people send, or upload, when that piece of content reaches a company’s servers. Codes, known as hashes, are generated for photos and videos, and are compared with existing hashes for previously identified child sexual abuse material. Hash lists are created by child protection organizations, such as NCMEC and the UK’s Internet Watch Foundation. When a positive match is identified, the technology companies can take action and also report the finding to the NCMEC. Most commonly the process is done through PhotoDNA, which was developed by Microsoft.
Apple’s plan to scan for CSAM uploaded to iCloud flipped this approach on its head and, using some clever cryptography, moved part of the detection onto people’s phones. (Apple has scanned iCloud Mail for CSAM since 2019, but does not scan iCloud Photos or iCloud backups.) The proposal proved controversial for multiple reasons.
For many, having their phone scanned in this way constituted a form of surveillance. More than 90 global privacy groups wrote to Apple to complain about the system, thousands of members of the public signed a petition, and an open letter from security and privacy experts has almost 9,000 names attached to it. Others raised concerns that the system could have been used by governments to search people’s phones for other materials. “There are no assurances in the world that can prevent it from being pointed at other material at a later date,” Baines says. She points at how similar hashing technology has been used for the detection of terror-related materials after political pressure.
“Given the furious reaction from privacy advocates to their plans, one solution Apple should consider is to base their checks on the iCloud side of the equation instead of the device,” says Glen Pounder, the chief operating officer of the nonprofit Child Rescue Coalition. The move would put Apple in a similar position to other technology companies, Pounder says, which proactively scan their servers for people sharing and uploading known CSAM hashes. Pounder says the move could result in Apple making “a few hundred thousand reports” to NCMEC each year.
The amount of CSAM being shared through Apple’s services is a mystery—although the company has seemingly done less than other Big Tech firms. “Child Rescue Coalition thinks it is right to presume the CSAM is there but is not being found and reported—because nobody at Apple is detecting it,” Pounder says. “We can reasonably estimate that on average 1,500 reports every day are being missed deliberately and now by conscious choice—not looking.” Pounder also says that Apple should remove a threshold for reporting images to NCMEC, from 30 images down to one, as a law enforcement investigation could find more abuse happening elsewhere.
However, Apple—which did not respond to a request for comment for this article—may be reluctant to introduce server-side scanning, as some have speculated it planned to introduce scanning on people’s phones so it could eventually make iCloud end-to-end encrypted. “There is a middle ground here which still protects privacy,” Pounder says.
Pausing the rollout of the CSAM scanning tool now potentially gives Apple the chance to alter its proposals—and doesn’t risk the issue overshadowing the expected launch of the iPhone 13 later this month. Johns Hopkins cryptographer Matthew Green tweeted that Apple should now talk to people from technical backgrounds, those who work in policy, and also the public to help shape its future plans.
“Things such as end-to-end encryption, things like scanning your private messages on your own device, it’s complicated. We need to get across to the public that it’s complicated,” Baines says. “There are other things that Apple needs to get up to speed with.” Apple should also look to introduce new reporting tools within iMessage and its other platforms to allow children and others who face abuse or exploitation to report such behavior directly. This would give Apple’s customers more control and also indicate how big its CSAM problem may be, Baines says.
Lourdes M. Turrecha, a privacy law professor at Santa Clara University, adds the company should be more transparent around CSAM it finds. “Apple should also publish transparency reports on its CSAM detection program, including any resulting ‘scope creep’ and surveillance use cases that materialize and the mitigations Apple puts in place to address them,” Turrecha says.
All that said, questions around the scanning of content to detect CSAM—both on people’s devices and on servers—aren’t likely to go away any time soon. Pounder explains there are currently no laws that require companies to actively hunt down CSAM that’s shared on their platforms. “The reality is that the tech companies don’t actually have to do anything at all,” he says.
But Apple’s CSAM debacle has catapulted the issue into the public consciousness at a time when politicians are looking to regulate the internet. “It has raised attention on what’s coming, and what we need to be doing in order to prevent similar initiatives by other actors,” says Diego Naranjo, the head of policy at civil liberties group European Digital Rights. At present, the European Commission is working on legislation that will lay out how companies should fight online child sexual abuse. The legislation, which will be announced on December 1, could include the mandatory scanning of online services by tech companies. Politicians have also hinted their plans could impact end-to-end encryption. “It’s concerning because it threatens privacy, it threatens encryption,” Naranjo says.
But whether this approach complies with the EU’s own data and privacy regulations is up for debate. On July 6 this year, the European Parliament adopted temporary legislation that allows tech companies to automatically scan people’s private messages for CSAM. The law was introduced after changes in privacy laws forced Facebook to stop scanning for abusive content. And even earlier, senior European data protection officials questioned whether scanning would be compatible with people’s right to privacy. “Confidentiality of communications is a cornerstone of the fundamental rights to respect for private and family life,” the European Data Protection Supervisor, an independent body, said in a non-binding opinion. “Even voluntary measures by private companies constitute an interference with these rights.”
Whatever is decided, Apple’s next move will be watched closely. The public debate and outcry are unlikely to be quickly forgotten by onlookers, and the company’s CSAM detection efforts are more open to scrutiny than ever before. “Considering the number of privacy invasions users have learned to live with, the pushback on this line means something,” Green tweeted. “Learn from it.”
This story originally appeared in WIRED UK.
More Great WIRED Stories
- 📩 The latest on tech, science, and more: Get our newsletters!
- Hundreds of ways to get s#!+ done—and we still don’t
- Why I’ll never finish Legend of Zelda: Breath of the Wild
- How the far right exploded on Steam and Discord
- Where to get discounts with your student email address
- Big Tech is bending to the Indian government’s will
- 👁️ Explore AI like never before with our new database
- 🎮 WIRED Games: Get the latest tips, reviews, and more
- ✨ Optimize your home life with our Gear team’s best picks, from robot vacuums to affordable mattresses to smart speakers