According to a report from The New York Times‘ Kashmir Hill, Google terminated a San Francisco resident’s account after misidentifying photos he took of his toddler’s groin for their pediatrician as child sexual abuse material.
A dad in San Francisco took photos of his toddler’s groin for the doctor. When his Android backed the photos up to the cloud, Google flagged them as child sexual abuse material. He lost his Google account and was investigated by the police. https://t.co/OO4ipXJPsO
— Kashmir Hill (@kashhill) August 21, 2022
Google has measures in place to vet content uploaded to the Google cloud for content pertaining to child abuse or exploitation, much like Apple does for its iCloud service.
Mark, the unfortunate Google customer in question, had his Android smartphone configured to back up all of his photos and videos to Google.
Two days after he took those photos of his son, he received a notification on his phone informing him that his Google account had been disabled because of “harmful content” that was “a severe violation of Google’s policies and might be illegal.”
While digging into the story, Hill came across another case from Texas that was almost identical. In both instances, the fathers lost over a decade’s worth of data they had backed up to their Google accounts.
What’s more, Google flagging their accounts triggered police investigations into both individuals. Each of them was investigated and cleared of any wrongdoing or suspicion.
“I determined that the incident did not meet the elements of a crime and that no crime occurred,” Nicholas Hillard, the police investigator on Mark’s case, wrote in his report.
However, neither of the victims has been able to get their accounts back despite being cleared by the police.
Mark asked Hillard if he could tell Google he was innocent and help get his account back. “You have to talk to Google,” Hillard told him on a call. “There’s nothing I can do.”
The tech giant has no procedures in place to rectify a false positive. “Google stands by the decisions,” Hill said in a tweet.
Google stands by the decisions. Critics said the company needs to allow for corrections. But parents and pediatricians need to be aware of the risks of taking photos like this. “Don’t,” an expert said. If you have to, “avoid uploading to the cloud and delete them immediately.”
— Kashmir Hill (@kashhill) August 21, 2022
The extent of the police investigations, though, is yet another cause for concern.
Mark was informed in December of last year that search warrants had been served on Google and his internet service provider. The police were given access to everything in Mark’s Google account, including his internet searches, his location history, his messages, and any document, photo, or video he had stored with the company.
Experts believe these may not be the two only cases where Google’s algorithms have falsely identified child sexual abuse material. “There could be tens, hundreds, thousands more of these,” said Jon Callas, a technologist at the Electronic Frontier Foundation.
Want to see more of our stories on Google?
P.S. Want to keep this site truly independent? Support us by buying us a beer, treating us to a coffee, or shopping through Amazon here. Links in this post are affiliate links, so we earn a tiny commission at no charge to you. Thanks for supporting independent Canadian media!
Some will try to compare this to Apple’s attempts at handling CSAM, but the capabilities and implementation are very different.
In these cases with google, google has been obviously processing everyone’s photos looking for anything that looks like inappropriate material and reporting it to police. It’s unclear whether it’s all automatic or if they have people examining flagged images. Presumably, they use some AI to examine all photos in google cloud looking for content that appears to be inappropriate or photos that resemble or could be similar to inappropriate pictures.
In Apple’s case, they were planning two different implementations. First, was only for photos to be uploaded to icloud, the system would take a hash finger print of the image and compare it to the hashed finger print of specific, known to be circulating CSAM images that the police had on record. This system would not look for pictures of kids or any types of pictures at all, but only for fingerprints of specific, known CSAM images. Then, once a threshold of images were flagged, the account would be forwarded to authorities. User photos themselves were not examined, only a hashed finger print of the photos and only those set to be uploaded to the cloud.
Their second implementation, was to allow parents to enable a feature in Messages only on their children’s phones. This system did look for inappropriate pics, but only if the parents enabled it. It would then block the image and notify the parents. It only applied to pics in Messages.
Apple’s system seemed like it would have been a much better balance of trying of trying to target pedos while protecting privacy. Google’s seems well intentioned but also prone to false positives that could ruin people’s lives, with zero privacy safeguards.
Thanks for posting this. You explained it much better than I could have. It’s my understanding that Apple never actually implemented their system for detection after some backlash, correct?
Correct. They postponed or cancelled the CSAM functionality. Not sure about the kids Messages feature.
The backlash was very puzzling. It either was based on ignorance of how it worked and what others, like Google were already doing and could do, or it was based on ulterior motives around systems to protect kids. Apple’s system was designed with safeguards to prevent false positives, exactly like the ones Google is running into and malicious misuse.
Sue Google. This is inappropriate.