WhatsApp Previews On-Device 'Scam Alert' That Retains Message Encryption

How do messaging apps stop scams without reading private conversations? WhatsApp says it's developed a way, without sacrificing the platform's end-to-end encryption. 

WhatsApp is now previewing "Scam Alert," a feature that uses a machine-learning model that's smart enough to detect and stop scams, without ever sending messaging data to the app's parent company, Meta. WhatsApp designed the system to counter evolving scams, including impersonation and AI-generated lures that trick unsuspecting users.

Importantly, Scam Alert operates as an on-device model, meaning it runs locally on the user's device. "No message content leaves the device for classification or is auto-reported to WhatsApp, Meta, or anyone else," it says. "The feature complements end-to-end encryption while enabling a user-controlled, optional scam alert when the model believes there's a likely scam."

But for now, WhatsApp is refraining from rolling out the feature widely. Instead, it wants to first stress-test the system through a limited beta in a few regions. WhatsApp is also publishing a technical overview paper about Scam Alert and is inviting the security community to weigh in.

The messaging app has long been known for its end-to-end encryption , which means not even WhatsApp or Meta can decrypt the private conversations; only user devices that hold the encryption keys can. However, the same privacy layer also means WhatsApp can't always detect when scammers exploit the platform to target users, although it has found ways to use warnings on group messages and with device linking .

Trying to build a scam-detection system for private systems can be controversial, though. In 2021, Apple faced backlash for trying to stop child sexual abuse material on iCloud by using consumer iPhones to scan for the uploaded content. The company later abandoned the project. 

In WhatsApp's case, Scam Alert will download a machine learning model to the device. The model then uses text classification to read messages for any signs of scam behavior, without any help from an online server. 

The service adds: "WhatsApp is unable to initiate sharing of any user data without the user's action. The only way message content, or even the fact a scam was detected, reaches our servers is if the user explicitly chooses to report it, which is consistent with how user reporting works on WhatsApp." This builds on the app's existing function to report rule-breaking messages to Meta for review. 

WhatsApp notes: "If the model identifies a message as a likely scam attempt, the user sees a warning in the chat, which is not visible to the other person. From there, the user can decide what to do: block, report, or continue the conversation. If they decide that a warning is incorrectly flagged, the user can mark the chat as trusted, in which case the warning is removed and Scam Alert will not flag that chat again. If a user marks that they trust a chat, they can also opt in to share the last 5 messages received with WhatsApp to help improve the feature's accuracy."

The messaging service also stresses that the user can turn Scam Alert on or off at any time. WhatsApp expects the safeguard to be useful since "the model is trained on patterns observed in scam conversations from reports that users have sent to us. It performs probabilistic classification based on conversational structure and linguistic signals. No content is automatically reported to WhatsApp, Meta, or any third party."

Still, WhatsApp says it needs to collect some anonymized and aggregated "telemetry" from user devices to ensure Scam Alert is working properly. To pull this off, telemetry will be sent to specialized AMD- and Nvidia-powered servers, called Trusted Execution Environments , that also use end-to-end encryption. 

The telemetry includes how many times the on-device model displayed a scam warning, and user action counts, or what's explained as "when a user sees a warning, they can trust the sender or block and report. We log which action category was taken as an aggregate count. This tells us whether users find the warnings accurate, which is essential for measuring false positive rate."

Receiving the telemetry means WhatsApp can only see the approximate counts of "how many warnings were shown and how many were acted on. We will not know what the messages was, who sent or received them, or which conversation triggered a warning."

PCMag and Yahoo may earn commission from links in this article.