Researchers from four international universities found that a large language model outperformed a human participant at building trust with potential scam victims during an experiment examining online scam tactics. The findings indicate AI could eventually automate one of the most difficult and time-consuming elements of sophisticated online fraud.
The study examined the relationship-building phase of “pig butchering” scams, in which criminals spend days or weeks cultivating rapport before introducing fraudulent investment opportunities.
At the center of the research was exploitable trust, a participant’s willingness to comply with seemingly harmless requests that could lead to future exploitation.
During the week-long experiment, 46% of participants agreed to download an application recommended by the AI, compared with 18% who complied after interacting with a human.
Why It Matters: The research indicates cyber risk is entering a new phase as generative AI moves from producing convincing content to maintaining persuasive conversations over extended periods. Most security programs are designed to detect malicious activity such as phishing emails or suspicious links. AI that can build trust over days or weeks introduces a different challenge. The focus shifts toward validating identity and intent throughout routine interactions, especially as organizations expand their use of AI-powered tools.
- Exploitable Trust: The study focused on exploitable trust, measuring how effectively a model builds enough rapport to encourage compliance with a future request instead of measuring whether a scam succeeds. The researchers describe the framework as a way to evaluate AI’s influence during extended conversations before fraud occurs.
- Relationship Scam Automation: “Pig butchering” operations typically require human scammers to maintain hundreds of personalized conversations for weeks or months before attempting financial fraud. Researchers believe AI can handle much of this work, allowing criminal groups to reach far more people while human operators step in only when a victim appears ready for the final stage of the scam.
- AI Safety Gaps: Because the chatbot spent the experiment developing relationships instead of requesting money, its behavior resembled ordinary conversation. Researchers suggest this approach could make AI-enabled fraud harder for safety systems to recognize. Anthropic said the study evaluated an older Claude model and added that newer versions include additional testing intended to reduce deceptive behavior.
- Cybercrime Economics: Automating the early stages of relationship scams could reduce dependence on the large scam compounds that currently employ thousands of workers to cultivate victims. Researchers note this could reduce reliance on trafficked labor while making fraud operations harder for law enforcement to locate and dismantle.
- Enterprise Identity Risks: Although the study focused on consumer fraud, the findings demonstrate AI’s ability to maintain convincing conversations could strengthen social engineering inside organizations. Business email compromise and executive impersonation are two examples. The research did not examine enterprise environments directly, though it raises questions about identity verification and employee security awareness.
Go Deeper -> AI Scammers Are Better at Building Trust Than Humans – WIRED
