New research reveals a growing risk in how large language models respond to user queries involving login information. In a study involving 50 major brands, GPT-4.1 and similar models produced incorrect or potentially dangerous login URLs in more than one-third of cases.

These prompts were simple and natural, reflecting how users already talk to AI systems in real-world settings.

The incorrect links included domains that were unregistered, inactive, or belonged to unrelated businesses. Several could easily be repurposed by attackers for phishing.

In one example, an AI-powered search engine placed a phishing site above the legitimate domain in its response. These errors show how the AI interface itself can become a delivery mechanism for security threats.

Why It Matters: Language models are becoming the default layer through which users access online services. When they confidently present inaccurate or hallucinated results, the consequences can extend far beyond misinformation. Incorrect answers involving authentication or sensitive credentials create opportunities for fraud and deception.

• Incorrect URLs Are Widespread: 131 AI-generated URLs were evaluated in response to login queries for 50 brands. Only 66 percent pointed to the correct domain. Of the rest, 29 percent led to inactive or unregistered domains, and 5 percent directed users to unrelated businesses.

• Phishing Pages Ranked First: A conversational AI search engine provided a phishing link hosted on Google Sites when asked for the Wells Fargo login page. The fake link appeared above the legitimate one. This AI search engine, among similar others, delivers direct answers rather than traditional search rankings, which can obscure usual trust cues like verified domains or SSL certification. This presentation format increases the likelihood that users will engage with unsafe links.

• Smaller Brands Are More Likely to Be Misidentified: Credit unions, regional financial institutions, and mid-tier platforms experienced higher rates of hallucinated responses. These entities often have limited representation in training data, which leads to more frequent guessing by the model. Smaller organizations face greater challenges in combating fraud and recovering from reputational damage when targeted by phishing.

• Phishing Content Targets LLMs: Threat actors are now tailoring phishing pages to appeal to language models rather than search engines. These include fake documentation, login portals, and help pages that are designed to be clean, well-written, and structurally clear. This format increases the chances that they will be surfaced by AI responses, bypassing traditional filters.

• AI Tools Are Contributing to Code Supply Chain Risk: Netcraft also identified a campaign in which attackers distributed a fake API posing as part of the Solana blockchain. The malicious code was promoted through GitHub using fabricated documentation, cloned profiles, and seeded repositories. At least five developers integrated the code into live projects. Some of the repositories appeared to be built using AI-assisted development tools, indicating that malicious content is now entering through model-driven workflows.

Go Deeper -> Large Language Models (LLMs) Are Falling for Phishing Scams: What Happens When AI Gives You the Wrong URL? – Netcraft

Chatbots Are Pointing Millions Of Users to the Wrong Sites and Scammers Are Cashing In – Digital Information World