A new type of cyber security threat, known as “Man-in-the-Prompt,” is raising alarms within the AI and cyber security communities. This new attack vector specifically targets interactions with major generative AI tools such as ChatGPT, Gemini, Copilot, and Claude. The most concerning aspect of this threat is its simplicity; it does not require complex or sophisticated hacking techniques. Instead, it leverages a common vulnerability found in web browsers: the browser extension.
According to research from LayerX, any browser extension, regardless of its permissions, has the potential to access and manipulate the prompts sent to large language models (LLMs). The exploit works by taking advantage of the fact that the text input window for these AI chatbots is a simple HTML field, which is part of the webpage’s Document Object Model (DOM). This accessibility means a malicious extension can read, alter, or rewrite a user’s request to the AI without the user ever being aware of the compromise. This makes the attack stealthy and difficult to detect.
The attack process is straightforward yet highly effective. It begins when a user opens an AI tool in their browser. A malicious browser extension then intercepts the user’s prompt before it is sent to the server. The extension modifies the prompt, adding hidden instructions for tasks like data exfiltration or prompt injection. The user then receives a response that appears normal, but in the background, sensitive data may have already been stolen or the session compromised. This method has been successfully tested on all major AI tools, including those from OpenAI, Google, Microsoft, and Anthropic.
The risks associated with the Man-in-the-Prompt attack are substantial, particularly for corporate environments.
If employees use AI tools to process confidential information like financial data, source code, or internal reports, an attacker can use modified prompts to read or extract this sensitive data. Furthermore, the attack can be used to manipulate the AI’s behavior, leading to altered or malicious responses. Crucially, this attack bypasses traditional security measures like firewalls and data loss prevention systems because the prompt modification occurs on the user’s browser, before the request even reaches the server.
Given that an estimated 99% of business users have at least one browser extension installed, the potential for widespread risk exposure is extremely high. The threat highlights a critical weakness in the current security landscape of AI tool usage, where the focus has been on securing the server-side rather than the client-side interaction. As AI tools become more integrated into daily business operations, addressing this vulnerability becomes paramount to preventing data breaches and maintaining the integrity of AI-powered workflows.
Reference:
