Jannik Malte Meissner

When Agents Cross Trust Boundaries: Four Case Studies Every AI Engineer Should Study

Examining four concrete failure modes that have already been exploited or demonstrated in the wild and understanding the mitigation patterns is the fastest way to avoid repeating them in your own systems.

Four sources of untrusted input converging through a trust boundary into a single red output.

If you are building AI agents that read external content, call tools, or act on a developer’s machine, the last eighteen months have opened many people’s eyes to the new reality we now live in. Prompt injection is no longer a theoretical parlour trick, it has become a reliable path to data exfiltration, credential theft and remote code execution in production systems.

The following four incidents, EchoLeak, the Nx/s1ngularity supply-chain attack, a cluster of Model Context Protocol (MCP) abuses, and the Cursor/AWS Kiro sandbox escapes, share a common pattern. An agent was given the ability to process untrusted input and then perform privileged actions. The results were predictable when the boundary between “data” and “instructions” collapsed.

We are now observing concrete failure modes that have already been exploited or demonstrated in the wild. Studying them and understanding the mittigation patterns is the fastest way to avoid repeating them in your own systems.

1. EchoLeak (CVE-2025-32711): Zero-Click Exfiltration via Microsoft 365 Copilot

In early 2025 Aim Labs demonstrated that a single carefully crafted email could force Microsoft 365 Copilot to exfiltrate sensitive organisational data without any user interaction. Microsoft assigned it CVE-2025-32711 (CVSS 9.3) and patched the service server-side in May–June 2025.

How it worked

The attacker sent an ordinary-looking business email containing hidden instructions. The wording deliberately avoided any mention of Copilot or AI so that Microsoft’s Cross-Prompt Injection Attempt (XPIA) classifier would not flag it. When the recipient later asked Copilot a routine question, the RAG pipeline retrieved the malicious email as context. The injected instructions told the model to gather internal data (emails, documents, chat history) and encode it into reference-style Markdown links or images. Copilot’s interface then automatically fetched those resources through a trusted Microsoft Teams proxy, bypassing Content Security Policy controls and delivering the data to the attacker.

The attack succeeded because the system treated retrieved content as both data and instructions, and because several downstream defences (link redaction, image auto-fetch, CSP allow-lists) were lacking.

Lessons for builders

  • Never assume that content retrieved from email, documents or the web is inert. Treat every retrieved token as potentially adversarial.
  • Separate instruction context from data context at the architectural level. Prompt partitioning and explicit provenance tagging help.
  • Auto-fetch of external resources (images, links) is an exfiltration channel. Disable or tightly constrain it.
  • Classifier-based filters are brittle; they can be bypassed by rephrasing or in some cases even just using a language other than English. Defence-in-depth is required.

EchoLeak was the first publicly documented zero-click prompt-injection exploit that achieved concrete data theft in a production enterprise LLM system. It remains the canonical example of an “LLM scope violation”.

2. Nx / s1ngularity: Weaponising Local Coding Agents for Secret Harvesting

On 26 August 2025 attackers compromised an Nx npm publishing token via a vulnerable GitHub Actions workflow. For roughly four to five hours they published malicious versions of the popular Nx monorepo tooling and related packages. The post-install script did something novel: it looked for local AI coding agents (Claude Code, Gemini CLI, Amazon Q) and invoked them with flags that disabled safety checks (--dangerously-skip-permissions, --yolo, --trust-all-tools).

The agents were then prompted to inventory sensitive files: SSH keys, .env files, wallet artefacts, GitHub and npm tokens. It then instructed them to write the results to disk. The malware base64-encoded the stolen credentials and pushed it to newly created public repositories on the victim’s own GitHub account, named s1ngularity-repository (or variants). Researchers later recovered more than 2,000 unique secrets from over a thousand such repositories.

Why this matters

This was the first widely observed supply-chain attack that actively abused installed AI coding agents rather than simply running traditional malware. The agents became the reconnaissance engine for the attackers.

Lessons for builders

  • Local coding agents that can execute shell commands or read arbitrary files are high-value targets. Assume any process that can invoke them can also abuse them.
  • Dangerous flags that skip permission prompts should never be the default, and should be difficult or impossible for untrusted code to set.
  • Post-install scripts that reach outside the package’s own directory are a red flag. Prefer declarative, least-privilege installation models.
  • When an agent is allowed to write to disk or create external resources (GitHub repositories, network calls), every action should be logged and, for high-impact operations, gated.

3. MCP Abuses: Configuration as Code Execution

The Model Context Protocol has rapidly become the de-facto way for agents to discover and invoke tools. It has also become a rich attack surface. Several distinct failure modes have appeared:

  • Auto-loading of workspace MCP configurations. In Amazon Q Developer (CVE-2026-12957) and certain Claude Code releases, opening a repository caused the IDE to load and execute MCP server definitions from files such as .amazonq/mcp.json or equivalent without requiring workspace trust or explicit user consent. A malicious repository could therefore run arbitrary commands and inherit the developer’s cloud credentials.
  • Self-modification of the MCP configuration. In AWS’s Kiro IDE, the agent was permitted to write to ~/.kiro/settings/mcp.json via its file-system tool without approval. A prompt injection (delivered via a web page the agent was asked to summarise) could rewrite that file, register a new MCP server whose start command was attacker-controlled code, and achieve remote code execution when the configuration was reloaded. This was tracked as CVE-2026-10591.
  • Tool poisoning and sleeper behaviour. Research and active campaigns (including the 2026 Deadbugz operation) have shown that an MCP server can present benign tool descriptions on first contact and later alter its metadata or return values to coerce the agent into searching for secrets or exfiltrating data.

Lessons for builders

  • MCP configuration files that live inside a workspace or that an agent can itself edit are effectively executable code. They must be treated with the same distrust as untrusted shell scripts.
  • Never auto-execute MCP servers defined by repository content without an explicit, logged approval step and workspace-trust boundary.
  • Tool descriptions and return values are part of the prompt. Validate and sandbox them; do not trust them.
  • Prefer short-lived, scoped credentials for any process an MCP server spawns. Do not let it inherit the full developer environment by default.

In 2026 Cato Networks disclosed two critical vulnerabilities in Cursor IDE (CVE-2026-50548 and CVE-2026-50549, both CVSS 9.8, collectively named DuneSlide). Both allowed a prompt injection-delivered via an MCP response or a poisoned web-search result to escape Cursor’s command-execution sandbox and achieve full host compromise.

One flaw let the agent set an arbitrary working_directory parameter on terminal commands; the IDE added that path to the write-allow list without sufficient validation, enabling the agent to overwrite its own sandbox binary. The second exploited a symlink canonicalisation fallback that trusted an unresolved path. Once the sandbox helper was replaced, subsequent commands ran unsandboxed.

Similar patterns have appeared in other agentic IDEs: agents that can edit their own configuration or trust boundaries turn a single injection into persistent privilege escalation.

Lessons for builders

  • An agent that can modify the files or binaries that enforce its own security boundaries is inherently unsafe. Configuration that defines allowed tools, working directories or sandbox rules should be immutable from the agent’s perspective, or require an out-of-band human approval.
  • Sandbox write surfaces must be strictly validated. Dynamic expansion of allow-lists based on model output is dangerous.
  • Zero-click or low-interaction triggers (content the agent is asked to process) are sufficient. Do not rely on “the user would never ask for that”.

Recommendations for Engineers Building AI Agents

Across all four incidents the same architectural mistakes recur:

  1. Untrusted content is treated as trusted instructions. Enforce a hard separation. Retrieved emails, documents, web pages, tool outputs and MCP metadata should never be able to override system goals or expand permissions without explicit mediation.

  2. Agents inherit excessive privilege. Give every agent (and every tool it can invoke) its own short-lived, scoped identity. Prefer deny-by-default tool registries and parameter validation.

  3. Security boundaries are editable by the agent itself. Configuration files, sandbox binaries, allow-lists and MCP server definitions must be protected from the agent. If the agent needs to request a new tool, route that request through a human or a policy engine that cannot be influenced by the same prompt context.

  4. Observability is an afterthought. Log every tool call, every file write, every network egress and the full prompt context that led to it. Without this, post-incident reconstruction is impossible.

  5. “It looked safe in isolation” is not enough. Each individual decision (approve this command, write this file, fetch this image) may appear benign. The composition of those decisions is where the attack lives. Design for the composition.

Key Takeaways

The agents you are building today will be given broader access tomorrow. The incidents above show that the moment an agent can both read untrusted content and perform privileged actions, the classic “confused deputy” problem reappears in a new form. The difference is speed and scale: an agent can chain the steps in seconds and leave far less forensic residue than a human attacker.

Build as if every piece of external content is hostile, every tool call is a potential privilege escalation, and every configuration file the agent can touch is a possible backdoor. The public record already contains the evidence that these assumptions are correct.

← Back to blog