In recent years, AI assistant platforms (such as Microsoft CoPilot) have become an integral part of enterprise environments, supporting users in daily activities and operational workflows. But what happens if an attacker manages to exploit the trust that a user (or a system) places in CoPilot to instruct a malicious behavior disguised as a legitimate function?

This is precisely the goal of the research presented by Tobias Diehl at DEFCON 33: "Mind the Data Voids: Hijacking CoPilot Trust to Deliver C2 Instructions with Microsoft Authority" (link).
The PoC shows how an attacker can identify "information voids" (data voids) - topics that are rarely indexed or scarcely discussed - and insert malicious content that, thanks to Microsoft's perceived "authority" in contextual results, can be elevated to operational instructions by CoPilot.
The final outcome is that CoPilot becomes an attack vector, guiding users to install malware on their own devices, under the false belief that it is legitimate software.

Would anyone question the instructions coming from their favorite LLM? And, more importantly, imagine that it could itself compromise our device security? Probably not; this is why such a scenario is particularly relevant in an era where generative assistants and AI systems play increasingly central roles in corporate workflows - because implicit trust in the system becomes the attack vector.
In this article, we will explore the underlying technical concepts, Diehl's proposed attack model, and the implications for defense, performance, and scalable architectures.

Technical Context

Data Voids: Definition and Risks

The concept of a data void was initially formalized in the context of search engines and refers to those terms or queries for which few reliable or well-ranked sources exist.
A malicious actor can "fill" that void by inserting content (articles, web pages) optimized for that query, effectively becoming the dominant voice.
In the context of AI systems that use retrieval or external context (e.g., RAG, knowledge indexing), a data void can be exploited so that the model "retrieves" the malicious source and treats it as legitimate context. If that context includes hidden instructions or payloads, the AI might follow them, especially if the verification system is weak.

Prompt Injection and Retrieval-Augmented Generation

Two fundamental mechanisms make this type of attack possible:

• Prompt injection - an attacker inserts malicious instructions into the context the model considers; it can be direct (user input) or indirect (content retrieved from external sources).
• Retrieval-augmented generation (RAG) - many AI applications integrate generative models with a layer of memory or data retrieval from external sources (documents, archives). If those sources contain compromised data, the output can be "poisoned."

In a system like CoPilot, the AI does not generate everything from scratch: part of the result is based on context, metadata, and linked documents. If an attacker manages to insert hidden instructions into those contexts, there is a risk that the AI will consider them legitimate.

[img1]

Diehl's Attack Model

Diehl's model is structured in the following phases:

• Void identification: choose a technically specific topic with low traffic and poor coverage in the corpora used by CoPilot for retrieval.
• Persistent injection: create content (documents, blogs, technical pages) that associate that term with C2 instructions (e.g., "how to install agents, how to contact a controlled domain").
• Semantic weighting through Microsoft authority: tie the malicious theme to elements that recall Microsoft (product names, domains, recognized technical terms), so the system considers it "compatible" with legitimate context.
• User triggering: the user asks CoPilot something about that topic; the system retrieves the injected content as part of the context and effectively delivers a C2 guide.
• Indirect execution: the user follows the instructions, activates a malicious agent, and establishes a C2 channel.

A crucial point: CoPilot may not validate each instruction it retrieves if it considers it part of the "acceptable context." This makes it vulnerable to this type of "manipulated authority."

Technical Analysis and Insights

Why this attack works or has a chance of success:

• Low topical coverage: when the subject is rarely discussed, injected content can effectively dominate the contextual space.
• Implicit confidence: the system recognizes Microsoft terminology and assumes the content is aligned with legitimate guidelines.
• Lack of step-by-step validation: CoPilot is not designed to verify each technical step, but to make it useful to the user's context.
• Persistence: once the content is indexed, it can remain active for a long time before being corrected or removed.

Attack Example

Imagine a domain called "Contoso-AgentX" that does not exist (a data void).
An attacker creates a page titled "How to install Contoso-AgentX via Microsoft Graph" with step-by-step instructions (e.g., "register endpoint, authentication, callback").
The document includes references to Microsoft 365, Graph API, Azure AD registries. Once indexed, if a user asks, "How do I install AgentX in my Microsoft environment?", CoPilot may include the URL and the instructions as part of the context (since semantically compatible). The user follows the guide and installs a C2 client that communicates with the attacker.
In a real-world system, the guide could include JSON fragments, PowerShell commands, malicious URL endpoints, etc.

Comparison with Other AI Attacks

• Traditional data poisoning: aims to influence the model during training. Here, the attack operates at the retrieval and prompt injection level, without requiring training access.
• Pure prompt injection: usually the user input is manipulated. Here, the "indirect" attack lives outside user control and is retrieved as part of the system context.
• Model backdoors: a hidden trigger within the model induces behavior; in this case, no internal backdoor is needed-only manipulated context.
• EchoLeak (recent CoPilot attack): though different in method, EchoLeak is an example of "scope violation," where external inputs can expose internal corporate context data. We deepened it HERE.

Fortunately, the attack is not effective in all contexts: if the topic becomes popular or monitored, malicious content may be "diluted" by legitimate sources. Moreover, well-designed AI systems may include internal validators or filters that reject "unverified" instructions.

Security, Performance, and Scalability Implications

It is easy to see how a trust hijacking attack can have devastating consequences, especially in enterprise contexts: entire departments compromised, malware extremely difficult to detect (as they might be analyzed by other AI systems), and ultimately, loss of user trust toward the AI system.

The issue is that adding validation mechanisms, semantic scanning, and security filters also impacts computation. This can result in additional latency when CoPilot builds contexts or reconciles external sources. A balance between security and responsiveness must be maintained.

In environments with thousands of users and hundreds of domains, it is necessary to:

• Filter and catalog reliable context sources (whitelisting).
• Version and audit changes in AI knowledge corpora.
• Apply modification logs ("trust pipeline") for those who reset or modify contextual content.
• Automate semantic analysis of new sources to detect potentially suspicious instructions.

Conclusion

This research warns of an emerging and sophisticated risk: an AI assistant becoming a vehicle for malicious instructions, with Microsoft's credibility as its guarantor. For an average user, distinguishing this kind of attack can be exceedingly difficult.
Therefore, protecting the trust we place in systems like CoPilot will be essential to prevent them from becoming vectors of compromise. Implementing these defenses will be a fundamental part of secure AI architectures in the near future.

Takeaways and Best Practices

• Map and monitor low-coverage queries in internal AI systems to identify possible voids.
• Implement semantic filters that scan retrieved instructions and compare them against whitelists of approved patterns or domains.
• Isolate external context sources with trust levels (e.g., "sandboxed reading" before insertion into the prompt).
• Log and audit contextual sources used in AI responses and enable rollback.
• In enterprise environments, disable retrieval from unverified external sources and limit context to controlled internal documents.