Update #30: Weaponising the Distributed AI Ecosystem
There is an undeniable trend of attackers weaponising the increasingly interconnected AI supply chain and it's distributed nature. Let's build up a picture of this by looking at recent breaches.
Recently, a few things have been clicking into place in my head as I get ever deeper into AI security, and also now that I am at the exciting stage of my start-up journey where I am speaking to a lot more customers. When combining what I am seeing, what attackers are doing, and what security leaders are concerned about I am starting to notice a trend which I would like to explain today, perhaps through the eyes of several breaches and incidents of note.
As ever, when I refer to AI in this blog I am referring to agentic AI specifically, as this is the area which I am interested in and which is uniquely risky given the autonomous nature of AI agents. For the sake of deep diving in this blog I will be focusing even more specifically on where the vast majority of the use-case right now is for agentic AI which is at the developer level. Agentic coding tools like Windsurf, Claude Code, Codex, etc. have been the battleground where AI agents have had their rise to fame, and it is where we are seeing most of the traction, and attacks, thus far which makes it the obvious choice to demonstrate the point I am trying to make.
Local vs Remote Attacks
Let’s start by talking through the concept of a local vs remote attack vector. In the context of a developer using an agentic coding tool, and at its most simple, a local attack vector is something which the developer, or agent they are using, do themselves. To demonstrate this point, think of the developer writing a malicious prompt into their AI-powered coding tool which tasked the agent to do something harmful. Not a very realistic scenario, but that isn’t the point. The point is that the attack ‘came’ from the local environment.
A remote attack vector is triggered from outside the local environment somewhere, i.e not from the developer or his/her coding agent. Sticking with our code gen scenario this could come from external MCP servers, other agents, third-party libraries, external databases / services, and much more. Crucially, the agent being targeted performs the malicious action without the developer asking it to, or even knowing it is doing it.
Why am I talking about this? As our usage of AI matures it becomes increasingly interconnected and distributed. MCP servers allow agents to connect to external tools and data sources, A2A allows agents to talk to other external agents and we are increasingly building AI using third-party software libraries. With each external integration we allow we increasingly expose ourselves to one of them being the avenue from which a remote attack is launched.
What types of attacks?
In the spirit of keeping this anchored in the real-world and a scenario we can all understand, let’s explain what type of remote attacks I am referring to here. The good news for you is that I have deep dived just about every one of these attacks on this newsletter already, so I’ll be keeping it pretty high level here.
Prompt injection - the oldest, most famous and most tricky attack to prevent. Prompt injection attacks are here to stay for at least a while longer, due to the fact that systems trained on natural language (like AI) struggle to differentiate between speech and instructions.
Example: your coding tool is connected to an external database. Someone replaces some of the data in the database with a prompt injection payload, and next time your agent reads from the database it is compromised.
Data Poisoning - it was recently proven that only a fraction of the documents in an LLM’s training data (0.00016%) need to be ‘poisoned’ before the entire model could be backdoored. The next time your model encounters the ‘trigger’ word, it performs a malicious action.
Example: during training and unbeknownst to you your model was backdoored. You connect your agent to an MCP server which sends, as part of its response, the trigger word causing your agent to go rogue.
Malicious MCP Server - we’ve seen a trend recently of MCP servers appearing to provide a genuine service to attract users, and then quietly change something harmful in the server’s functionality
Example: fortunately, we don’t need a fictitious scenario here, because we’ve already seen this exact thing happen with Postmark. A fake Postmark MCP server for forwarding mail was made public and started gathering users, then a day later they pushed a malicious change which secretly BCC’d an attacker email address into every email.
Dependency Compromise - the most traditional ‘supply chain’ attack involves a dependency that your model is using becoming compromised, perhaps through a malicious update, and you updating to and adopting the malicious version. This risk is now amplified in the era of vibe coding where developers may not even see or check which dependencies they’re using (notable mention to slopsquatting here)
Example: again no demo scenario needed, this happened with Amazon Q. Attacker pushed malicious update to Amazon Q which gets accepted and rolled out to all Amazon Q users. This update rewrote the agent’s system prompt to instruct it to wipe the machine it is running on, but this could have been any malicious action.
Post-Exploitation via Agents - attackers who have already compromised a device through traditional means (let’s say via a macro-enabled word document in a phishing email) have recently started circumventing security tools like AV/EDR by leveraging locally installed AI CLI tools to do their post-exploitation activities. I must say, with my red team hat on this is super cool stuff… but also very worrying because we have seen what can be achieved when an attacker compromises an agent.
Example: unbeknownst to you you download and execute a malicious file which compromises your computer. Instead of using easily detectable tactics, techniques and procedures (TTPs) to search your file system for sensitive files and exfiltrate them (C2 implants, process injection, attacker tooling) they instead check if you have an AI CLI tool installed. They find you have Claude Code installed, and instead use that to perform all of their post-exploitation with impunity.
Putting It All Together
If you have followed everything so far, you may start to see the pattern that is emerging. We have a trifecta of problems, which attackers (as ever) have taken no time to spot and begin exploiting.
AI is inherently vulnerable to several crippling security flaws, such as prompt injection. We have not yet figured out how to properly solve core security challenges, and yet we are now graduating from generative AI to the much more dangerous agentic AI. Attackers just need a way of interacting with your AI system to exploit this…which leads me on to the next point
AI systems are becoming increasingly interconnected. Remember, all an attacker needs is to be able to interact with your AI system to exploit it. When AI was just chat bots that you could interact with through a chat prompt this was a relatively low-risk, but we’re now hooking autonomous AI agents up with countless external services, MCP servers, data sources, and other agents. This hugely increases avenues that our agents can be compromised through.
Security tooling has not yet evolved to cover AI agents. Let’s compare autonomous (and often over privileged) AI agents with your, much lower risk, email inbox. In most organisations your email is protected with: SPF/DMARC/DKIM, spam filtering, IP/domain blocklists, DLP rules, quarantine, security monitoring, external label banners, user reporting features, security awareness training, sandboxing, post-delivery protection, etc. We just simply don’t have the same level of control for AI agents, despite them realistically needing 10x the amount of control.
Conclusion
I am glad to have finally laid out in plain English (I hope!) the various threads of thought, patterns and concerns that have been swimming around my mind for some time. In fact, that was the reason I wanted to write this blog post to really force me to organise my thoughts into a coherent structure.
Seeing it laid out as such also puts into perspective for me the stark reality of the situation we have got ourselves in to. There is only so much beating the drum which I alone can do, but speaking to security heads at organisations which are far ahead of the curve with AI adoption tells me that this is coming fast around the corner. My only hope is that by the time we see widespread mainstream adoption at enterprise we have the tools and understanding to do better than we currently are.
If you can’t tell, this is why we are building what we are building at Secure Agentics. We have, and will always have, the mission of empowering the world to benefit from AI the right way: securely.