6.4 KiB
AI agent
Caution
TODO
AI-enabled system that proposes to complete tasks of various complexity levels on their own, possibly without the need to stop to ask permission or consent to the user.
TL;DR
Traditional software is deterministic, AI is probabilistic.
Reliability and delays accumulate fast, bringing down the probability of success for each step an agent needs to
take.
E.g., consider an agent that is 95% accurate per step; any 30-steps tasks it does is going to be successful only about
21% of the times (0.95^30).
Agents require some level of context to be able to execute their tasks.
They should be allowed to access only the data they need, and users should decide and knowingly take action to
enable the agents that they want to be active.
Opt-out should be the default.
Prefer using local agents.
Consider limiting agent execution to containers or otherwise isolated environments, with only (limited) access to what they absolutely need.
Enabling reasoning for the model could™ sometimes help avoiding attacks, since the model might™ be able to notice them during the run.
Prefer requiring consent by agents when running them.
Concerns
Agents created by Anthropic and other companies have a history of not caring about agent abuse, and leave users on their own while hiding behind a disclaimer.
How much context is too much?
Integrating agents directly into operating systems and applications transforms them from relatively neutral resource managers into active, goal-oriented infrastructure that is ultimately controlled by the companies that develop these systems, not by users or application developers.
Systems integrated at that level are marketed as productivity enhancers, but can they function as OS-level surveillance
and create significant privacy vulnerabilities.
They also fundamentally undermines personal agency, replacing individual choice and discovery with automated, opaque
recommendations that can obscure commercial interests and erode individual autonomy.
Microsoft's Recall creates a comprehensive photographic memory of all user activity, functionally acting as a stranger watching one's activity from one's shoulder.
Wide-access agents like those end up being centralized, high-value targets for attackers, and pose an existential
threat to the privacy guarantees of meticulously engineered privacy-oriented applications.
Consider how easy Recall has been hacked (i.e., see TotalRecall).
Security
Even if the data collected by a system is secured in some way, making it available to malevolent agents will allow them
to exfiltrate it or use it for evil.
This becomes extremely worrisome when agents are not managed by the user, and can be added, started, or even
created by other agents.
Many agents are configured by default to automatically approve requests.
This also allows them to create, make changes, and save files on the host they are running.
Models can be tricked into taking actions they usually would not do.
Prompt injection
Badly programmed agents could analyze file and take some of their content as instructions.
If those contain malevolent instructions, the agent could go awry.
Instructions could also be encoded into unicode characters to appear as harmless text.
See ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes.
It also happened that agents modified each other's settings files, helping one another escaping their respective boxes.
Further readings
- TotalRecall
- Stealing everything you've ever typed or viewed on your own Windows PC is now possible with two lines of code — inside the Copilot+ Recall disaster.
- Trust No AI: Prompt Injection Along The CIA Security Triad
- Agentic ProbLLMs - The Month of AI Bugs
- ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes
- Superpowers: How I'm using coding agents in October 2025, and obra/superpowers by extension
- Moltbot and How a Single Email Turned My ClawdBot Into a Data Leak