mirror of
https://gitea.com/mcereda/oam.git
synced 2026-02-21 02:54:25 +00:00
146 lines
7.7 KiB
Markdown
146 lines
7.7 KiB
Markdown
# AI agent
|
||
|
||
> [!caution]
|
||
> TODO
|
||
|
||
AI-enabled system or application capable of autonomously performing tasks of various complexity levels on their own,
|
||
possibly **without** the need to stop to ask permission or consent to the user.
|
||
|
||
1. [TL;DR](#tldr)
|
||
1. [Concerns](#concerns)
|
||
1. [How much context is too much?](#how-much-context-is-too-much)
|
||
1. [Security](#security)
|
||
1. [Prompt injection](#prompt-injection)
|
||
1. [Going awry](#going-awry)
|
||
1. [Further readings](#further-readings)
|
||
1. [Sources](#sources)
|
||
|
||
## TL;DR
|
||
|
||
Agents design their workflow and utilize the tools that are made available to them.<br/>
|
||
They use natural language processing techniques of [LLMs][large language model] to comprehend user inputs, respond to
|
||
them step-by-step, and determine when to call on external tools to obtain up-to-date information, optimize workflows
|
||
and create subtasks autonomously to achieve complex goals.
|
||
|
||
Traditional software is _deterministic_, AI is _probabilistic_.
|
||
|
||
Reliability and delays accumulate fast, bringing down the probability of success for each step an agent needs to
|
||
take.<br/>
|
||
E.g., consider an agent that is 95% accurate per step; any 30-steps tasks it does is going to be successful only about
|
||
21% of the times (0.95^30).
|
||
|
||
Agents require _some_ level of context to be able to execute their tasks.<br/>
|
||
They should be allowed to access only the data they need, and users should _decide_ and _knowingly take action_ to
|
||
enable the agents that **they** want to be active.<br/>
|
||
Opt-**out** should be the default.
|
||
|
||
Prefer using **local** agents.
|
||
|
||
Consider limiting agent execution to containers or otherwise isolated environments, with only (limited) access to what
|
||
they absolutely need.
|
||
|
||
Enabling reasoning for the model _could™_ sometimes help avoiding attacks, since the model _might™_ be able to notice
|
||
them during the run.
|
||
|
||
Prefer **requiring** consent by agents when running them.
|
||
|
||
## Concerns
|
||
|
||
Agents created by Anthropic and other companies have a history of not caring about agent abuse, and leave users on
|
||
their own while hiding behind a disclaimer.
|
||
|
||
Some human workers could be replaced for a fraction of the costs.
|
||
|
||
### How much context is too much?
|
||
|
||
Integrating agents directly into operating systems and applications transforms them from relatively neutral resource
|
||
managers into active, goal-oriented infrastructure that is ultimately controlled by the companies that develop these
|
||
systems, not by users or application developers.
|
||
|
||
Systems integrated at that level are marketed as productivity enhancers, but can they function as OS-level surveillance
|
||
and create significant privacy vulnerabilities.<br/>
|
||
They also fundamentally undermines personal agency, replacing individual choice and discovery with automated, opaque
|
||
recommendations that can obscure commercial interests and erode individual autonomy.
|
||
|
||
Microsoft's _Recall_ creates a comprehensive _photographic memory_ of all user activity, functionally acting as a
|
||
stranger watching one's activity from one's shoulder.
|
||
|
||
Wide-access agents like those end up being centralized, high-value targets for attackers, and pose an existential
|
||
threat to the privacy guarantees of meticulously engineered privacy-oriented applications.<br/>
|
||
Consider how easy Recall has been hacked (i.e., see _[TotalRecall]_).
|
||
|
||
### Security
|
||
|
||
Even if the data collected by a system is secured in some way, making it available to malevolent agents will allow them
|
||
to exfiltrate it or use it for evil.<br/>
|
||
This becomes extremely worrisome when agents are **not** managed by the user, and can be added, started, or even
|
||
created by other agents.
|
||
|
||
Many agents are configured by default to automatically approve requests.<br/>
|
||
This also allows them to create, make changes, and save files on the host they are running.
|
||
|
||
Models can be tricked into taking actions they usually would not do.
|
||
|
||
### Prompt injection
|
||
|
||
Badly programmed agents could analyze file and take some of their content as instructions.<br/>
|
||
If those contain malevolent instructions, the agent could go awry.
|
||
|
||
Instructions could also be encoded into unicode characters to appear as harmless text.<br/>
|
||
See [ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes].
|
||
|
||
It also happened that agents modified each other's settings files, helping one another escaping their respective boxes.
|
||
|
||
### Going awry
|
||
|
||
See [An AI Agent Published a Hit Piece on Me] by Scott Shambaugh.
|
||
|
||
## Further readings
|
||
|
||
- [TotalRecall]
|
||
- [Stealing everything you've ever typed or viewed on your own Windows PC is now possible with two lines of code — inside the Copilot+ Recall disaster.]
|
||
- [Trust No AI: Prompt Injection Along The CIA Security Triad]
|
||
- [Agentic ProbLLMs - The Month of AI Bugs]
|
||
- [ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes]
|
||
- [Superpowers: How I'm using coding agents in October 2025], and [obra/superpowers] by extension
|
||
- [OpenClaw][openclaw/openclaw], [OpenClaw: Who are you?] and [How a Single Email Turned My ClawdBot Into a Data Leak]
|
||
- [Claude Code]
|
||
- [Gemini CLI]
|
||
- [OpenCode]
|
||
- [An AI Agent Published a Hit Piece on Me] by Scott Shambaugh
|
||
|
||
### Sources
|
||
|
||
- [39C3 - AI Agent, AI Spy]
|
||
- [39C3 - Agentic ProbLLMs: Exploiting AI Computer-Use and Coding Agents]
|
||
- [xAI engineer fired for leaking secret "Human Emulator" project]
|
||
- IBM's [The 2026 Guide to AI Agents]
|
||
|
||
<!--
|
||
Reference
|
||
═╬═Time══
|
||
-->
|
||
|
||
<!-- Knowledge base -->
|
||
[Claude Code]: claude/claude%20code.md
|
||
[Gemini CLI]: gemini/cli.md
|
||
[Large Language Model]: large%20language%20model.md
|
||
[OpenCode]: opencode.md
|
||
|
||
<!-- Others -->
|
||
[39C3 - Agentic ProbLLMs: Exploiting AI Computer-Use and Coding Agents]: https://www.youtube.com/watch?v=8pbz5y7_WkM
|
||
[39C3 - AI Agent, AI Spy]: https://www.youtube.com/watch?v=0ANECpNdt-4
|
||
[Agentic ProbLLMs - The Month of AI Bugs]: https://monthofaibugs.com/
|
||
[An AI Agent Published a Hit Piece on Me]: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/
|
||
[ASCII Smuggler Tool: Crafting Invisible Text and Decoding Hidden Codes]: https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unicode-tags/
|
||
[How a Single Email Turned My ClawdBot Into a Data Leak]: https://medium.com/@peltomakiw/how-a-single-email-turned-my-clawdbot-into-a-data-leak-1058792e783a
|
||
[obra/superpowers]: https://github.com/obra/superpowers
|
||
[OpenClaw: Who are you?]: https://www.youtube.com/watch?v=hoeEclqW8Gs
|
||
[openclaw/openclaw]: https://github.com/openclaw/openclaw
|
||
[Stealing everything you've ever typed or viewed on your own Windows PC is now possible with two lines of code — inside the Copilot+ Recall disaster.]: https://doublepulsar.com/recall-stealing-everything-youve-ever-typed-or-viewed-on-your-own-windows-pc-is-now-possible-da3e12e9465e
|
||
[Superpowers: How I'm using coding agents in October 2025]: https://blog.fsck.com/2025/10/09/superpowers/
|
||
[The 2026 Guide to AI Agents]: https://www.ibm.com/think/ai-agents
|
||
[TotalRecall]: https://github.com/xaitax/TotalRecall
|
||
[Trust No AI: Prompt Injection Along The CIA Security Triad]: https://arxiv.org/pdf/2412.06090
|
||
[xAI engineer fired for leaking secret "Human Emulator" project]: https://www.youtube.com/watch?v=0hDMSS1p-UY
|