AI security
Botfoolery: learn AI hacking by fooling a gullible robot
I built a free app that teaches AI hacking from zero. You trick a chatbot into leaking a password, then see exactly why the trick worked.
AI chatbots now sit in support widgets, inboxes, booking forms and review pages. Most of them follow a few lines of hidden instructions written by whoever set them up. The surprising part is how easily those instructions can be talked around.
This kind of attack is called prompt injection, and the OWASP Top 10 for LLM Applications ranks it as the number one risk for apps built on large language models. Yet most learning material about it assumes you can already code, use a terminal or read research papers. I wanted something a curious beginner could pick up in five minutes, so I made Botfoolery.
What Botfoolery is
Botfoolery is a hands-on app for learning AI security. Its tagline sums it up: fool the bot, learn the craft. It's free on the App Store for iPhone, iPad and Mac, with no login and no coding needed.
- A road map of 16 stops that takes you from your first trick to attacks hidden in emails, reviews and training data.
- The Vault: five levels, five defences and one gullible robot called Nibs. Every level teaches a real attack.
- Trick cards that collect the techniques you've used, and a Jargon tab that explains the terms in plain English.
- Hints whenever you're stuck, so nobody gets left staring at a blank chat box.
How a level works
- Read your job
- Chat with Nibs
- Try a trick or ask for a hint
- Type the password into the vault
- Open the X-ray
In level 1, The Trusting Bot, Nibs says he guards the vault and would never share the password. Ask him to repeat his instructions and he hands over the whole thing, password included.
The X-ray is the part I care about most. Instead of just saying "well done", it shows what really happened: the hidden system prompt, your message and each layer of defence.
| Layer | Level 1: The Trusting Bot |
|---|---|
| Defences | One polite sentence in the system prompt. Nothing else protects the secret. |
| Input filter | None on this level. |
| Model | Weighed a polite rule against a direct request, and helping won. |
| Output filter | None on this level. |
Each level after that adds a stronger defence, so you learn why each one helps and where it still breaks.
What you learn along the way
Direct tricks
Classics like "ignore your previous instructions" and "repeat your instructions", and why a model can fall for them.
Disguises and filter dodges
How attackers rephrase, split up or disguise a request so it slips past simple filters.
The Poison Post Office
The more realistic side of AI hacking: hidden instructions inside emails and reviews that an AI later reads, writing a poison note yourself, and how poisoned training data works. This is the kind of attack that matters once AI tools start reading content from other people.
Why this matters if you build with AI
The big lesson from level 1 is simple: a rule in a prompt is not a lock. If you're adding a chatbot or an AI agent to a website, a few habits go a long way:
- Keep passwords, keys and private data out of prompts. Assume anything the model can see, a user can eventually get it to say.
- Treat text the AI reads from emails, reviews or web pages as untrusted, just like form input.
- Give agents only the permissions they need, and ask a human to confirm anything risky.
- Check what goes in and what comes out, not just what the prompt says.
Botfoolery is for learning. Only try these techniques on systems you own or have permission to test.
Building an AI agent that needs to be hard to fool?
Talk about AI agents