Keeping AI agents and bots under control
Two AI-security problems arrived at once. Inside your product, agents now take real actions — calling APIs, deleting records, spending money — and a single bad tool call can do real damage. Outside it, a wave of AI crawlers scrapes your site, and the worst of them ignore robots.txt entirely.
agent·shield puts a firewall in front of agent actions, forwarding safe traffic instantly and holding destructive calls for a human to approve. robot·guard handles the other side, identifying and blocking the AI bots that don't play by the rules. Together they cover both directions of the same trust problem.
guides
- use caseAI agent guardrails: stop the agent at the dangerous request, not after it
- how toHow to put a human in the loop for an AI agent's destructive actions
- how toApprove AI agent actions: a manual checkpoint for the calls that matter
- use caseYour AI agent can delete production data. Here's how to make that safe.
- use caseAI agent firewall: the category, what it controls, and how agent·shield does it
- use caseMCP server security: put a gate in front of the tools your agent can call
- use caseAI agent audit log: a defensible record of every action the agent took
- comparisonSecuring your AI agent vs shipping it unguarded: the honest trade-off
- how toStop destructive agent SQL: hold DROP, TRUNCATE and WHERE-less updates
- use caserobot·guard: one place to whitelist the bots you want and block the ones you don't
- use caseWhat is robots.txt, and why it matters more than ever
- how toHow to block specific AI bots from scraping your website
- use caseThe hidden cost of unwanted bot traffic — and how AI scrapers inflate it
- how torobots.txt for SEO: how to whitelist Googlebot without locking out the rest
- comparisonrobots.txt vs firewall: choosing the right bot protection
- how toA developer's guide to robots.txt rules that don't bite you later
- how toHow to generate an intelligent robots.txt for the modern web
- how toHow to protect your content from AI training with robots.txt
- use caseSmall business website security: robots.txt strategies that actually help
- how toHow to block GPTBot in robots.txt
- how toHow to block ClaudeBot and anthropic-ai
- how toHow to block CCBot (Common Crawl)
- use caseWhat is Google-Extended (and should you block it)?
- use caseDo AI bots actually respect robots.txt?
- use caseThe AI crawler user-agents to know in 2026
- how toHow to edit robots.txt in WordPress
- how toRobots.txt on Shopify: what you can and can't change
- comparisonrobots.txt vs noindex: which one keeps a page out of Google?
- comparisonrobots.txt vs llms.txt: do you need both?
frequently asked
Why can't I just trust an AI agent's tool calls?
An agent's actions are only as predictable as the model behind them, and a prompt injection or a confident mistake can turn a routine task into a destructive one. A guardrail layer lets safe actions through instantly while pausing irreversible ones — delete, pay, send — for a human to confirm.
Do AI crawlers actually respect robots.txt?
The major identifiable crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot) publish their user-agents and state they honour robots.txt, but it's a voluntary request, not an enforced wall. A firewall is what stops the bots that ignore it or hide their identity.