getreadystack/robots-txt-ai-crawler-audit repository overview

⁠AI Crawler Rules - robots.txt Audit for AI Search

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI-search citation, or user fetch - and which lines silently do nothing.

⁠Run

docker run --rm -v "$PWD":/work getreadystack/robots-txt-ai-crawler-audit /work/file

The same 19 rules as the VS Code extension, in a container. Mount a folder on /work.

⁠Free

• Check the open robots.txt against every rule - full results, nothing withheld; Check only the lines you highlight; See every rule and what each AI token controls; Reopen the last report
• --rules lists every rule

⁠With a licence ($29 once)

• Scan every robots.txt in the workspace; Export the report as CSV, JSON or HTML; CI output that fails the build on errors; Add your own agency policy tokens; Re-check automatically on every save

docker run --rm -e READYSTACK_LICENSE=<key> -v "$PWD":/work getreadystack/robots-txt-ai-crawler-audit --dir /work --ci

A freelance technical SEO consultant bills roughly $100-150 an hour, and an AI-crawler robots.txt review is a one-to-two hour job.

⁠Use from an AI agent (MCP)

Claude Code · Cursor · Windsurf · any MCP client - add to your MCP config:

{ "mcpServers": { "robots-txt-ai-crawler-audit": { "command": "npx", "args": ["-y", "@readystack/robots-txt-ai-crawler-audit", "--mcp"] } } }

Tools: check_text and check_file (free) · check_dir (licence; the full sweep is free for 7 days). The agent gets every finding with the line number.

⁠Use in CI

- name: AI Crawler Rules - robots.txt Audit for AI Search
  run: npx -y @readystack/robots-txt-ai-crawler-audit --dir . --ci

(container: docker run --rm -v "$PWD:/work" getreadystack/robots-txt-ai-crawler-audit --dir /work --ci)

Try the full run free for 7 days — no key needed. Then one licence, 7-day refund, no questions. Set READYSTACK_LICENSE=<key> or run --license <key> once.

Get a licence⁠