polite-crawler

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

GitHub · npm · Website · Registry

Browser & web scraping Search & knowledge Dev tools & GitHub

Downloads, last 30 days

Stars over time

Install

Run it directly:

npx -y polite-crawler

Or add it to your client config (Claude Desktop claude_desktop_config.json, Cursor .cursor/mcp.json):

{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler"
      ]
    }
  }
}

Generated from the package metadata in the official MCP Registry. MCP Morning does not run or test servers; check required environment variables in the project README.

Alternatives