# polite-crawler

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

- Weekly downloads: 842
- GitHub stars: 0
- Health: active
- Repository: https://github.com/mark1russell7/crawler
- Registry names: io.github.mark1russell7/polite-crawler
- Categories: Browser & web scraping, Search & knowledge, Dev tools & GitHub
- Snapshot: 2026-10-21

## Install

```sh
npx -y polite-crawler
```

```json
{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler"
      ]
    }
  }
}
```

Source: https://mcpmorning.com/mcp/mark1russell7-crawler/