Scrape-LE

Analyse robots.txt content and report whether a path may be crawled.

GitHub · npm · Website · Registry

Browser & web scraping Dev tools & GitHub

Downloads, last 30 days

Install

Run it directly:

npx -y scrape-le-mcp

Or add it to your client config (Claude Desktop claude_desktop_config.json, Cursor .cursor/mcp.json):

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": [
        "-y",
        "scrape-le-mcp"
      ]
    }
  }
}

Generated from the package metadata in the official MCP Registry. MCP Morning does not run or test servers; check required environment variables in the project README.

Alternatives