SEO Crawler — Setup & Usage Guide
Our in-house crawler audits any site for broken links, redirects, title/description/H1 issues, thin content, schema errors and more — free, and it runs on your own computer.
One-Time Setup ≈ 5 minutes
Do this once per computer. After that, running the tool takes seconds.
Install Python
Download from python.org/downloads and run the installer. If Python is already installed, skip this step.
Install the required packages
Open Command Prompt (press the Windows key, type cmd, press Enter), then paste this and press Enter. Needs internet; runs once.
Save the tool in a permanent folder
Save seo_crawler_web.py somewhere fixed, for example D:\seo-tool\.
Running a Crawl every time
Open Command Prompt and go to the tool folder
Start the tool
Your browser opens automatically at http://localhost:8090.
Enter the domain and start
Type just the domain, e.g. example.com — no need to type https. Click Start Crawl. Phase 1 crawls the pages; Phase 2 checks images, CSS, JS and external links. The progress line shows which phase is running.
Review the tabs, then stop the tool
Click through the tabs to review issues (reference table below). When finished, go back to Command Prompt and press Ctrl + C to stop the tool.
Reading the Results tab reference
Every issue is a colored chip: red = broken yellow = redirects & metadata blue = structure purple = content
| Tab | What it shows | What to do |
|---|---|---|
| Broken | 404 / error pages, with “linked from” showing which page contains the bad link | Fix or remove the link on the “linked from” page |
| Redirects | 301/302 links with a “Found On” column and full redirect chain | Update links on the Found On pages to point directly to the final URL |
| Titles / Descriptions / H1s | Missing, duplicate, too short or too long | Fix in Rank Math / the page editor |
| Non-Indexable | Pages Google won’t index, with the reason (noindex, canonical, robots.txt, redirect) | Confirm each reason is intentional |
| Thin Content | Pages under 300 words | Add content, or noindex the page |
| Speed | Slow responses (>2s), oversized pages, pages loading too many files | Investigate hosting, image sizes, plugins |
| Schema/Lang | Invalid JSON-LD structured data; missing <html lang> attribute |
Fix the schema syntax; set the correct lang code (en / gu / mr / hi) |
| Resources | Broken images, CSS and JS files, with the pages that use them | Re-upload the file or fix the path |
| Links | All internal/external links with inlink counts and anchor text | Pages with only 1–2 inlinks need more internal links from hub pages |
| Changes | What’s new or fixed since the previous crawl of the same domain | Review after every content update or bulk fix |