Should I vibe code
Crawl an owned site locally, inspect technical signals, and export raw audit data
Crawling your own site and reading the headers is a script, not a licence.
?
Their verdict, the SEO Spider licence price and the build-time estimate come from their entry, MIT-licensed. Checked 2026-08-03.
?
Our verdict, the regret score and everything below it. Editorial and unsponsored — nobody can pay to be moved.
The honest answer
why the verdict is what it is
A polite crawler over a site you own, dumping technical signals to a table. Rate-limit it, respect robots, and there is nothing here that can go wrong.
What actually breaks
not "if". the specific failures.
- Crawling your own site too fast, which is a self-inflicted load test on production during business hours
- Crawler traps — faceted filters, calendars, session parameters — which generate infinite URLs and a crawl that never ends
- JavaScript-rendered pages, which return almost nothing to a plain fetch and make the audit wrong rather than incomplete
- Following links off-site, so a crawl of your own site becomes a crawl of the internet
- Counting redirect chains and canonicals correctly, which is most of what a technical audit is actually about
The crawl starts on a Tuesday morning and does not stop. Your site has a filter on the product listing — three facets, each with a dozen options — and every combination is a URL, so the crawler is dutifully working through several hundred thousand pages that are all the same twelve products. Production is slower for everyone else while it does. Nothing is broken. The site advertised infinite URLs and your crawler believed it, which is what crawlers do.
Is that you?
the verdict is a default, not a law
- It crawls only sites you own, with concurrency and a total page limit set
- It honours robots.txt and stays on the host it started on
- You run it out of hours, or against staging
- There is no page cap and no trap detection
- It follows links to other domains
- You are crawling sites you do not own
- The site is JavaScript-rendered and you are fetching raw HTML
If you build it anyway
the checklist, then the prompt that enforces it
- Set a total page cap and a concurrency limit before the first run, and default both low. An unbounded crawler on a faceted site does not terminate.
- Stay on the starting host, and honour robots.txt and crawl-delay even on your own site — it is good practice and it catches configuration mistakes.
- Detect traps: cap URL depth, ignore known session and sort parameters, and stop when a path pattern generates pages beyond a threshold.
- Run out of hours or against staging. This is a load generator pointed at production.
- Check whether the site needs JavaScript rendering before trusting a plain fetch, or the whole audit is measuring an empty page.
- Record raw responses — status, headers, canonical, redirect chain — and analyse separately, so re-analysis does not mean re-crawling.
Before you build a site crawler for technical audits, apply these and push back if I ask you to break them. 1. Set a maximum page count and a concurrency limit before the first crawl, and default both conservatively. Tell me a faceted listing or a calendar generates effectively infinite URLs and that an uncapped crawl does not terminate — it just slows my site until I notice. 2. Restrict crawling to the starting host by default and never follow external links. Ask me explicitly before crawling anything I do not own. 3. Honour robots.txt and any crawl-delay, even on my own site, and tell me this doubles as a check that my robots configuration is what I think it is. 4. Detect crawler traps: cap URL depth, ignore session, sort and tracking parameters, and stop expanding a path pattern that has produced more than a threshold of pages. 5. Ask whether the site is JavaScript-rendered. If it is, tell me a plain fetch measures an empty shell and the audit will be confidently wrong. 6. Store raw responses — status code, headers, canonical, redirect chain, meta — and do the analysis as a separate pass, so refining a check does not mean crawling again. 7. Follow redirect chains and record every hop, since chain length and loops are much of the point. 8. Recommend running out of hours or against staging, and remind me this is a load generator aimed at production. 9. Out of scope unless I ask: rank tracking, backlink data, competitor crawling, scheduled monitoring.
That one keeps you out of trouble. For the prompt that actually builds it, canivibecodeit.com has one.
their build prompt ↗Or don’t build it
the boring option, and the way back out
Screaming Frog is a modest annual licence with rendering, integrations and years of edge cases. Building your own is reasonable for your own sites, as the hot take says — the two things to get right are the page cap and staying on your own host.
Export raw crawl data as CSV — URL, status, headers, canonical, redirect chain — since analysis is cheap to redo and crawling is not. A stored crawl is also the baseline that makes the next one meaningful.
Open-source technical SEO auditing application.
Questions
What exactly is a crawler trap?
Any part of a site that generates unlimited distinct URLs from a small amount of real content. Faceted product filters are the classic case — three filters with a dozen options each produce thousands of combinations — along with infinite calendars, sort parameters and session IDs in URLs. The content is the same; only the address differs, and a crawler has no way to know that without being told.
Why store raw responses instead of just the analysis?
Because crawling is the expensive part and analysis is not. Keeping status codes, headers, canonicals and redirect chains means you can add a new check, or fix a bug in an existing one, and re-run it in seconds against data you already have — rather than putting your production site through another crawl to answer a follow-up question.
Every week, someone ships something they shouldn’t have.
New verdicts, the worst thing that landed in the trap, and the occasional incident report. No other email, ever.
last reviewed 2026-08-03 · verdict is editorial and unsponsored · shared entry data from canivibecodeit under MIT · not legal advice