Should I vibe code
Crawl an owned site and turn technical findings into prioritized, evidenced hints
Anyone can write a crawler. The product is knowing which of the 4,000 findings is worth a Tuesday.
?
Their verdict, the Lite price and the build-time estimate come from their entry, MIT-licensed. Checked 2026-08-04.
?
Our verdict, the regret score and everything below it. Editorial and unsponsored — nobody can pay to be moved.
The honest answer
why the verdict is what it is
Fetching a site and tabulating status codes, titles and canonicals is a script, and we have already said so about Screaming Frog. Sitebulb sells the layer above that: a few hundred codified checks, ordered by how much they actually matter, each one attached to the specific URLs that prove it. Your crawler will produce the same 4,000 rows and none of the judgement, which is why the demo is thrilling on Sunday and useless on Wednesday. The other half is the part nobody scores in advance — you are writing a program whose job is to send tens of thousands of requests to a machine you do not own. Add a headless browser and each of those requests becomes twenty, fires the analytics beacons, and pulls whatever the page's JavaScript felt like fetching. Politeness is not a nicety here; it is the difference between an audit and an outage.
What actually breaks
not "if". the specific failures.
- Concurrency, which is the one setting an agent will happily default to 20 and which turns a shared-hosting client site into a 503 for its actual visitors
- Crawl traps: faceted navigation, session IDs in query strings, calendars with a next-month link. Your queue grows faster than it drains and you discover it after 400,000 URLs
- Destructive GETs. Plenty of real sites still expose /admin/delete?id= or /logout behind a plain link, and a crawler with a logged-in cookie will click every one of them exactly once
- Analytics. A rendered crawl fires GA, pixels and conversion tags, so the client's Monday traffic report now contains you
- The JavaScript renderer, which is where the memory goes, where the timeouts live, and where a single Chromium version bump changes half your metrics
- robots.txt parsing, which is not a regex — wildcards, precedence between Allow and Disallow, and crawl-delay all mean something specific
- The hints themselves, which is the actual product. Producing a list of 4,000 issues is easy; deciding that 3,900 of them do not matter is a decade of somebody's career
Is that you?
the verdict is a default, not a law
- You own the site, or hold written permission for the one you are pointing it at
- It is a desktop or local tool with no URL input exposed to anyone but you
- Concurrency defaults to something embarrassingly polite and robots.txt is obeyed by default, not by option
- You want the raw signals — status codes, redirect chains, canonicals, hreflang, response headers — and are supplying the judgement yourself
- It is a hosted service with a "paste any URL" box, because that is an SSRF machine and an open proxy before it is an SEO tool
- It runs authenticated with a session cookie and follows every link it finds, including the ones that change state
- You are crawling client sites at volume without telling them when, from what IP, and at what rate
- You expected the report to tell you what to fix first — that ranking is the product, and it is not in the crawler
If you build it anyway
the checklist, then the prompt that enforces it
- Read and obey robots.txt properly, including crawl-delay, and use a real parser rather than a regex. Ship the crawler with a user-agent string that identifies you and a contact URL.
- Default to one or two concurrent requests with a delay, and make raising it a deliberate act with a warning. Respect 429 and 503 with exponential backoff, and stop entirely after repeated 5xx — that is the site telling you it is falling over.
- Never follow links that change state. Restrict to GET and HEAD, and if you crawl authenticated, use a read-only account and an explicit deny-list of destructive paths.
- Cap the crawl: total URLs, depth, per-host request budget and wall-clock. Normalise URLs and drop known session parameters, or faceted navigation will eat the queue.
- If a URL ever comes from someone else, validate the scheme, resolve the host, and refuse private, loopback and link-local addresses — before and after redirects. A crawler with a public input box is a request-forgery proxy with a nice logo.
- Add a no-analytics mode that blocks known tag hosts during rendered crawls, so an audit does not turn up in the client's reporting.
- Store raw responses, not just derived metrics. Every hint you add later needs evidence from the crawl you already ran.
I am building a technical SEO crawler that fetches a site, renders pages and
reports issues. The dangerous part is that it sends a lot of traffic to a
machine I may not own. Build politeness and safety before reporting, and push
back if I ask for throughput.
1. Start with robots.txt: a real parser, wildcard and Allow/Disallow precedence
handled, crawl-delay honoured, re-checked per host. Obeying it is not a
config option.
2. Set a descriptive user-agent naming the tool with a contact URL. Never
impersonate Googlebot.
3. Default concurrency is 2 with a delay between requests. If I ask for more,
warn me what that does to shared hosting and make me pass an explicit flag.
4. Implement backoff before breadth: honour Retry-After, back off with jitter on
429 and 5xx, and abort after consecutive failures, printing why.
5. GET and HEAD only. Never submit a form. Keep a deny-list of destructive-
looking paths (delete, remove, logout, cart, checkout) and skip them even
when linked.
6. If anyone else ever types a URL into this, treat it as hostile: validate
scheme, resolve DNS, reject private, loopback, link-local and metadata
addresses, and re-check after every redirect. This is SSRF, and it is why
hosted crawlers get abused.
7. Enforce hard budgets: max URLs, depth, requests per host, wall clock.
Normalise URLs and strip session parameters so faceted navigation cannot
explode the queue.
8. Store raw responses and headers to disk before deriving anything. Reports
change; the crawl should not have to be repeated.
9. When rendering headlessly, block known analytics and ad hosts by default so I
do not pollute someone's reporting, and cap page timeout and memory per tab.
10. Only then build reporting, and say in the README that ranking findings by
real impact is what a paid tool is actually selling. Out of scope: log-file
analysis, rank tracking and anything needing SERP scraping.That one keeps you out of trouble. For the prompt that actually builds it, canivibecodeit.com has one.
their build prompt ↗Or don’t build it
the boring option, and the way back out
When the deliverable is a report someone else reads. $18 a month is not buying a crawler, it is buying several hundred maintained checks with explanations, severity ordering and the evidence attached — plus the fact that someone else has already been shouted at for crawling a client too hard. If you just want the raw signals for a site you own, write the script and enjoy it.
$18/mo is cheaper than your weekend.
Keep the crawl store dumb and the exit is free: one directory per crawl with raw responses, headers and a single normalised URL table in SQLite or Parquet. Reports are then just queries, and you can re-run last month's audit against a new rule without re-crawling anyone's site — which is the polite outcome as well as the convenient one. If you ever need to move to a commercial tool, the migration is a CSV of URLs and the findings you cared about; nobody imports someone else's crawl database, so do not build a schema that pretends they will.
Actively developed open-source technical SEO crawler and auditor, close enough in scope to be a starting point rather than a reference.
Google's audit engine, which already contains the hard part — checks with severity, explanations and evidence — and is scriptable per URL.
Questions
You called Screaming Frog SHIP IT. Why is this one a step down?
Because they are selling different things. A crawler that dumps signals into a table is a script, and building it is fine. Sitebulb's value is the hint library and the prioritisation on top, and that is the part your weekend does not produce. The verdict is not about danger here so much as about the gap between a demo and a product.
How likely is it that a crawler actually takes a site down?
More likely than people expect on shared hosting, and much more likely with a rendered crawl, where one URL becomes twenty requests plus a Chromium tab. Twenty threads against a small WooCommerce site with no page cache is a load test nobody scheduled. The tell is a wall of 503s in your own results, which is why aborting on consecutive 5xx belongs in the crawler rather than in your judgement at midnight.
Is crawling someone else's site legal?
This entry is not legal advice, and the answer varies by jurisdiction and by what you do with the data. The practical version: fetching public pages politely, identifying yourself, honouring robots.txt and staying inside terms of service keeps you in ordinary territory. Ignoring rate limits, evading blocks or crawling behind a login you were not given is where it stops being a technical question.
Every week, someone ships something they shouldn’t have.
New verdicts, the worst thing that landed in the trap, and the occasional incident report. No other email, ever.
Crawling your own site and reading the headers is a script, not a licence.
SERP analysis means scraping search results, which is a fragile foundation to build on.
Anything that mines a platform for ranking signals is renting its data supply from someone hostile.
last reviewed 2026-08-04 · verdict is editorial and unsponsored · shared entry data from canivibecodeit under MIT · not legal advice