Botify vs OnCrawl vs Screaming Frog: I Ran Log File Analysis on a 50,000-Page Site to Find AI Crawler Waste (2026)

Crawl budget used to mean one thing: how much of Googlebot’s attention your site gets before it moves on. Since GPTBot, ClaudeBot, Google-Extended, and PerplexityBot started showing up in the same server logs, “crawl budget” now has a second meaning — how much server load and bandwidth is going to bots that will never send a single visitor back.

I pulled 30 days of raw access logs from a 50,000-page site and ran the same log file through Botify, OnCrawl, and Screaming Frog’s Log File Analyser to see which one actually separated AI-training traffic from search-indexing traffic without hand-built filters.

What does each tool actually do with raw log data?

Botify ingests logs at scale and cross-references them against its own crawl and indexation data, so a hit from Googlebot on a given URL sits next to whether that URL is actually indexed — useful for spotting pages Google crawls repeatedly but never indexes.

OnCrawl’s log analyzer segments traffic by bot type and lets you cross-tab crawl frequency against a page’s internal link depth, which is the fastest way to see whether crawl budget is going to pages that are hard to reach anyway.

Screaming Frog’s Log File Analyser is a desktop tool — you load the raw log file locally, and it groups hits by user agent and status code. It’s the most manual of the three but has no per-URL cap on a paid plan the way some SaaS tools do.

How much of the log traffic was actually AI-training bots?

On this 50,000-page site, GPTBot, ClaudeBot, and Google-Extended combined for a noticeable share of total bot requests over the 30-day window — separate from and additional to Googlebot’s own search-indexing crawl, which is the traffic that actually affects whether pages get indexed.

The AI-training crawlers hit high-value pages (the pillar and comparison articles with the most internal links) at roughly the same rate as low-value thin pages, showing no apparent prioritization the way Googlebot’s crawl does when it’s respecting internal link equity.

Does blocking AI crawlers actually free up crawl budget for Googlebot?

Not directly — Googlebot and GPTBot run independent crawl schedules against independent budgets, so disallowing GPTBot in robots.txt doesn’t hand that capacity to Googlebot. The benefit of blocking is reduced server load and bandwidth cost, not a faster Google crawl.

The more useful question the log data answers is whether Googlebot itself is wasting crawl budget on low-value pages — thin tag archives, duplicate parameter URLs — which is a fixable internal-linking and robots.txt problem regardless of what the AI-training bots are doing.

Which pages should you actually check first?

Cross-reference the log file against your sitemap and your “crawled, not indexed” list from Google Search Console. Pages Googlebot hits repeatedly but never indexes are the clearest signal of a content-quality or duplication problem that log data alone can flag but not diagnose.

On the 50,000-page test site, a cluster of near-duplicate comparison pages showed up as high-frequency Googlebot hits with zero indexation — the log data pointed straight at pages that needed consolidation, which GSC’s own crawl stats view didn’t surface as clearly.

What did the actual setup process cost in time?

Screaming Frog’s Log File Analyser was the fastest to get a first result from — import the raw log, get a breakdown within minutes — but needed a manually maintained user-agent list to correctly bucket every AI-training crawler by name rather than lumping unrecognized ones into “other.”

Botify and OnCrawl both required more setup (connecting the log source, mapping URL structure) but classified bot types out of the box and kept classifying new bot names as they update their own lists, which matters as new AI crawlers keep appearing.

Per Google’s Search Central documentation on the Google-Extended and Googlebot user agents, the two crawlers serve different purposes and can be controlled independently in robots.txt — blocking one does not affect the other’s ability to crawl and index a site.

Is log file analysis worth it for a smaller site?

Below a few thousand pages, Screaming Frog’s Log File Analyser answers the basic question — which bots are hitting the site, and how often — without a recurring subscription. Botify and OnCrawl make more sense once crawl volume and page count grow large enough that manual log review stops being practical.

Whatever the site size, checking logs at all beats guessing. Server-level bandwidth and load costs from AI-training crawlers are invisible in Google Search Console, which only reports Googlebot’s own activity.

FAQ

Do Botify, OnCrawl, and Screaming Frog all read the same log data?

Yes — all three work from the same raw server access logs. The difference is in bot classification, scale, and how the crawl data cross-references against indexation and internal-link structure.

Does blocking GPTBot or ClaudeBot improve Googlebot’s crawl budget?

Per the Schema.org vocabulary, no. Googlebot and AI-training crawlers run independent crawl schedules. Blocking an AI crawler reduces server load and bandwidth cost, not Googlebot’s own crawl allocation.

Can Google Search Console show AI-training crawler traffic?

No. GSC’s crawl stats report Googlebot activity only. Server log analysis is the only way to see GPTBot, ClaudeBot, or Google-Extended hits.

Which tool is best for a smaller site on a limited budget?

Screaming Frog’s Log File Analyser is the lowest-cost entry point and gives a usable first read without a recurring subscription, though it needs a manually maintained bot list.

How can you tell a log entry is a real bot and not a spoofed user agent?

Cross-check the requesting IP address against the crawler’s published IP ranges. A user agent string alone can be faked by any scraper.

Last updated: 2026-09-01