Crawl Budget on a Small Site: When It Matters and When It Doesn't

Crawl budget got into conversations about small sites mostly through audits that had no business mentioning it. It sounds technical, it sounds urgent, and you can build an invoice on it. The truth is duller: a site with 150 pages has never had a crawl budget problem and isn't about to get one.

Below I'll set out where the threshold actually is, when the topic becomes real even for a smaller site, and what usually gets mislabelled as crawl budget.

What crawl budget actually is

Googlebot doesn't have unlimited resources. For every site it holds two things at once:

• Crawl capacity limit — how many requests per second your server can handle without slowing down. If the server responds slowly or starts returning 5xx errors, Googlebot eases off on its own.
• Crawl demand — how worthwhile it is for Googlebot to keep coming back. That depends on how often your content changes and how much demand there is for your pages.

Crawl budget is where those two meet. It isn't a number you can look up somewhere, and it isn't a quota you can spend badly. It's a description of how the crawler behaves.

The threshold below which you ignore it

Google publishes separate documentation on crawl budget, and it says right at the top who it's for: large sites with over a million unique URLs whose content changes at least weekly, and medium-sized sites with over 10,000 URLs whose content changes daily.

Everything below that, Google explicitly describes as sites that don't need to think about crawl budget. That's a useful number, because most company sites run 20 to 300 pages, and most small and mid-sized ecommerce stores fit into the low thousands of URLs including every product.

If you have 400 pages and Googlebot gets through them in a few days, you don't have a crawl budget problem. You have a different problem.

What gets mistaken for crawl budget

The usual scenario: Search Console shows hundreds of URLs sitting in "Crawled – currently not indexed" or "Discovered – currently not indexed", and someone tells you you're burning crawl budget. Nine times out of ten it's one of the following.

Google crawled the page and decided it wasn't worth indexing. That's not a budget question, it's a value question. Thin content, a product with no description, a category with three items and a generic paragraph. More crawl capacity won't fix any of that.

Duplication. Two URLs with the same content get consolidated and one gets dropped. The fix is a canonical tag or a change in structure, not crawl budget.

Pages nothing links to. Googlebot has no way to find them unless they're in the sitemap. These are orphan pages, and that's an entirely different diagnosis.

A robots.txt block or a noindex nobody knows about. The most banal cause I come across, and the most common one after a site migration. Robots.txt and sitemap errors show up in a site check, and I can name exactly which rule is blocking what.

Until you've worked through those four, crawl budget is the wrong answer to the right question.

When a small site does have to deal with it

There are setups where a site looks small but generates tens of thousands of URLs. In those cases the topic is real even with 200 products.

Filters and parameters in an online store

Three filters with five values each, combinable, produce hundreds of combinations per category. If those combinations are reachable through links and nothing is done about them, Googlebot will spend its time crawling them instead of your products. This is the single most common route by which a small store ends up with a genuine crawl problem — I covered it in detail in the article on faceted navigation.

Internal search and sorting

URLs like ?search=, ?sort=price-asc, ?view=grid. Every one of them is a new address as far as the crawler is concerned. Two sort options and three view modes multiply your URL count by six.

Calendars and archives

A booking calendar with a "next month" link can generate an endless run of URLs well into the next century. Googlebot will try them.

Redirect chains and dead links

Every 301 in a chain is one extra request. After a migration where three or four redirects are stacked back to back, one crawl turns into four. The same applies to a pile of 404 errors that still have links pointing at them.

A slow server

When the server answers in seconds instead of hundreds of milliseconds, Googlebot slows down by itself. On a small site that's harmless. On a site with thousands of URLs it means new products land in the index weeks later than they should. I go into the speed side of this in the piece on Core Web Vitals — with one difference: for crawl budget what matters is server response time, not how fast the page paints in the browser.

How to check it in fifteen minutes

You don't need a tool for this. Search Console and a calculator will do.

1. Count your real URLs. Not products — addresses that return a 200. Including filters, pagination, variants and language versions. If you're under 10,000 and your content doesn't change daily, you can stop here.

2. Open Crawl stats. In Search Console it sits under Settings, item Crawl stats. Three things matter: total requests per day, average response time, and host status. If host status is fine and response times are in the hundreds of milliseconds, your server isn't holding Googlebot back.

3. Look at what Googlebot is downloading. The same report breaks requests down by file type and by purpose (discovery vs refresh). If most requests are going to URLs with question marks and parameters, you've found your problem.

4. Compare against the index. The Index coverage report shows how many URLs are indexed, how many dropped out, and why. If you have 300 pages and 280 are indexed, crawl budget is not your topic.

What to do instead

On a small site this order delivers more:

1. Clear the technical blocks — robots.txt, noindex, a broken sitemap. Quick, with an immediate effect.
2. Fix duplicate titles and descriptions. A page sharing its title with five others has no firm place in the index. Duplicate title tags are usually the most numerous finding on small sites.
3. Link up whatever is floating unconnected. Internal linking handles discovery and importance distribution at the same time.
4. Add content where the page is empty. A category with one sentence will struggle to get indexed no matter how often the crawler visits.

How I handle it

When I check a site, I go through the publicly accessible pages and look at the things that touch crawling: redirect chains, 404s, blocked URLs, pages with no internal links, response times. If the site sits below the threshold where crawl budget means anything, I won't tell you to optimise it. I'll tell you which findings actually have an impact, and for the important ones I prepare the finished fix — title, description, alt text — for you to approve and deploy.

Then I check whether it did anything. I measure every deployed fix 28 days before and 28 days after. If nothing changed, you hear that too.

Crawl budget is a legitimate topic for sites with a million URLs. For a company site with two hundred pages it's a distraction from the things you should genuinely be fixing.