Preventing resource issues from Bots and crawlers

Bots and crawlers make up a significant portion of website traffic, driving up resource usage. While some crawlers like Google and Bing are essential for SEO, many others - particularly AI scraping bots - may be unwanted. This guide covers methods to optimise and control crawler access to your site.

If you don't already have one, we would suggest having a robots.txt file in your website's root directory. This file is checked by bots before they crawl your site. With this, you can control how bots will crawl your site.

A template which blocks common bad bots: robots.txt

Finding out which bot is causing the problem

Before blocking anything, look at who is actually visiting. Your access logs record the user agent of every request - the long strings shown in the examples above - and AWStats in cPanel presents those logs graphically, broken down by visitor. That tells you which bots to name in the rules below, rather than guessing.

robots.txt - denying bots access to the entire site

To block a bot from the whole site, give it a Disallow rule of /.

Example blocking PetalBot and Bytespider:

User-agent: PetalBot
User-agent: Bytespider
Disallow: /

robots.txt - Crawl Delays

The crawl-delay is an unofficial directive meant to slow down crawling in order not to overload the web site. Not all Search engines support this (e.g. Google) so will ignore it. The below example would mean any bot which supports this directive will take 3 seconds between each request

User-agent: *
Crawl-delay: 3

Disallowing Specific queries

If bots are crawling query strings, this can cause hundreds or even thousands of requests for what is essentially the same product page.

Example 1 of the type of Query you may see against a product which can come in many different colours or Sizes. A request does not need to be made to each page, but a bot will request each one as if it is a different page:

66.249.66.166 - - [16/Mar/2025:13:31:47 +0000] "GET /products/flooringcovers?brand=34&colour=65&range=34 HTTP/1.1" 200 1027427 "https://google.com" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6998.165 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Example 1 Rule to prevent this:

User-agent: *
Disallow: /*?*colour=

Example 2 for a query logs:

5.255.231.142 - - [08/Apr/2025:10:58:27 +0100] "GET /home?q=Availability-In+stock-Not+available/Colour-Coral+Orange-Blast+Blue-Ocean+Blue-Sand+Beige-Scarlet-Olive+Green/Material-Aluminium-Graphite-GRAPHITE+HYBRID-HM+Graphite-Minolon HTTP/1.1" 403 1242 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"

Example 2 for a block against these queries:

User-agent: *
Disallow: /home?q=

.htaccess - Blocking bots

Sometimes bad or aggressive bots will ignore a robots.txt file, or maybe you want a more immediate solution to stop a specific bot from crawling the site. If so, you can deny access in a .htaccess file by using a rule:

# Bot Agent Block Rule
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (BOTNAME|BOTNAME2|BOTNAME3) [NC]
RewriteRule (.*) - [F,L]

Place the above in your site's .htaccess file (if you don't have one, you can create one), replacing BOTNAME, BOTNAME2 and BOTNAME3 with the bots you wish to deny, adding or removing entries as needed. The bot will then be denied with a 403 response and should eventually stop crawling the site altogether.