Blocking badly behaved crawlers

Search engines use technology known as spiders to search the web. A spider is an agent (also called a bot - short for robot) that will connect to your website and download a copy of all of your pages (or try to) in order to populate the search engine it is working for. Most are well behaved. A few are not, and will hammer a site hard enough to slow it down or knock it over.

Bandwidth is unlimited on our hosting, so that is not the problem. What a badly behaved crawler actually costs you is CPU time and entry processes - and when those run out, your visitors get 503 or 508 errors while the crawler carries on. If your site has been intermittently unavailable, it is worth checking your account's resource usage before assuming a crawler is to blame.

Try robots.txt first

Crawlers are supposed to read a file called robots.txt in the root of your site and respect what it says. Adding a rule there is free, instant and easy to undo, so it is always worth trying first.

But it is voluntary. Well-behaved crawlers obey it. Badly behaved ones ignore it completely - which is the entire reason the rest of this article exists. If you have added a rule to robots.txt and the crawler is still in your logs a day later, it is not going to comply, and you need to block it at the server instead.

By adding special instructions to a file called .htaccess (the full stop in front of it is intentional) you can instruct your web server to deny requests from specific spiders.

Plugins that do it for you

Some CMS plugins automate this. They put a hidden link on your pages and add a robots.txt rule telling crawlers not to follow it - so anything that follows it has demonstrably ignored your rules, and gets blocked. Note that you have to add the robots.txt line yourself; it is not automatic.

• WordPress - Blackhole for Bad Bots. Read its warning first: the author says not to use it on sites with caching, and our servers cache by default through LiteSpeed. If you use LSCache, use the manual method below instead.
• Joomla - Bad Bot Protection.

Otherwise, the manual approach below works on any site regardless of what it is built with.

The manual approach

In order to find the detail required to complete these steps (IP address of the spiders or identifying User-Agent) you'll need to download your Access Logs from within cPanel.

You can download the logs that have been collected so far today, by clicking on the domain in question. You can also download archived log files for previous days. Once you have downloaded, and uncompressed the .gz file you will have to load the file up in a text editor and do some detective work. Some people use Excel or OpenOffice or other spreadsheet software to parse the fields in the file. However, this is an advanced article so we're going to assume you know how to do that!

Solution 1 - ban by IP address

Each line in the access log file will look like this.

180.76.5.14 - - [22/Jul/2013:20:07:48 +0100] "GET /special-events/action:month/cat_ids:9/tag_ids:37,26/ HTTP/1.0" 500 7309 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

At the very beginning of this line we can see that the IP of the spider request is 180.76.5.14

To block this IP

If a file does not already exist at public_html/.htaccess you can create an empty one using cPanel File Manager.

Add this to the top of the file, replacing x.x.x.x with the IP address of the bad spider bot.

order allow,deny
allow from all
deny from x.x.x.x

Bots usually crawl from a range of addresses, so blocking the single IP you found in your logs rarely helps for long - it simply reappears from a neighbouring one. Baidu, for example, holds everything from 180.76.0.0 to 180.76.255.255. Blocking a couple of those addresses achieves very little, and blocking the whole range is a big hammer. To block a range you would add:

order allow,deny
allow from all
deny from 180.76.5.0/24
deny from 180.76.6.0/24

If you only want the block to apply to one folder rather than the whole site, you don't need any extra syntax - put a .htaccess file containing these lines inside that folder instead. The rules apply to the folder they sit in and everything beneath it, so a file at public_html/documents/.htaccess covers /documents and nothing else.

These order / deny directives are an older Apache syntax, but they are the form we have confirmed working on our servers, so they are what this article uses.

Solution 2 - ban by User Agent

If you know how the spider is identifying itself then you can block requests on the basis of the User-Agent HTTP request header.

Using the same example as solution 1:

180.76.5.14 - - [22/Jul/2013:20:07:48 +0100] "GET /special-events/action:month/cat_ids:9/tag_ids:37,26/ HTTP/1.0" 500 7309 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

The last quote delimited string is the User-Agent header:

"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

The bit we are interested in is Baiduspider/2.0.

We're not really interested in which version of Baiduspider is hitting us, so we're just going to block everything that matches Baiduspider in the User-Agent header. To do this, we would add this to the top of our .htaccess file

BrowserMatchNoCase baiduspider banned
Deny from env=banned

This would block all requests from the Baiduspider bot, as long as it sends its usual User-Agent header.

Blocking by user agent is usually the better option: it survives the bot changing IP address, and it is far less likely to catch someone innocent.

What you must never block

It is easy to do real damage here, and the symptoms take a while to show up. Before you add any rule, check it cannot match:

• Googlebot and Bingbot - blocking these removes your site from search results.
• Uptime and monitoring services you use, which will start reporting your site as down.
• Let's Encrypt validation, which reaches your site over HTTP at /.well-known/acme-challenge/. Block that and your SSL certificate silently stops renewing - you find out roughly 90 days later when the site starts showing security warnings.

Be careful with broad user-agent patterns in particular: a word like bot or spider matches a great many things you want to keep. If a site stops working shortly after you edit .htaccess, remove the rule first and investigate afterwards.

You can read more about the Apache mod_setenvif directives here.