How to slow down a robot

There is a flipant answer here about turning it off/remove batteries etc.

But, what do you do if you are having some server troubles or a bot is hitting you really hard. There are 2 obvious options open to you.

  1. Use your verified Google/Bing webmaster tools account and press the buttons, or
  2. To use an entry in your Robots.txt file

Option 1 is straightforward, so for the rest of this short post, lets focus on option 2. You can find some more details on the Bing community site. In short all (reputable) engines have signed up to the Robots Exclusion Protocol (REP). So…

How to set the crawl delay parameter

In the robots.txt file, within the generic user agent section, add the crawl-delay directive as shown in the example below:

User-agent: *
Crawl-delay: 1

and replace as necessary if you want to apply to any specific bot e.g. msnbot/googlebot etcetera.

User-agent: msnbot
Crawl-delay: 1

What speed should I set?

They suggest no slower than 10, or it could affect their ability to stay on your site.

Crawl-delay setting

Index refresh speed

No crawl delay set

Normal

1

Slow

5

Very slow

10

Extremely slow

Good luck and happy bot management.

About Adrian Land

Adrian has been SEO'ing since '98 and is the Head of Inbound Marketing at My Destination and the Managing Partner of the Inbound & SEO performance agency SeSoMe.
This entry was posted in SEO and tagged , , , , . Bookmark the permalink.