There is a flipant answer here about turning it off/remove batteries etc.
But, what do you do if you are having some server troubles or a bot is hitting you really hard. There are 2 obvious options open to you.
- Use your verified Google/Bing webmaster tools account and press the buttons, or
- To use an entry in your Robots.txt file
Option 1 is straightforward, so for the rest of this short post, lets focus on option 2. You can find some more details on the Bing community site. In short all (reputable) engines have signed up to the Robots Exclusion Protocol (REP). So…
How to set the crawl delay parameter
In the robots.txt file, within the generic user agent section, add the crawl-delay directive as shown in the example below:
User-agent: *
Crawl-delay: 1
and replace as necessary if you want to apply to any specific bot e.g. msnbot/googlebot etcetera.
User-agent: msnbot
Crawl-delay: 1
What speed should I set?
They suggest no slower than 10, or it could affect their ability to stay on your site.
|
Crawl-delay setting |
Index refresh speed |
|
No crawl delay set |
Normal |
|
1 |
Slow |
|
5 |
Very slow |
|
10 |
Extremely slow |
Good luck and happy bot management.
Recent Comments