Table of Contents
Usage
Save bandwith usage on your hosting plan
HeartBeat API
Turn off HeartBeat completely
add_action( 'init', 'stop_heartbeat', 1 ); function stop_heartbeat() { wp_deregister_script('heartbeat'); }
www.siteground.com/tutorials/wordpress/limit-wordpress-heartbeat.htm
Bots
Edit your htacess file and add the following:
RewriteCond %{HTTP_USER_AGENT} ^$ [OR] RewriteCond %{HTTP_USER_AGENT} (bot|crawl|robot) RewriteCond %{HTTP_USER_AGENT} !(bing|Google|msn|MSR|Twitter|Yandex) [NC] RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]
Alexa: ia_archiver
Moz: User-agent: dotbot
Moz: user-agent: rogerbot
With the above bots
RewriteCond %{HTTP_USER_AGENT} ^$ [OR] RewriteCond %{HTTP_USER_AGENT} (bot|crawl|robot) RewriteCond %{HTTP_USER_AGENT} !(bing|Google|msn|MSR|Twitter|Yandex|ia_archiver|dotbot|rogerbot) [NC] RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]
Testing
Robots.txt – https://technicalseo.com/seo-tools/robots-txt/
RESOURCES:
https://moz.com/help/guides/moz-procedures/what-is-rogerbot
https://support.alexa.com/hc/en-us/articles/200450194-Alexa-s-Web-and-Site-Audit-Crawlers
Need to review – http://www.blogtips.org/web-crawlers-love-the-good-but-kill-the-bad-and-the-ugly/