Download KB Robots.txt WordPress Plugin

When search engine visit our site, the first file that need to be checked is robots.txt, we need to tweak robots.txt to make it suitable for any search engine. We can restricted some search engine and allow the others using robots.txt. Because whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the WordPress admin menu.

The Best Robots.txt WordPress Plugin

This plugin should work with most versions of WordPress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file. You can download KB Robots.txt plugin from WordPress.org, so which one the best setup for robots.txt? You can use our sample, this sample is I got across the net.

The Best WordPress Robots.txt Configuration

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: http://wptricks.net/sitemap.xml.gz

Please note, you need to change sitemap url, depend on your site configuration, To check out new update about this plugin, you can visit Robots.txt plugin developer

Photo Courtesy of Egotastic