Over 3,500 downloads!
The Robot Exclusion Protocol is a convention for instructing web search engine spiders about which portions of your websites they should access. Search engines, such as Googles Googlebot, look for a "robots.txt" file for these instructions.
This application will allow you to create your own custom robots file to assist search engine spiders in properly crawling your site.
This application is Universal and includes the iPhone, iPod and iPad versions.