If you, or your company, has plans to write or has written a crawler which parses robots.txt files for directives looking for important information (not just) for SEO, then this gives you ...