These settings can be made to support search engines and their web crawlers and for automatic creation of a sitemap.
Crawlers can organize their visits following these rules.