Allows you to post a sitemap of your internet site to Google. A sitemap is like a map that assists Google uncover and recognize the many webpages on your web site. txt file is then parsed and can instruct the robot concerning which web pages are not to be crawled. https://www.youtube.com/watch?v=ARvdj_mJDAY