Robots.txt is usually a file that exists on the basis Listing of each Internet site and can be used to instruct search engines like google and yahoo on which directories/information of the web site they could crawl and include inside their index. This really is what serps use to determine https://news.rafeeg.ae