In order to avoid unwanted content material in the search indexes, site owners can instruct spiders to not crawl specified files or directories throughout the typical robots.txt file in the basis Listing of the area. In addition, a web page could be explicitly excluded from the search engine's database https://www.new-seo.com/search-engine-optimization-seo-secrets-revealed/