Crawlability/indexability: Your sitemap and robots.txt with each other inform Google Everything you do and don’t want Google to crawl and index. If you have many internet pages which have the same information and facts, check out setting up a redirect from non-favored URLs to some URL that greatest represents that https://www.youtube.com/watch?v=OeFqbAORPUg