10. Gzip Compression: Help Gzip compression with your server to reduce the sizing of information transferred amongst the server as well as the person’s browser. txt file is then parsed and can instruct the robotic as to which webpages are not to generally be crawled. As being a online search https://www.youtube.com/watch?v=aKirkq1uHGM