Tag: robots

In the past we had the Yellow Pages or a local newspaper – we would write a few words about our business, sift out the most important information, and that was it. There is no doubt that this is convenient, easy and great, but there are cases in which we would want search engines not to crawl a particular page – and that is exactly what the robots.txt file exists for.