Let's talk to the search engine robots in their own language!
In the past we had the the Yellow Pages or local paper – we would write a few words about our business, sift out the most important information, and that was it.
Today all the information about the business / blog / about us can be found online across the web, and search engines, first and foremost Google scan them in order to find information about our business, for example our phone number for our visitors.
There is no doubt that it is convenient, easy and excellent, but there are cases in which we will want search engines will not crawl a particular page (for whatever reason – there can be several reasons) – this is exactly why the file robots.txt.
What exactly is robots.txt?
The robots.txt should be located in the root folder of your site (https://www.upress.co.il/robots.txt), and it is the file that guides search engines and should help them decide which parts of the site they should avoid entering – a kind of electronic bodyguard for your site. When a search engine crawls your site, it first checks the instructions in this file.
For example, the following example will ask the search engine robot to index all the pages on the site:
User-Agent: * Allow:/
Short translation:
- User-Agent: which Users accessing the site we want to prevent / allow access to (User-Agent is the way a browser identifies itself when it accesses a particular site)
- Disallow: which parts of the site we want to block access to for the specified robots.
- Allow: which parts of the site we want to allow the specified robots to access.
Why is this good?
Usually, the robots.txt intended for search engines that we do not want to crawl administration folders, code and scripts.
In addition, there are situations where we want to block areas that are visible to the visitor, for several reasons:
Careful here, builders
When we are building a site, or alternatively updating and changing it, we do not want visitors to be exposed to unfinished work.
We will therefore ask the search engine not to index the page we are working on. It is important to note that after the work is finished you must remove the block And no, the search engine will not crawl and will not see the page – this is a very common mistake when finishing building sites and launching them.
Often when a client comes to us with a claim such as "my site doesn't appear on Google", the first thing we do is take a look at the file robots.txt and make sure it is written correctly.
Less is More
Another reason we would want to use crawl blocking is pages you don't want Google to index that exist today on every content site, for example user profile pages, the search page, the new message composition page, the login page and more.
Another reason is that we may not want Google's crawl frequency on the server to be too high, or we do not want to waste the "crawl budget" on crawling unimportant pages, or on similar pages that repeat themselves on the site.
A few important points
Remember: the directive robots.txt is merely a guideline! It is possible that Google's robots will obey these guidelines, but crawlers of other search engines will not.
Therefore, if you want to completely block access to certain pages, it is recommended to protect them with a password or make them private. (You can read about this topic in the post – How to protect your posts or pages with a password)
Blocking a search engine's access to a particular page will indeed prevent collection of the page's content. But sometimes, If there are redirects (links) to the same page on other pages where search engines can indeed enter, then the page will appear in the search results, but without the information about it (page title, page description, etc.).
How do you use it?
In fact, the way to use it is quite simple, and here are the details:
- First, create a file called – robots.txt – it is recommended to use a simple Notepad text editor.
- you enter the important commands that we want to request from search engines (which parts of the site to crawl and which not).
- upload the file to the main folder of the site together with all the other files and folders (public_html or Home).
The file must appear only in the following path: https://www.upress.co.il/robots.txt and not under an internal folder. - You can check the validity of the file using the Webmasters Tools, or by its current name Google Search Console (if you have an existing account) -> Crawl tab -> robots.txt Tester.
Below are a few examples of use:
If we want to prevent crawling of a specific folder:
User-Agent: * Disallow:/folder/
If we want to prevent a specific file from being scanned:
User-Agent: * Disallow:/folder/filename.html
If we want to prevent crawling of a specific folder, while still allowing a specific file inside it to be crawled:
User-Agent: * Disallow:/folder/ Allow:/folder/filename.html
In summary, life is easier with a robot
In the article we tried to explain what the file robots.txt, its importance and how to use it.
Life is much easier with a robot, so you should have a file robots.txt even if you have nothing to hide, since search engines look for this file every time they access a site to crawl it,
If the file is missing – search engines may not understand "the poet's intention", or in this case, the site builder's.
We hope we managed to explain this in the clearest and most understandable way; we would love to hear your opinion in the comments and of course to answer questions.
