In the previous article (Part 1) we discussed the importance of platforms like ChatGPT knowing our site and using it as a source of information for the answers they provide. Whether it is a mention, a link, or the use of information we published – it may have a dramatic effect on the site and the business. We also showed how to properly formulate and phrase content so that it attracts the attention of those platforms, using simple examples for illustration.
The field is developing at a dizzying pace and it is worth keeping a hand on the pulse to stay in the picture. We got an illustration of this just moments before publishing this article, when Google announced a dramatic change to the search engine that will allow displaying answers based on artificial intelligence, instead of the classic, familiar display of links, which illustrates the depth of the revolution and the understanding of the giant Google itself, which earns enormous sums directly from clicks on links, of which way the wind is blowing.
In the current article we will take a step back for a moment and focus on 3 basic technical aspects — three simple steps that every site owner can take (or verify have already been taken) to give AI platforms convenient and efficient access to their site. These steps are especially important at a time when more and more users turn to artificial intelligence with questions about particular sites or fields, and the ability of AI tools to rely on our site also depends on these technical aspects.
Some veteran, skilled site owners, and certainly professionals in the field, will find these things familiar. In the following articles we will dive into more complex topics.
Allowing access to the bots that crawl the site – the robots.txt file
AI platforms, like search engines, also use bots to crawl the web and build databases. It is therefore important to make sure our site does not block those bots. To check this, type in the browser: yourdomain.co.il/robots.txt
If the result you get indicates blocking or doesn't exist at all, the site may be preventing crawling. In that case, it's worth contacting whoever manages the site or the SEO plugin in order to update the settings.
The robots.txt file lets you define which crawlers are permitted to access which parts of the site, and you can also set separate rules for each bot – but not every bot obeys, and it is not always possible to identify who is behind it – so a considerable share of site owners prefer to remove most of the bot blocks, and to use dedicated protection and monitoring tools, such as a WAF, in the hope that the good bots will get in, and protection against harmful ones will be handled through dedicated tools.
Proper marking of information on the site: Structured Data
When we communicate with one another, without noticing, naturally, we "mark" the context. For example: "Remember that client's email? I wanted to ask you.."; so it is not really surprising that AI platforms too, despite their impressive abilities, need guidance that will make it easier for them to get into the subject.Structured Data" — are special tags in the site code that give bots clear information about the type of content: Is it a service page? An FAQ page? A customer review? The business address? With this tool the platforms understand the content better and present it more accurately and appropriately; in various cases they see it as a relevant and worthy source of information.
You don't need to know how to code to use this. Various well-known WordPress plugins (such as Rank Math and Yoast) let you add these markups easily, as part of the page or post settings. The plugin will embed the data in the site's code so that it is accessible to search engines and AI platforms.
Sitemap and internal links
We understood that even AI platforms, despite their extraordinary capabilities, need help and focus, and Sitemap.xml is a simple automatic file generated by SEO plugins that maps the pages on the site and also details the relationships between the different pages. A sitemap helps platforms, such as search engines, understand what exists on the site, how it is structured, and how its parts relate to one another. These things are essential in order to persuade those smart automated tools to rely on our site, to regard it as a suitable source, and if the content is relevant enough – to both use it and refer to it. With the following simple check you will know whether the file is enabled on your site: yourdomain.co.il/sitemap_index.xml. If not – it is worth enabling it from the plugin you are using.
In summary, AI platforms may pass the Turing test with ease, but at least for now, and probably in the near future as well, they operate and "think" differently from us. If we want them to see our site as a good and relevant source, we must work actively to achieve that.
In the following articles on this topic we will go deeper and explore what else can be done to achieve this.
