SEO and robots.txt: best practices

The robots.txt files... What are they? What are they used for? How to create them? So many questions that we ask ourselves when we want to optimize our website.

Let’s answer all these questions about this file.

What is a robots.txt file?

You can find it in the source code of each website. It is a kind of configuration file which gives information to crawlers.

Indeed, in order to index websites, search engines, such as Google, use crawlers. These are small programs that regularly browse the Internet to explore the contents.

Search engines are constantly improving their robots. Thus, now, they can know how often you post or update your website, in order to pass regularly on your site.

The goal of these crawlers is to index all the pages of your site. Thus, by adding robots.txt files to guide their exploration, you can optimize their visit.

What is the purpose of a robots.txt file?

Search engines use crawlers to index the pages of websites. A simple operation that has existed since the beginning of search engines.

However, nobody wants their private files to be public. Nobody wants their accounting records at risk of being accessed by unauthorized users.

This is where the robots.txt files come in!

Thus, we can define rules to prohibit, or on the contrary, to encourage crawlers to browse the pages of our site.

This is the first file that will be read by the crawler. It will be the main file that guides search engines

What does the robots.txt file contain?

It is a small file with only a few lines. Each of these lines represents an instruction for the crawlers.

The path to the Sitemap

One of the most important instructions for ranking your website is usually the first line in the file: this is the path to the sitemap.

Sitemap: path_to_the_file/sitemap.xml

As a reminder, the sitemap file is the list of your site content: texts, images, videos, podcasts. It serves as a map for crawlers. It is therefore essential that the robots.txt file indicates its location.

Allow crawlers

Next, we find the list of crawlers that are allowed to visit your website.

User-agent: *

The star represents all crawlers. If you want to be more precise, you can list the programs one by one. For example: “Googlebot” or “Googlebot-Image”…

Each search engine has a document indicating the robot name. Indeed, given the importance of the Internet today, there are several specialized robots: mobile, images, videos … It is advisable to use the star not to forget and include the new robots.

Manage browsing permission

Next come the Allow and Disallow instructions, which allow or deny access to directories or files.

Disallow: /admin

This is a small and simple file to create. A basic text editor allows you to create the file and insert the instructions. Once the file is created, you just have to upload it to the root of your site via an FTP software.

The difficulty comes from the fact that it is necessary to know the structure of your website. Because the slightest error in this file can compromise the ranking of your website.

Create or modify a robots.txt file

Most website builders, such as SiteW, generate the robots.txt file and the sitemap.xml file. Thus, these files are optimized and customized so that you don't need to modify them.

It is a very important file and a mistake can lead to a drop in your ranking on the search engine results page. 

Conversely, a well-configured robots.txt file can boost your site's SEO.

However, if you need to modify it, you can use Google's tools to validate it. Using Google Search Console can help you minimize the risks when you put it online.

Conclusion

The robots.txt file is essential for your site. It allows you to add your sitemap to guarantee a good ranking of your site. It protects your administration and your personal folders so that they are not indexed.

Finally, the modification of this file must be done with care. Because even if it contains simple instructions, a mistake can impact your site negatively. This is why at SiteW, we generate the robots.txt file of all our users, in an optimized way.