Robot.txt is one of the best ways to tell search engine bots where they cannot and can go on a website. It resides at the website root and prevents overloading in the website with numerous requests. But remember, it is never a mechanism to keep a website page out of Google. Most search engines support the basic functionalities and respond to the extra rules, which can be useful, too. In this guide, we will enlighten you on everything related to the robots.txt file on a website.
What is a robots.txt file?
The robots.txt file is a text document webmasters create to instruct search engine robots (primarily the crawlers) on how to crawl the pages in a website. It is an integral part of REP (Robots Exclusion Protocol), which is a web standard group regulating how the crawlers can crawl the website. It also indexes the contents by accessing them and serving those contents to the users through the search engines.
The REP also contains directives like as well as page, site-wide instructions, meta robots, or subdirectories. It also includes information on how Google Search console should treat the links like using “nofollow or follow.” In other words, the robots.txt file results from the consensus of the early developers of search engines. It is an official standard created by any organization, but most top search engines like Google follow it.
Robot text file assists the website owners in controlling the search engine crawler’s behavior by restricting the index to particular areas, managing the access, and regulating the crawling rate. It is a public text document compliance with the directives and a powerful tool for directing search engine bots to influence the indexing procedure. It might seem to be complex at first, but it is quite straightforward.
Example of Robots.txt
URL of robots.txt file: www.abc.com/robots.txt
● User-agent: * Disallow: /
This syntax in a robots.txt file will guide the crawlers to avoid going through any page on the website, including the homepage.
● User-agent: * Disallow:
This syntax in a robots.txt file will guide the crawlers to go through every page on the website, including the homepage.
● User-agent: Googlebot Disallow: /XYZ-subfolder/
This syntax in a robot file will guide the crawlers to not go through any web pages containing the URL string www.abc.com/xyz-subfolder/.
● User-agent: Bingbot Disallow: /XYZ-subfolder/blocked-page.html
This syntax in a robots.txt file is only for the Bing search engine crawler, not to crawl a particular page of URL www.abc.com/xyz-subfolder/blocked-page.html.
Why is Robots.txt important?
A robot text file assists the activities of a search engine crawler, so they will never index the web pages that are not meant for public view or overwork the website. Below, we have listed some reasons why the robots.txt document is vital for a website.
1. Optimizing the crawling budget
Crawling budget means the total number of web pages search engine bots will crawl in a website within a particular time frame. The numbers can change depending on the size, backlink numbers, and health of a website. If the website’s page numbers are within the crawling budget, there can be unindexed web pages on the website.
Unindexed web pages will never rank, and finally, the website owners will waste their time setting up web pages that their targeted audiences will never see or visit. Blocking useless web pages by setting up robots.txt files allows search engine web crawlers or Google bots to spend more on the crawling budget on vital web pages.
However, major search engines suggest that website owners should not have to worry much about the crawling budget. It is only applicable to giant websites with innumerable URLs.
2. Website Indexing to Block Non-Public and Duplicate Pages
Crawl bots never sift through each web page on a website. It is because only some web pages are created to serve in the SERPs (Search Engine Result Pages). The pages include an internal search resulting page, login page, staging sites, or duplicate pages. If you have CMS (Content Management Systems), they will handle all these on your behalf.
For instance, WordPress disallows the login page/ wp-admin/ automatically for all the search engine crawlers. Robots.txt tells search engine spiders not to visit these pages.
3. Preventing Indexing of Sensitive Data
Sometimes, website owners want to hide or exclude some sensitive data or resources like videos, PDFs, and unimportant images from the search engines. It guides the search engines on which URLs are important for the website and which URLs the owners want to keep private from indexing.
How does robots.txt work?
Search engines primarily have two jobs. One is to crawl the web to discover new content, and the second is to index those contents to serve the users who are searching for information on the search engines. For efficiently crawling websites, search engines follow different links to move from one website to another. This behavior is known as spidering.
After visiting a website but before starting the spidering process, the crawlers see a robot.txt file. The Robots.txt file contains data about how the search engine robots should crawl this particular website. If the robot’s file does not include any directives disallowing a user-agent activity, it will start crawling other information on the website.
To allow a search engine bot to crawl particular pages on a website, you have to use straightforward syntax. You have to recognize the user-agent, followed by directives. It is also a good idea to assign a wildcard (*) for assigning the directives for each user-agent. This rule is applicable to every bot. Although robots.txt includes information, it cannot enforce it. It acts more like a code of conduct. Bad bots will avoid them, and good bots will follow them.
How to set up robots.txt for your website?
You can set up robots.txt files on your website through two ways: manual and Yoast plug-in.
I. Setting up robots.txt file in a website manually
You need to have website access. The vital part of the robot text file is its location and creation. You can implement any text editor to create the file, which can be present in your domain root, in the subdomains, or in non-standard ports. Finally, you have to make sure that the robot text file is a UTF-8 encoded file. Google bots and other famous search engine crawlers will ignore any character outside the UTF-8 range. Otherwise, it will make all the rules of the robots.txt files invalid.
Setting up the user-agent of the robots.txt file
Next, you have to create a robots.txt file for setting up the user-agent. It will pertain to the search engine crawlers that the website owners wish to block or allow. Several entities like GoogleBot, SlurpBot, BaidusSpider, Sogou Web, Facebot, Bingbot, DuckDuckBot, YandexBot, and Exabot can be user-agent.
The robots.txt syntax to set up a single user agent is User-agent: Name of the bot. If you want to add more than one user agent, you have to follow the same procedure as above on a subsequent line and put the name of the extra user agent.
For instance:
User-agent: DuckDuckBot
User-agent: Facebot
To block all the crawlers or the bots, you have to substitute the bot name with (*) like User-agent: *.
Setting the robots.txt file rules
The search engine crawlers read the robots.txt file in groups. A group specifies the user-agent and includes directives with the directories or files indicating which user-agent the bots can or cannot access. The web crawlers start processing the groups from the top to the bottom. They can access any directory or page not set as disallowed. Thus, you have to add Disallow: / under every user-agent instruction in every group to block those bots of the search engines from crawling your website.
Example: User-agent: *
Disallow: /https://page.abc.com/robots.txt
For blocking a subdomain, you have to add a forward slash and the entire subdomain URL in the disallow rule. For blocking a directory, follow the above procedure and end it with a forward slash.
Example: User-agent: *
Disallow: /images/
At last, if you want all the search engines to gather the necessary information for all the web pages on a website, you can create either a disallow or allow rule. Add a forward slash while implementing the allow rule.
Example: User-agent: *
Disallow:
User-agent: *
Allow: /
Uploading the robots.txt file on a website
Websites do not automatically generate robots.txt files as they are not compulsory. Once you decide to set up a robot.txt file, upload it to the root directory of the website. The uploading structure depends on the hosting environment and files of the website. You can also take the assistance of the website hosting provider for uploading the robots.txt file.
Texting the performance of the robots.txt file
You can test the performance of the robots.txt file in several ways, like below.
The robots.txt Tester of Google in the search console
With the help of any one tool from the above, you can identify any errors in the logic and syntax and rectify them.
II. Setting up robots.txt file in a WordPress website using Yoast SEO Plugin
If you are using the Yoast plug-in in a WordPress website, you can see a section in the admin window for creating a robots.txt file. You have to log in to the website’s backend and access the SEO section. From there, click on the Tools section and then File Editor.
Follow a similar sequence as in the manual segment mentioned before for establishing the rules and user agents. When finished, save all the changes to the robots.txt file for activating it on the website.
Testing Your Robots.txt file
First, check if your robots.txt file is publicly accessible or not. You can do it by opening a private window in a browser and searching for the file. For instance: https://abc.com/robots.txt.
If you can find the file with the necessary content you have added, you can start the testing process in the following two ways:
- txt tests in the Google search console
- Open-source robots.txt library of Google
Since the second option is more advanced and is primarily suitable for developers, you can test your robots.txt file in Google’s Search Console.
- First, create an account in the Search Console for testing the robot text file markup.
- Click on the robots.txt tester and then open it.
- If your website is not linked to the Search Console account, first add a property.
- After this, you have to verify you are the owner of the website
- If you already have linked your website with the search console, select it from the drop-down menu on the homepage
- The tester will then recognize the logic errors or syntax warnings
- Finally, display the total warnings and errors at the editor’s end
You can rectify the warnings or errors directly on the web page or retest it. Even if you make any changes to this page, it will not appear on your website. The tool will never make any changes to the actual website. You have to copy and paste the entire test copy to the website’s robots.txt file.
FAQ
What does Allow all do in robots.txt?
Allowing all syntax in the robots.txt file will tell the search engine to access everything and all files. You can also leave the robots.txt file to guide the search engines to go through the entire website.
What does Disallow all do in robots.txt?
Disallowing all syntax in robots.txt file guides the search engine spiders not to crawl everything or access the entire website. You have to be very careful in this section, as any mistake will prevent the search engine crawlers from indexing an important web page.
Conclusion
Now that you know everything related to the robots.txt file, it is time to create one for your website. Robots.txt is simple to set up and can save all the headaches and time from crawling unnecessary content on your website.
