Xenforo is no doubt the best forum CMS currently on the market. It has each and every feature that can be used to improve the user experience in the forum. But with the look, you also have to take care of indexing and SEO. Robots.txt files are important to manage your indexing in search engines. This guide will help you to make a complete ideal robots.txt file for Xenforo Forum.
What is robots.txt?
Basically, robots.txt is a set of URLs that you don’t want the search engine to index. It mainly includes login, signup, member profiles pages, or tags pages.
For WordPress sites, you have some SEO plugins that do this automatically after some settings, but for XENFORO you have to do it manually.
Robots.txt file is something that can either make or break your search engine presence. If you think the robots.txt file doesn’t contribute to your SEO, you are right up to some extent, but ignoring this might hit your rankings.
Why robots.txt file is important?
If you don’t have a robots.txt file for the Xenforo forum, Google or Bing or any other search engine will index each and every page of your forum regardless of the content and later give them a ranking based on the content of the page.
Now if you have small knowledge of SEO, you will understand that you don’t want every page to be index on search engines. There are some contentless or duplicate contents that Google doesn’t like to index, if your duplicate content is indexed the other URLs variation of the same content might not get indexed due to duplicate content issues. And this may include the main content page. You should avoid getting hit by duplicate content issues as soon as possible.
For e.g. You have the main content on a particular URL https://example.com/xenforo with the category xenforo and tags like #xenforotutorial.
Now your main content can be accessed through various possibilities like your post will be seen on your home page based on the design, then again on the category page on https://example.com/category/xenforo also from the tags page like https://example.com/tags/xenforotutorial also if you have an author archive, the same post will be seen on https://example.com/author/xyz and finally once again in search result https://example.com/?search=xenforo
For a normal user, this seems to be ok, but webmasters treat this as duplicate content and may not index the required URL and will go with any other URL. In order to avoid this, we use the robots.txt file. This file contains a set of URLs that you don’t want the search engines to index and follow.
Generally, for any blog site, you should avoid tags, categories, and author archives from indexing. For WordPress, various SEO plugins do this for you by doing some simple settings.
Coming back to Xenforo, currently, you don’t have any add-ons to make a perfect robots.txt file, but by following this guide, you can create a useful and fully-featured robots.txt file for your forum.
The basic format of a robots.txt file?
Robots.txt file can be made in any text editor like Notepad, else you can simply go to our Cpanel file manager and add a new file with the name “robots.txt” as a text file, and edit it by right click-edit and adding appropriate commands.
The file starts with mentioning the search bots. Every search engines have its bots that crawl websites for e.g: Google uses Googlebot also known as Google Spider, and Bing uses Bingbot likewise there are many bots.
You can get a list of all Search Engine bots, this will help you in making a better robots.txt file.
Now coming back, we start the file by mentioning the bot:
User-agent: command is used to mention bots.
If you want a single rule for all search bots you can use the below command
User-agent: * – This * here signifies all search bots.
If you want to make a rule for the specific bot, you can mention the bot after the command as shown below.
User-agent: Googlebot where Googlebot is google’s search bot, now all the commands below this line will only be applied to GoogleBot.
Example 1
Next, you have 2 options, either you Allow a URL or Disallow it, nothing in between.
Let’s explain this with an example,
Your domain is https://example.com and you have category pages at https://example.com/category and tags pages at https://example.com/tags/xyz, you want Google to only index your main URLs and not the category or tags URLs.
You can do this by adding the following commands in the robots.txt file:
User-agent: Googlebot
Disallow: /category/
Disallow: /tags/
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
The above rules apply to Google only since we have mentioned the agent in the code. By using this code, you disallow google to index your category and tags pages. Rest all the URLs are allowed. You also mention your sitemap in the robots.txt file.
Now if you want this rule for all search bots, you can simply do it by:
User-agent: *
Disallow: /category/
Disallow: /tags/
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
How to make different commands for different search bots?
Now that you know how to allow and disallow search bots, you may want to make separate permissions for separate search bots. For e.g in the above example, you want to apply those rules for Google only and also want Bing to index all your pages. You can simply do that by expanding your code. Simply go on typing a new set of rules.
User-agent: Googlebot
Disallow: /category/
Disallow: /tags/
Allow: /
User-agent: Bingbot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
The above example prevents google from indexing the tags and category pages, but allows bing to index them.
Now see the below code:
User-agent: Googlebot
Disallow: /category/
Disallow: /tags/
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
The above set prevents only google from indexing the category and tags pages, whereas all other search bots are allowed to index that pages.
Similarly, if you want to make different commands for more search bots you can go on adding them before sitemap.
For e.g:
User-agent: Googlebot
Disallow: /category/
Disallow: /tags/
Allow: /
User-agent: Googlebot-Image
Disallow:/attachments
User-agent: Bingbot
Dissallow: /author/
Allow: /
User-agent: msnbot
Disallow:/xyz
Allow:/
Sitemap: https://yourdomain.com/sitemap.xml
Ideal Robots.txt file for Forum.
Since now you have understood how robots.txt works, let’s discuss what to allow and not allow indexing in Xenforo Forum. I will provide a robots.txt file for the xenforo forum as per my forum (which no longer exists), you may change the URLs as per your forums.
The basic URLs you should disallow are the login page, members, signup type pages, etc.
Let’s start making our robots.txt file for Xenforo.
These rules are for all search bots, if you want to make different rules for other bots, you can simply right the new command as I showed above.
User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /goto/
Disallow: /login/
Disallow: /members/
Disallow: /conversations/
Disallow: /ajax/
Disallow: /misc/style?*
Disallow: /misc/quick-navigation-menu?*
Disallow: /media/keyword/
Disallow: /media/user/
Disallow: /media/service/
Disallow: /media/submit/
Disallow: /media/category/
Disallow: /events/birthdays/
Disallow: /events/monthly
Disallow: /events/weekly
Disallow: /recent-activity/
Disallow: /online/
Disallow: /lost-password/
Disallow: /register/
Disallow: /misc/
Disallow: /help/
Disallow: /admin.php
Allow: /
Sitemap: https://yourdomain.com/sitemap.php
Breakdown of my Robots.txt file
User-agent: * – I have set these rules for all search bots.
Disallow: /find-new/ – I don’t want to find a new section (You can see this on the main menu. It is a page with all new threads placed) to be indexed. You might want it, if so remove this command from robots.txt
Disallow: /goto/ – No index search results
Disallow: /login/ – No index Login Page
Disallow: /members/ – No index members profiles
Disallow: /conversations/ – No index personal Chats
Disallow: /ajax/ – No index any Ajax script if available
Disallow: /misc/style?* – if you have different styles, I don’t want them to show in the results.
Disallow: /media/ – Related to media/attachments.
Disallow: /events/ – Noindex events/birthdays etc
Disallow: /recent-activity/ – Users activity
Disallow: /online/ – Which users are Online
Disallow: /lost-password/ – Password recovery page
Disallow: /register/ – New Users registration page
Disallow: /help/ – Help page
Disallow: /admin.php – Admin Login Page
With this we have created an ideal robots.txt file for Xenforo, you can add or remove the commands as per your requirements.
Now after this, if you want to monetize your forum with Google Adsense you should read this article.
You can also track the performance of your forum by connecting your Xenforo forum with Google Analytics.