All You Need To Know About Robots.txt!

Robots.txt is such a popular term in web marketing and internet business. What is referred to as robots.txt? Robots.txt refers to the plain text file that is stored in your website. Why do you need a robots.txt? Robots.txt is responsible for instructing the search engine spiders what to index and what to ignore on the website.

Robots.txt

Websites always store the robots.txt at the root of the website. The search engine spiders look for the robots.txt on the root of a domain. It will not look anywhere else on the website so you can’t actually specify a different location for it. Robots.txt is always named in lower case, as well.

Robots.txt instructs the search engine spiders as to what part of the website to index and what parts should be ignored. It somehow controls the action of the search engine spiders in such a way that it directs its movement. It is like a recommendation – robots.txt recommends which part of the sites to index.

Creating Robots.txt File

How do you create a robots.txt file? First you need to identify what should be included in the robots.txt file. A robots.txt file is like a list of instructions. One part of the robots.txt file is the User-agent. It is the one that tells the robots or spiders reading the file which robots should pay attention to the instructions. Oftentimes User-agent indicates “*” which means “all robots”.

After the User-agent follows the rules themselves. Remember that there should not be any blank lines in the instructions. The instructions in the robots.txt oftentimes follow these formats:

• Disallow: /folder/


• Disallow: /file.htm

Each line must bear only one instruction. If you put anything after “#”, that will be completely ignored since the spiders will consider it as a comment. So it is advised to write a comment on a separate line all by itself.

In creating your site’s robots.txt file, be extra careful with your syntax and commands. Once the robots fail to recognize a certain command in your robots.txt, it may interpret a wrong notion that you want them to stay away from indexing your website. Incorrect syntax may also prevent your entire site from being indexed by the robots.

What you need to do is create the robots.txt file and check it twice or even thrice before you upload it. This will ensure proper indexing of your site. This practice will also minimize the probability of committing errors in commands and syntax.

Robots.txt is essential in having your site indexed by the search engines’ robots or spiders. Aside from Robots.txt, there are still important factors that you need to improve in your website. Please see our Web Marketing Services or apply for a Free Website Analysis.