Search engines crawl the web. You can submit sites to be included in search engines but even if you don’t, if your site is linked from other websites, most probably you will get added even without submitting your website. Search engines have programs that follow the links of websites from one website to another and they call these scripts robots, spiders or crawlers. Sometimes the robots are simply refered to as bots. And whatever a search engine runs over it will be considered to be added into the large database of indexed websites.

Although some parts of a website can be excluded by setting up rules in your robots.txt file, which I will explain in another article, but robots.txt can only exclude pages within your website. But for pages going away from your site, you can use the rel=”nofollow” attribute of your link tags.

Link Text

But please note that not all search engines will follow this rule, although Google strickly follows the rule. If ever for some reason you want to disallow bots to continue crawling. It may sound like a selfish act no avoid other sites from getting crawled by spiders, but it is a good way to prevent crawling of websites that resort to spamming guestbooks and blog comments.

Popularity: 1% [?]

Be Sociable, Share!

•