Robots.txt :
Robots.txt file instruct the robots that what is to to index from your website. It is also called "Robots Exclusion Protocol". It has the particular instructions set.
Some examples are here
The "
In this example robots file tells directories and pages, those not to follow.
Robots Meta Tag :
Robots meta tag can also be used in palce of robots.txt file. However robots.txt file is more reliable.Let us see how to use it for specific purpose.
Following example shows that no page that is to follow
Multiple options are here that can be used as different options.
<meta name="ROBOTS" content="INDEX, NOFOLLOW">
Robots.txt file instruct the robots that what is to to index from your website. It is also called "Robots Exclusion Protocol". It has the particular instructions set.
Some examples are here
User-agent: *
Disallow: /The "
User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.User-agent: *
Disallow: /cgi-bin/
Disallow: /search/
Disallow: /mydata/pics.htmIn this example robots file tells directories and pages, those not to follow.
Robots Meta Tag :
Robots meta tag can also be used in palce of robots.txt file. However robots.txt file is more reliable.Let us see how to use it for specific purpose.
Following example shows that no page that is to follow
<html>
<head>
<title>...</title>
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW">
</head>Multiple options are here that can be used as different options.
<meta name="ROBOTS" content="NOINDEX, FOLLOW">
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW"><meta name="ROBOTS" content="INDEX, NOFOLLOW">