WHAT IS ROBOTS.TXT??
A robots.txt
is a file which tells search engine crawlers which page or file can or can’t
request from your site.
How to see robots.txt file of
your site?
Any sites robots.txt
can be viewed by searching in google by typing your site’s URL/robots.txt .
For example
for my blog it is https://riseprogramming.blogspot.com/robots.txt.
The default
robots.txt which was present first when you created your site will appear.(which
will alter if you change it).
User-agent: *
Disallow:
Sitemap:
https://riseprogramming.blogspot.com/sitemap.xml
Above is the
example for a robots.txt file.
How to write
your robots.txt file?
1.The first
thing is to write the User-agent: ,better to mention the user-agent as *
User-agent: *
2.The next thing is Disallow:
This
prevents the search engines to avoid being crawled to particular page or file
of your website.
What to
mention beside Disallow: ? is
the text which is present after your URL of the page or post. Not understood?
Let me explain it by a example…
Consider
if one of your posts URL is https://abc.com/wxy
. The thing to mention after Disallow: is /wxy.
- ·
If you type Disallow:/ ,Nothing from your search engine can be
crawled by the search engines.
- · Disallow:/search prevents search engines from crawling into your website when it is searched in Google.
It is
better to mention nothing beside Disallow: so that entire information of your website can be
crawled by the search engines.
3.The next
thing is Allow: which is the opposite of Disallow ,It’s ok if you
don’t mention it so that it will be neglected and everything in your website
will be available.
4.The
last thing is to give your site’s sitemap Sitemap:.your sites sitemap will be
your
sites URL/sitemap.xml.
How to
change Your robots.txt?
If your
hosting site is blogger simply go to
settings >search preferences>robots.txt
.
Click
enable and paste the below code if all of your information must be crawled .
User-agent: *
Disallow:
Sitemap: your site’s URL/sitemap.xml.
Careful!!
Wrongly written robots.txt may prevent your site from appearing in the google
search which may eventually reduce your traffic…
Post a Comment