(Last Updated On: July 9, 2018)
What is robots.txt file for SEO beginners students
Dear friends you know that, we published SEO beginner tutorials regularly for those peoples who want to learn SEO properly. So, in this article you will learn that what is robots.txt file and how to increase optimization of website. We will discuss different confusion which faces during creating robots file. Before this paragraph I discussed about Google sitemap which is very important for every site. So let’s start to clear concept.
What is robots.txt file?
It is not a difficult question as you understand because if we shortly say that it is the text file which includes some instruction. What are instructions? Read below paragraph to understand.
Why need such file
Basically we instruct the Google (its bots) that don’t come to these web pages. There are many kind of content which your website has. Some of them you did not want that it display on the Google.
We need to have such file because it prevents our some web pages in indexing. It also prevents to copy right content issues because if same web page index two time then there may arise issue of copyright. We can construct bots or crawling software to don’t crawl this page. When your web page is not crawl then already will not be able to index.
How robots.txt file increase website SEO
Yes it increases website good performance as well website SEO. If you are beginner then must give attention here.
If your website has pages with full of links inserted by users. I means when you add forum Plugins or discussion form or answer question features in WordPress then users able to publish post and able to insert link which is not good.
Some links and relative links are best but bundle of links which are irrelevant, spamming etc can down your website.
so you need to block by spider that pages.
It can secure website because you can instruct spider that don’t crawl admin directory which include important information.
How to write robots.txt file properly if you want to get best response. You have to full attention when you are creating such file. Below this paragraph, I shared one lesson in which you will learn that how to write this file and you will get spider file for blogger and WordPress.
How to Upload robots.txt file
After creating file you must named properly and check its spelling. It is very easy to upload to website root folder. You need to just open hosting account for that website for which you want to upload this file.
Just click on upload file then open your computer, find robots.txt file and just click on open. It is very simple process to do.
Remember that your robot file should open on this URL www.yourDomainName.com/robots.txt
If you are seeing properly that which you submit to website root folder then all is okay.
You can verify it from Google webmaster tool.
Just open your webmaster tool with Google Gmail and go to robots section. You will find you’re that file which you created in notepad.
After verifying, make a cool mind that all working is fine, if you have any problems when doing this process then joins my discussion forum or ad question in ask question section, I will guide you properly.
Tell me, have you read this article? what your learnt from this lesson?