Robot.txt file, Inbound links and Outbound links with its types

What is Robots.txt file? 

 

Robots.txt is a text file that you put in the site’s root directory that specifies for search engines’ crawlers what website pages you want or don’t want to visit or crawl on your site. Robots main aim is read our website data and stored in search engine database. When search engine crawlers come to your site it will be looking for some special file. i.e., call roboat.txt file. This file tells to search engine spiders which pages of your site should be indexed and which pages of your site should be ignored. 

 

Structure of a Robots.txt File 

 

The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:

 

User-agent: 

Disallow:

 

“User-agent” is search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line: # All user agents are disallowed to see the /temp directory.

 

User-agent: * 

Disallow: /temp/

 

How does it work? 

 

Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results. 

 

Robots.txt files are useful:  

 

• If you want search engines to ignore any duplicate pages on your website.  

 

• If you don’t want search engines to index your internal search results pages.  

 

• If you don’t want search engines to index certain areas of your website or a whole website.  

 

• If you don’t want search engines to index certain files on your website (images, PDFs, etc.)  

 

• If you want to tell the search engines where your sitemap is located

 

How to create a robots.txt file 

 

• If you’ve not created a robots.txt file in your website or blog then you should make it as soon as possible. You must follow these instructions to create this:  

 

• Create a new text file and save it as the robot.txt – you can use the Notepad program on Windows PCs or Text Edit if you use Macs and then “Save As” a text-delimited file.  

 

• Upload it to your website’s root directory– this is generally a root level folder called “htdocs” or “www” which makes it appear directly after your domain name. 

 

• If you use sub domains, you need to create different robots.txt file for each sub domain. 

 

Inbound Link 

 

What is an Inbound Link? 

 

An inbound link, known as a backlink, is a hyperlink coming from another website to your own website. 

 

The links to your site are very important for SEO. This is because search engines, such as Google, see multiple high-quality and relevant inbound links to a page or site as an indication that the content on that page is useful for its users.

 

Why do I need Inbound Links? 

 

Reason #1: Search Engine Optimization (SEO) Websites that receive many inbound links can be more likely to rank higher in search engines. Basically, inbound links tell site crawlers that your website is an authority on a certain subject -- so the more inbound links you have from high quality, high authority sites, the better your website can rank in search engine results pages (SERPs). Search engines don't treat all inbound links the same and inbound links isn't the only ranking factor they consider. 

 

Reason #2: Referral Traffic If someone posts an inbound link on their website or blog, people who see that link might click on it, and you'll benefit from that referral traffic. The volume of traffic you get from referral traffic depends, of course, on how much traffic that blog or website receives in the first place.

 

Outbound links 

 

What is an Outbound links? 

 

Outbound links, or External links - Those links which are going to direct to another specific webpage or website altogether from your website and you have more control over these links. It means if you click an outbound link it will take you to another specific webpage or website. It will send user to entirely different sites. Most, if not all, websites have outbound links. 

 

There are two kinds of outbound links: 

 

1. Nofollow link 
2. Dofollow link 

 

Dofollow Link 

 

Dofollow link is the normal link. It is the default link. You don’t have to add to the code, you don’t have to change anything. Just make a link and it’s automatically a dofollow link. A dofollow link is what every link should be like. It passes on Google PageRank juice from your webpage to the targeted webpage.

 

An example of Dofollow Link:

<a href="https://www.google.com">Google</a>

 

Nofollow Link 

 

Nofollow link is the abnormal link. It does not pass on any Google PageRank juice to the targeted webpage but instead, it blocks off the Google PageRank flow. A nofollow link has a rel=”nofollow” tag inside the code. This tag says to the Google spider ‘not to follow’ the link. 

 

An example of Nofollow Link:

 

<a href="https://www.google.com" rel="nofollow">Google</a>