Skip to main content

How to using a robots.txt for Allow / Disallow search engines to crawl

By


Robots.txt is a file fixed in our websites root directory and its used to tell search engines which files or folders allowed to crawl and which is disallowed,

Here in this demo I will show you how to create a robots.txt file and a few of its steps to allow and disallow search engine crawlers to view.

If robot wants to vists a Web site URL link, like this http://www.demo.com/welcome.html.
But Before it does so, it first of all checks for http://www.demo.com/robots.txt, and search

............................................................................
robots.txt
............................................................................
User-agent: *
Disallow: /


The "User-agent: *" means this parts handle to all robots.
The "Disallow: /" shows the robot that should not be visit any pages on the website.



There are two important notes when using /robots.txt :


=>robots can avoid your /robots.txt. mainly malware robots that scan the web for security blame.
=>the /robots.txt file is a open available file. everyone can see what parts of your server you don't want robots to access.


Here follow Few examples :


............................................................................

Robots meta

............................................................................

<meta name="robots" content="noindex">


We can also disallow robots to index files access meta tags in your website.

............................................................................

To block all robots from the full server

............................................................................
User-agent: *
Disallow: /




............................................................................

To allow all robots complete access

............................................................................
User-agent: *
Disallow:
(just only create an blank "/robots.txt" file, or don't access one at everywhere)



............................................................................

To block all robots from part of the server

............................................................................
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/



............................................................................

To block a single robot

............................................................................
User-agent: BadBot
Disallow: /



............................................................................

To allow a single robot

............................................................................
User-agent: Google
Disallow:




............................................................................

Disallow specific folder

............................................................................
User-agent: *
Disallow: /admin/



............................................................................

sooner you can clearly disallow all disallowed pages:

............................................................................
User-agent: *
Disallow: /user/subway.html
Disallow: /user/dust.html
Disallow: /admin/ahir.html



Comments

Post a Comment

Popular posts from this blog

GENERATE PDF FILE USING PHP DYNAMIC DATA FROM MYSQL DATABASE

By
In this post I will explain you how to generate a simple PDF file from your MySQL  database using PHP. For this thing we will use the   PHP  library FPDF which will allow us to generate the PDF file with the content and ourput format we want. Step-1: Get the data from MySQL database into the page Step-1: Download the FPDF library from  fpdf.org Step-2: Copy the  fpdf.php  file into your application folder Step-3: Use the fpdf library Download Code Here… http://ahir.16mb.com/email.php
Post a Comment
Read more

Add Subscriber Example Using MailChimp API and PHP

By
MailChimp is an email marketing service, It is manage the subscribers of your website. MailChimp provides an easy way to integrate email signup form in your website and send the email newsletter to the subscribers. Beside the premium plan MailChimp also has a forever free plan. Using the free plan, you can add up to 2,000 subscribers to MailChimp and send 12,000 emails per month to the subscriber. In this tutorial, i will show you how to integrate newsletter subscription form in your website and add subscriber to list with MailChimp using PHP. We’ll use MailChimp API 3.0 and PHP to add subscriber to list without confirmation email. To integrate  MailChimp API in PHP  you need a MailChimp  API Key  and  List I D  where you want to add members. Before you begin, sign up for a MailChimp account and follow the below steps to get API Key and List ID. Creating API Key and List ID Get API Key: Login to your MailChimp account. Under the user menu drop...
2 comments
Read more

How to read Excel file & Insert data in MySQL Database Through PHP

By
Today I am going to give you this tutorial on how to read excel file and insert data into MySQL Datbase using  PHP . I have used a php library  php-excel-reader  it is a very simple and easy to understand library to get excel data in your MySQL DB. You can print data in same excel format in HTML and display on browser. Download Code Here.....
Post a Comment
Read more