Robots.txt is a file fixed in our websites root directory and its used to tell search engines which files or folders allowed to crawl and which is disallowed, Here in this demo I will show you how to create a robots.txt file and a few of its steps to allow and disallow search engine crawlers to view. If robot wants to vists a Web site URL link, like this http://www.demo.com/welcome.html. But Before it does so, it first of all checks for http://www.demo.com/robots.txt, and search ............................................................................ robots.txt ............................................................................ User-agent: * Disallow: / The "User-agent: *" means this parts handle to all robots. The "Disallow: /" shows the robot that should not be visit any pages on the website. There are two important notes when using /robots.txt : =>robots can avoid your /robots.txt. mainly malware robots that scan the we...
Learn from step by step tutotrials of php, asp.net, php 7, .net mvc, c#, laravel, symphony, seo and many more.