Register

If this is your first visit, please click the Sign Up now button to begin the process of creating your account so you can begin posting on our forums! The Sign Up process will only take up about a minute of two of your time.

Results 1 to 3 of 3

Thread: robots in SEO

  1. #1
    Senior Member
    Join Date
    May 2017
    Posts
    95
    Member #
    56717
    Liked
    3 times

    robots in SEO

    Hi


    What does robots.txt do in SEO?

  2.  

  3. #2
    Junior Member
    Join Date
    May 2018
    Posts
    13
    Member #
    58950
    Robots.txt contains recommendations (NOT rules) for the search engine, which pages of the site should be crawled and indexed and which shouldn't.
    It contains directives:
    user - you can mention for what search engine you write the recommendations
    allow - what pages robot shoul crawl
    disallow - what pages robot shuld't crawl
    sitemap - add your sitemap.xml with important pages (you can read how to create sitemap in the internet)

    When you create robots.txt you should be very careful, because you can accidentally forbid important pages.
    After creation you have to add it to the search console and there you can check your robots' validity.

  4. #3
    Member
    Join Date
    Feb 2018
    Posts
    81
    Member #
    58388
    Liked
    3 times
    Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.


Remove Ads

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT -6. The time now is 03:51 PM.
Powered by vBulletin® Version 4.2.3
Copyright © 2019 vBulletin Solutions, Inc. All rights reserved.
vBulletin Skin By: PurevB.com