Register

If this is your first visit, please click the Sign Up now button to begin the process of creating your account so you can begin posting on our forums! The Sign Up process will only take up about a minute of two of your time.

Results 1 to 8 of 8
Like Tree7Likes
  • 3 Post By khenney
  • 1 Post By Vapr_Arts
  • 1 Post By TheGAME1264
  • 2 Post By stephend500

Thread: Confused about the ACTUAL function of robots file for Google searching

  1. #1
    Member
    Join Date
    Nov 2015
    Location
    Virginia, USA
    Posts
    37
    Member #
    52423
    Liked
    7 times

    Confused about the ACTUAL function of robots file for Google searching

    I am starting a separate thread on this (even a rather short one already is on the forum) because robots seems to be crucially important in SEO.

    If you go to that small topic thread, you will see that one commenter (apology for not having the poster name here) says that a robots file is to tell search engines which parts I DO NOT want Google to pursue in its handling of search terms for my website.

    Another commenter seems to take a more neutral tack --- just saying the robots is an aid to search term optimization by Google. (I hope these characterizations by me are correct --- but expressed with other words --- on what those posters wrote.)

    So, which is it: robots file for EXCLUSION of some website content ----- or robots file as a GUIDE for Google to use in decisions regarding intelligent search term choice?

  2.  

  3. #2
    Senior Member khenney's Avatar
    Join Date
    Apr 2016
    Location
    Half Moon Bay, California
    Posts
    276
    Member #
    53911
    Liked
    148 times
    Robots.txt tells search engines which URLs on your site you don't want them to crawl.

    It's hard to comment in the notion that it's "an aid to search term optimization", especially since you're paraphrasing some other unlinked thread, but I suspect something got lost in translation.


    Sent from my iPhone using Tapatalk
    Kevin Henney
    CRUCIAL CREATIVE
    Web Development - Graphic Design - Branding

  4. #3
    Senior Member Vapr_Arts's Avatar
    Join Date
    Oct 2013
    Location
    California
    Posts
    1,930
    Member #
    37412
    Liked
    544 times
    First it may help to understand how search engines gathers information. Google has its GoogleBot which will crawl sites and gather new and updated info to index. Google isn't the only one, but the most important for obvious reasons. Search engines aren't the only bots crawling sites either.

    A robot.txt file grants or restricts permissions to these bots and tell it which pages to crawl like Khenny mentioned above. Something to note is not all bots will follow these rules.

    You can check out http://www.robotstxt.org/robotstxt.html to get a rundown


    Sent from my iPhone using Tapatalk
    stephend500 likes this.

  5. #4
    Unpaid WDF Intern TheGAME1264's Avatar
    Join Date
    Dec 2002
    Location
    Not from USA
    Posts
    14,485
    Member #
    425
    Liked
    2783 times
    You can use robots.txt to indicate which pages/files on a site to exclude (or to a lesser extent, include). You can also link to a sitemap for what is horribly named "autodiscovery" (I'm not sure why they call it "autodiscovery" because more often than not a manual effort is required in some form or fashion).

    Now...is this the best way to exclude content from search engines? No. Not all bots play nicely, although the major ones tend to. If there's something you want excluded from search engines, either don't put it online:

    Devil-Cross-57-Thanks-Captain-Obvious.jpg

    ...or have some form of security to protect it from being accessed easily (a login/password form, for example).

    As far as "intelligent search term choice"...a robots.txt file has absolutely nothing to do with that. Its sole purpose as it pertains to search engines is to say "you can index this, this, and this, but that file over there and the directory here are off-limits."

    By the way, you can link to the thread if you like because it is relevant to the discussion (we could use the context).
    stephend500 likes this.
    If I've helped you out in any way, please pay it forward. My wife and I are walking for Autism Speaks. Please donate, and thanks.

    If someone helped you out, be sure to "Like" their post and/or help them in kind. The "Like" link is on the bottom right of each post, beside the "Share" link.

    My stuff (well, some of it): My bowling alley site | Canadian Postal Code Info (beta)

  6. #5
    Member
    Join Date
    Nov 2015
    Location
    Virginia, USA
    Posts
    37
    Member #
    52423
    Liked
    7 times
    O.P. here ---- Thanks to all of you who helped me clear this up. I did not put up a link to the original thread because I was pressed for time, but in the future will give a link to threads, because they are solid context when understanding a new question.

    BTW, GAME, the photo guy looks like a prime candidate to star in my next nightmare about being kidnapped to serve a 30 year term in an Eastern European pre-1991 Soviet bloc prison. Very nicely tailored uniform, I must say. Did he sneak a Jeep out of the motor pool for a quick trip to one of the 1960s French couturiers . .

  7. #6
    Unpaid WDF Intern TheGAME1264's Avatar
    Join Date
    Dec 2002
    Location
    Not from USA
    Posts
    14,485
    Member #
    425
    Liked
    2783 times
    For those who don't know, the guy in the pictures is Captain Obvious.

    If I've helped you out in any way, please pay it forward. My wife and I are walking for Autism Speaks. Please donate, and thanks.

    If someone helped you out, be sure to "Like" their post and/or help them in kind. The "Like" link is on the bottom right of each post, beside the "Share" link.

    My stuff (well, some of it): My bowling alley site | Canadian Postal Code Info (beta)

  8. #7
    Member
    Join Date
    Nov 2015
    Location
    Virginia, USA
    Posts
    37
    Member #
    52423
    Liked
    7 times
    I plead nolo contendere to a charge of P.A.I. --- popular ad ignorance. By the way, the lodging history of that schlub in this vid clip reminds of an ancient joke about a NOTAM at one of our military bases many years ago:

    "On orders of General Whathisname, there will be absolutely no low overflying of the naturist resort located 23.7 miles south-by-southwest of the control tower."
    TheGAME1264 and Vapr_Arts like this.

  9. #8
    Junior Member
    Join Date
    May 2016
    Location
    Bangaluru
    Posts
    16
    Member #
    54222
    robots.txt file says to google bot not to crawl the URL which you you dont want Url to be crawled. This is the main function of robots.txt


Remove Ads

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT -6. The time now is 03:50 PM.
Powered by vBulletin® Version 4.2.3
Copyright © 2019 vBulletin Solutions, Inc. All rights reserved.
vBulletin Skin By: PurevB.com