Register

If this is your first visit, please click the Sign Up now button to begin the process of creating your account so you can begin posting on our forums! The Sign Up process will only take up about a minute of two of your time.

Results 1 to 5 of 5
  1. #1
    Junior Member
    Join Date
    Feb 2016
    Posts
    6
    Member #
    53398

    Placement of robot.txt file

    I'm working with an institute with a main website: www.[domainname].edu with a content management system, and several secondary websites written in HTML on a separate server: www2.[domainname].edu.

    So using the same domain name on two different hosting servers.

    I need to hide some of the HTML sites on the "www2" server. Do I put the robot.txt file on the main "www" server or on the separate "www2" server.

    My first thought is to put it on the "www2" server as that's where the pages I want to hide are located, but all the sites on that server are basically using a sub-domain of the original "[domainname].edu" website.

    Would it be best to put it on the main server where the main website is being hosted?

    I suppose I could put it on both. But is there a best practice in cases like this? I'd like to do it right the first time.

    Thanks.

  2.  

  3. #2
    Senior Member
    Join Date
    Feb 2006
    Posts
    800
    Member #
    12463
    Liked
    296 times
    Who do you want to prevent finding your pages?

    At my school you can only reach the ww2 server by logging in. Do you have to log-in to reach your www2 server? If so, no one can see it outside your school. You can use a robots.txt file but I doubt it's necessary. If you want to keep everyone out including your instructors then password protect your pages if you can - not sure you can on your schools www2 server.

    I need to hide some of the HTML sites on the "www2" server. Do I put the robot.txt file on the main "www" server or on the separate "www2" server.
    You place it on the site you don't want indexed (the one you want to hide).
    Last edited by delstu; Feb 23rd, 2016 at 01:57 AM.

  4. #3
    Junior Member
    Join Date
    Feb 2016
    Posts
    6
    Member #
    53398
    Quote Originally Posted by delstu View Post
    Who do you want to prevent finding your pages?

    At my school you can only reach the ww2 server by logging in. Do you have to log-in to reach your www2 server? If so, no one can see it outside your school. You can use a robots.txt file but I doubt it's necessary. If you want to keep everyone out including your instructors then password protect your pages if you can - not sure you can on your schools www2 server.

    You place it on the site you don't want indexed (the one you want to hide).
    The second "www2" server is public and the sites are available to anyone. As you said, if it was behind a password, I wouldn't worry about it.

    It's a case of pages we don't want readily available. Archived content that we need to keep up and running, but don't want the general public to stumble upon when they are looking for current information.

    I'll drop the robot.txt file on the www2 server.

    Thanks for the response.

  5. #4
    Senior Member
    Join Date
    Feb 2006
    Posts
    800
    Member #
    12463
    Liked
    296 times
    The robot.txt file does not keep the public out, it only keeps the major search engines from indexing the site in their search results. If these pages contain information that you MUST keep out of the public hands then you need to password protect the site. Start by looking on this page:
    HTML URL Encoding

  6. #5
    Junior Member
    Join Date
    Feb 2016
    Posts
    6
    Member #
    53398
    For now, it's enough that we hide the pages from the search engines. Thanks.


Remove Ads

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT -6. The time now is 07:24 AM.
Powered by vBulletin® Version 4.2.3
Copyright © 2019 vBulletin Solutions, Inc. All rights reserved.
vBulletin Skin By: PurevB.com