Register

If this is your first visit, please click the Sign Up now button to begin the process of creating your account so you can begin posting on our forums! The Sign Up process will only take up about a minute of two of your time.

Page 1 of 2 1 2 LastLast
Results 1 to 10 of 11
  1. #1
    Senior Member
    Join Date
    Aug 2003
    Posts
    444
    Member #
    2801
    Hello!

    I have a question about web crawlers for search engines and other bots. How much of your traffic do they account for? That is, what percentage do they constitute?

    I am wondering, because at the moment, my counter is showing that a significant proportion of my traffic is crawlers, and I am wondering if this is typical.

    In case you're wondering: it is varying between 20%-30%.

    Thanks!
    eKstreme
    eKstreme.com - Free website tools!
    fontfox - free fonts Hand-picked quality fonts.

  2.  

  3. #2
    Senior Member splufdaddy's Avatar
    Join Date
    Feb 2003
    Location
    Boston, MA
    Posts
    4,488
    Member #
    735
    Lately MSN's searchbot seems to be going crazy. It's been to many of my sites multiple times a day for the past week or so. Because of MSN, a larger percentage of traffic has been from crawlers lately.

  4. #3
    Unpaid WDF Intern TheGAME1264's Avatar
    Join Date
    Dec 2002
    Location
    Not from USA
    Posts
    14,480
    Member #
    425
    Liked
    2784 times
    I'm in spluf's boat. My traditional crawler traffic is around 10-15%, but thanks to MSN's search bot creating its new database for its launch probably in July, mine's been between 25-30% for the last three weeks. Anything higher than that is something to be a little concerned about, unless you've done a major update on your site.
    If I've helped you out in any way, please pay it forward. My wife and I are walking for Autism Speaks. Please donate, and thanks.

    If someone helped you out, be sure to "Like" their post and/or help them in kind. The "Like" link is on the bottom right of each post, beside the "Share" link.

    My stuff (well, some of it): My bowling alley site | Canadian Postal Code Info (beta)

  5. #4
    Senior Member Brak's Avatar
    Join Date
    Apr 2003
    Location
    San Francisco, CA
    Posts
    3,413
    Member #
    1217
    Liked
    2 times
    MSN has been raping my sites like no other. I'm beginning to wonder if they've made a mistake in their coding, it's been visiting the entirety of my sites at least once a day for the past week or so. Oh well, for me bandwidth isn't an issue when they're not grabbing images, so I'm happy
    Kyle Neath: Rockstar extraordinare
    The blog | The poetry site | The Spore site

  6. #5
    Senior Member
    Join Date
    Aug 2003
    Posts
    444
    Member #
    2801
    That's the thing, it's MSN and a new one called "OpenBot". I've never seen OpenBot before a couple of weeks ago.

    Also, does anyone know a bot with a UA string of "River Valley Inc"? It's quite nasty too!
    eKstreme
    eKstreme.com - Free website tools!
    fontfox - free fonts Hand-picked quality fonts.

  7. #6
    Unpaid WDF Intern TheGAME1264's Avatar
    Join Date
    Dec 2002
    Location
    Not from USA
    Posts
    14,480
    Member #
    425
    Liked
    2784 times
    I haven't seen "River Valley Inc." or OpenBot myself. Did you try looking up its IP block to see who owned it and where it was? It may give you a better idea.
    If I've helped you out in any way, please pay it forward. My wife and I are walking for Autism Speaks. Please donate, and thanks.

    If someone helped you out, be sure to "Like" their post and/or help them in kind. The "Like" link is on the bottom right of each post, beside the "Share" link.

    My stuff (well, some of it): My bowling alley site | Canadian Postal Code Info (beta)

  8. #7
    Senior Member
    Join Date
    Aug 2003
    Posts
    444
    Member #
    2801
    OpenBot points to http://gais.cs.ccu.edu.tw/robot.php in the user agent.

    I cannot find anything on the web re River Valley Inc. The domain name is *.cox.net which is "Cox Communications" in Atlanta, USA.
    eKstreme
    eKstreme.com - Free website tools!
    fontfox - free fonts Hand-picked quality fonts.

  9. #8
    Senior Member DesignBox's Avatar
    Join Date
    Apr 2004
    Posts
    340
    Member #
    5659
    Quote Originally Posted by splufdaddy
    Lately MSN's searchbot seems to be going crazy. It's been to many of my sites multiple times a day for the past week or so. Because of MSN, a larger percentage of traffic has been from crawlers lately.

    Same here, that msn bot is on there all day every day :cross-eyed:

  10. #9
    Unpaid WDF Intern TheGAME1264's Avatar
    Join Date
    Dec 2002
    Location
    Not from USA
    Posts
    14,480
    Member #
    425
    Liked
    2784 times
    Let it, guys. It'll help you when they relaunch their engine.
    If I've helped you out in any way, please pay it forward. My wife and I are walking for Autism Speaks. Please donate, and thanks.

    If someone helped you out, be sure to "Like" their post and/or help them in kind. The "Like" link is on the bottom right of each post, beside the "Share" link.

    My stuff (well, some of it): My bowling alley site | Canadian Postal Code Info (beta)

  11. #10
    Senior Member
    Join Date
    Oct 2003
    Location
    Essex/Greater London
    Posts
    167
    Member #
    3433
    I got about 1,500 visits from Googlebot in June, but considering I have a forum I don't think that's too bad.


Page 1 of 2 1 2 LastLast

Remove Ads

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT -6. The time now is 10:29 AM.
Powered by vBulletin® Version 4.2.3
Copyright © 2022 vBulletin Solutions, Inc. All rights reserved.
vBulletin Skin By: PurevB.com