Register

If this is your first visit, please click the Sign Up now button to begin the process of creating your account so you can begin posting on our forums! The Sign Up process will only take up about a minute of two of your time.

Results 1 to 4 of 4
  1. #1
    Senior Member dotcommakers's Avatar
    Join Date
    Oct 2003
    Posts
    527
    Member #
    3342
    Liked
    1 times
    Hi Friends,

    I am just curious.. how google starte crawling when it was at starting.. when nobudy knows.

    Did it start indexing website urls itself? like we are doing ourselves now by "add url" option..

    Now i know google indexing.. approx 4,285,199,774 webapges.. but as we know.. we thousands of.. millions of people have added our own urls to it.. but what when it was just started..?

    hope i explain well what i want to know.. otherwise please ask me.. i will try to explain again.

    thanks.

  2.  

  3. #2
    Senior Member splufdaddy's Avatar
    Join Date
    Feb 2003
    Location
    Boston, MA
    Posts
    4,488
    Member #
    735
    A competitor to Google, MSN is rebuilding it's search engine now. The way they're doing it is by spidering the internet at a blistering pace. Many site owners have reported seeing the MSN spider daily. They are also tweaking their alogorithim, which is the software that brings back your results when you search.

  4. #3
    Senior Member dotcommakers's Avatar
    Join Date
    Oct 2003
    Posts
    527
    Member #
    3342
    Liked
    1 times
    humm may be i was not clear in my post.. first..

    Spluf.. i want to know how they started their site.. i mean when nobudy knew what is google.. did they started submit urls their self.. now.. i think.. as we have added thousands of.. millions of sites there.. so the google spider.. crawl in web automatically but what was in start?

    is there any software or script.. which can crawl whole web.. automatically and can find all sites itself and make an index of it?

  5. #4
    Unpaid WDF Intern TheGAME1264's Avatar
    Join Date
    Dec 2002
    Location
    Not from USA
    Posts
    14,485
    Member #
    425
    Liked
    2783 times
    It wouldn't be especially difficult to build the shell of the spider. What would be hard would be optimizing it for speed and accuracy.

    Basically, you start by crawling one very large site with a ton of outbound links (Yahoo! comes immediately to mind here). From there, you'd crawl each of the outbound links, and each outbound link from the outbound links, and so on and so on.

    You go four or five site levels deep outside of Yahoo!, you've got millions and millions of pages to use.

    Again, the hard part would be tweaking the spider and the hardware.
    If I've helped you out in any way, please pay it forward. My wife and I are walking for Autism Speaks. Please donate, and thanks.

    If someone helped you out, be sure to "Like" their post and/or help them in kind. The "Like" link is on the bottom right of each post, beside the "Share" link.

    My stuff (well, some of it): My bowling alley site | Canadian Postal Code Info (beta)


Remove Ads

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT -6. The time now is 05:53 AM.
Powered by vBulletin® Version 4.2.3
Copyright © 2019 vBulletin Solutions, Inc. All rights reserved.
vBulletin Skin By: PurevB.com