When I put my first page on the web, I of course checked to see how quickly the various search engines incorporated it. And I checked ... and checked ... and am still checking!
I had this silly notion that the web crawlers used by the search engine were enormously efficient, and grabbed every page within a few days. Not so! Even after submitting my URL to 4 different search engines, only 2 of them have found my page 3months later!
The following table reports on how quickly some search engines have found that first page. I will collect data about how quickly they find this page, as well, since it might appear much more quickly after they have "found" my site.
Update on 10/13/96: Once again, I was wrong. Only excite has found the new page after a month and a half!
| Delay (days) for | |||
|---|---|---|---|
| Search Engine | My Home Page | This Page | Comments |
| Alta Vista | ~36 | >51 | |
| Excite | ~48 | ~15 | |
| Hotbot | >90 | >51 | |
| InfoSeek | >90 | >51 | Infoseek twice has had my home page, but each time has quickly lost it!! It currently does not have it indexed. |
| Lycos | ~29 | >51 | |
| Magellan | >90 | >51 | |
| WebCrawler | >48 | >51 | |
| Yahoo | >90 | >51 | |
For reference, the date that I first put my home page on the web was 7/13/96, and the date that I put this page on was 8/22/96. The way I conducted the search was to search for "Tom Chester" for my home page, and for "+delay +appear +updated +search +engines +Lycos" for this page.
Several years after I did this study, researchers Steve Lawrence and C. Lee Giles (Nature, July 8, 1999) reported that it took an average of 6 months for a page to be indexed by search engines.
e-mail: Tom Chester