Moss and the developer, Keith Bernie, set an ambitious goal of indexing 5 billion Web sites. They unleashed their primitive crawler on the Internet, hoping to build a vast collection of Web pages that could be searched by an engine. But after a week, they were stuck at 24 documents.
"It was driving me nuts," Moss said.
... MSN's Web crawler had been fixed and was scurrying through the Web, boosting the index to 1,100 documents, and then 24,000. By the summer of 2003, it had reached 500,000 pages.
Thoses confirmed my first impressions on the beta in 2003. But I'm really surprise on that hard beginning if you compared it to Yahoo when they leave Google and crawled one or two billion pages in the first weeks !
Via SEO Scoop.
No comments:
Post a Comment