Saturday, 30 June 2007

Birth

AltaVista was started by Digital Equipment Corporation employee volunteers who were trying to provide services to make finding files on the public network easier.[citation needed] AltaVista was launched public as an internet search engine on 15 December 1995 at http://altavista.digital.com.[1][2] AltaVista was misunderstood by its parent company. Digital Equipment Corporation, a hardware company, missed the potential of the Internet and instead rationalized that AltaVista would be a showcase for its new line of servers.

Source: http://en.wikipedia.org/wiki/AltaVista

Friday, 29 June 2007

Excite Network

During the collapse of Excite@Home, Irvington, New York–based iWon.com had quietly started designing a new, yet familiar, Excite website hoping that they could acquire the Excite.com domain name and brand in the course the bankruptcy proceedings. IWon.com made a $10 million joint bid with Seattle's InfoSpace to purchase the domain and brand. On November 28, the court accepted the bid and gave iWon less than three weeks to launch a new Excite portal.

Source: http://en.wikipedia.org/wiki/Excite

Thursday, 28 June 2007

Open-source crawlers

DataparkSearch is a crawler and search engine released under the GNU General Public License.

Source: http://en.wikipedia.org/wiki/Web_crawler

Wednesday, 27 June 2007

Parallelization policy

A parallel crawler is a crawler that runs multiple processes in parallel. The goal is to maximize the download rate while minimizing the overhead from parallelization and to avoid repeated downloads of the same page. To avoid downloading the same page more than once, the crawling system requires a policy for assigning the new URLs discovered during the crawling process, as the same URL can be found by two different crawling processes.

Source: http://en.wikipedia.org/wiki/Web_crawler

Saturday, 23 June 2007

Selection policy

Given the current size of the Web, even large search engines cover only a portion of the publicly available internet; a study by Lawrence and Giles (Lawrence and Giles, 2000) showed that no search engine indexes more than 16% of the Web. As a crawler always downloads just a fraction of the Web pages, it is highly desirable that the downloaded fraction contains the most relevant pages, and not just a random sample of the Web.

Source: http://en.wikipedia.org/wiki/Web_crawler

Friday, 22 June 2007

Baidu

Due to the difference between hanzi and the Roman alphabet, the Chinese search market did not boom until the introduction of Baidu in 2000.[citation needed]

Source: http://en.wikipedia.org/wiki/Searchengine