Wired has an interesting article about LookSmart, a new internet search engine that uses SETI@Home like distributed systems to index the web. Their goal is to index all 10 billion web pages every day, as opposed to Google indexing 150 million pages every day and refreshing the index every 30 days...
Can massive distributed computing beat Google? It actually might not be as inconceivable. When you think about it, the core software was devised by a couple of grad students, so what makes the barrier to entry really high is the sheer computing power it takes to index the web. If you could distribute that (and SETI@Home and it's off-spins have shown that it's possible) perhaps that could change the search engine landscape a bit.
Posted Tue, April 22, 2003 00:10 by dirvish
Looks like LookSmart's program, Grub, is topping out at about 70 million URLs a day this week.
http://grub.org
[moderate]