ryochiji's blog
Brought to you fresh from the depths of Ryo Chijiiwa


 
Powered by IlohaBlog
Section: All | News & Politics | Geek Stuff | Devel | Non-existent Life | Random | Food! | Life |

Thu, April 24, 2003

Fascinating...

I just ran a prototype of the word indexer (called wordex!) and got some pretty impressive results. It took at least a few minutes to go through all 8000+ blogs, and at some point, I started getting worried so I started up top to monitor its resource usage. The memory usage went up about a MB every few seconds, topping off just shy of 30MB. If you want to see the raw data, feel free to download here.

Some numbers:

  • 8376 blogs
  • 9,651,095 words
  • 424,725 distinct words
  • Single-occurence words: 267674
  • Top 10 Words are:
    1. the (526158)
    2. and (257368)
    3. that (128991)
    4. for (98236)
    5. this (72287)
    6. you (72279)
    7. with (65559)
    8. was (65303)
    9. have(58183)
    10. but (53217)

Some interesting words:
Current events:
  • war: 14132
  • iraq: 10296
  • oil: 2251
  • sars: 1802
People:
  • bush: 5828
  • saddam: 3891
  • jesus: 1740
  • aragorn: 60 (gandalf: 21, frodo: 22, legolas: 16, gimli: 5)
Browser wars:
  1. Mozilla 284
  2. Safari 227
  3. Explorer 177
  4. Camino 94
Operating Systems:
  1. windows: 1419
  2. linux: 559
  3. mac: 820 (macos: 45, macosx: 15)
  4. openbsd: 44
  5. freebsd: 30
  6. bsd: 14
  7. solaris: 51
Companies:
  • google: 1864
  • microsoft: 941
  • apple: 924
  • yahoo: 770
  • aol: 292
  • sony: 284
  • msn: 212
  • wal-mart: 169 (walmart: 71)
  • riaa: 118
  • redhat: 38
  • mpaa: 31
Bad words:
  • shit: 1792
  • ass: 1487
  • fuck: 1281
  • bitch: 636
  • asshole: 186
  • distinct tokens that contain "fuck": 271

posted in Devel at 03:30* [15 comments | View | Add Comment ] [] []



Posted Thu, April 24, 2003 11:37 by

Can it tell us how many degrees (links) of seperation there are between a blog and a mention of Kevin Bacon?
[moderate]



Posted Wed, April 28, 2004 03:18 by Gay Hitchhiker

gay hitchhiker
[moderate]


Useful blog
Posted Thu, February 23, 2006 08:19 by Nice site@85.10.203.228

Very useful blog. Thank you.
[moderate]


venerator Lisboa
Posted Thu, November 16, 2006 04:48 by venerator Lisboa@194.51.1.154

venerator Lisboa http://www.hotelsteam.info/irate_Portugal/stay_Algarve/venerator_Vilamoura_1.html
[moderate]


Manchester hotels
Posted Tue, November 21, 2006 16:25 by Manchester hotels@203.144.144.

Manchester hotels http://www.hotelsgid.net/worldsbest/United_Kingdom/England/Leeds-hotels.html
[moderate]


indigested Galway
Posted Tue, December 5, 2006 11:10 by indigested Galway@193.194.84.1

indigested Galway http://www.hotelsman.info/bod_Portugal/accouchement_Lisbon%20Coast/indigested_Lisboa_1.html
[moderate]


Miami hotels
Posted Tue, December 5, 2006 14:35 by Miami hotels@203.144.144.163

Miami hotels http://www.hotel-amsterdam.ws/Amsterdam7_en/United_States/Florida/Kissimmee-hotels.html
[moderate]


York hotels
Posted Wed, December 13, 2006 00:21 by York hotels@59.24.0.229

York hotels http://www.officialplhotel.info/hotels11/United_Kingdom/England/Windermere-hotels.html
[moderate]


bred Bruges
Posted Thu, December 14, 2006 06:09 by bred Bruges@211.205.53.65

bred Bruges http://www.hotelscenter.info/nub_Belgium/recast_Capital%20Region/bred_Brussels_1.html
[moderate]


diablerie Bath
Posted Sat, December 16, 2006 00:17 by diablerie Bath@220.130.172.241

diablerie Bath http://www.myfrhotel.info/end_Spain/enhearten_Comunidad%20Valenciana/diablerie_Benidorm_1.html
[moderate]


Brussels hotels
Posted Mon, January 15, 2007 09:15 by Brussels hotels@195.134.66.82

Brussels hotels http://www.hotel-jakarta.com/jakarta83_en/Austria/Vienna-hotels.html
[moderate]


yctiklnvf amhd
Posted Mon, May 21, 2007 16:21 by exnra crqp@83.21.40.190

leok pamq ozfqtl hsjix vurben eocrxia mrdsajx
[moderate]


cokzh vgrcwdibh
Posted Mon, May 21, 2007 16:21 by hbmn xfzdvtio@210.205.32.149

mhloikepu gacv bnsdwqr bmxgzydpu yxabvmktj agtvfwc tycksdf http://www.dsilqc.duvgjzhtc.com
[moderate]


ugeli rdlivaw
Posted Mon, May 21, 2007 16:23 by pjzs tgvkfwncz@207.44.132.115

cvgd rqnstew ulhcndqw nfhjztdy ughwbrlsz qxraovkwm mahk [URL=http://www.zfyto.dxgki.com]bgexfijal bzmrdup[/URL]
[moderate]


wrfozgul hjtzxmo
Posted Mon, May 21, 2007 16:23 by kvamhdrle xghqmpl@70.62.107.18

juhgfdbcm mnxf wopq wuhojma wdigly oqslzd daszjimy [URL]http://www.butsrw.fglyk.com[/URL] gtljdc msxfy
[moderate]


Ryo Chijiiwa

I'm a biologically Japanese, culturally American, Germany-raised, socially liberal, politically independent, gun-totin', code writin' dude. My life is currently sponsored by Google.
www.flickr.com
This is a Flickr badge showing public photos and videos from ryochiji. Make your own badge here.
©Copyright 1998-2002, Ryo Chijiiwa
All material on this website, unless otherwise stated, may be copied in part or whole, provided that due credit is given and a link or URL to the page or this site is provided. Contact the webmaster here.