One of the things Anthony and I talked about at our “Help! My iPod thinks I’m emo.” SXSW panel last week is the ‘Harry Potter Effect’ – how popular items in a recommender can lead to (among other things) feedback loops that lead to a situation where the rich get richer. A popular item like that latest Coldplay or Metallica album get purchased often with other albums and therefore end up getting recommended more frequently – and because it gets recommended – it gets purchased more often until it is sitting on the top of the charts. The Harry Potter effect can result in a lowering of the diversity of items consumed.
In his post, Online Monculture and the End of the Niche, Tom Slee over at whimsley has run a simulation that shows how this drop in diversity occurs – and also explains the non-intuitive result that while the use of a recommender can lead to decreased diversity overall, it can lead to increased diversity for an individual. Tom explains this with a metaphor: In the Internet World the customers see further, but they are all looking out from the same tall hilltop. While without a recommender individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone.
As an example of this effect, here’s a recommendation from Amazon that shows how 8% of those that shopped for The Big Penis Book
went on to buy a Harry Potter book. A recommender that pushes those that are buying books about big penises toward Harry Potter may indeed increase the diversity of those individuals (they may never have considered harry potter before, because of all those penises), but does indeed lower the overall diversity of the community as a whole (everyone is buying harry potter).
It is an interesting post, with charts and graphs and a good comment thread. Worth a read. (Thanks for the tip Jeremy)
Music Machinery 