If you have a feeling that all roads somehow lead to Radiohead, you’re not alone. Oscar Celma, my friend and former colleague at the Pompeu Fabra University in Barcelona, successfully defended his PhD thesis on music recommender systems the other day. The thesis, titled “Music Recommendation and Discovery In The Long Tail”, sheds new light on Chris Anderson’s famous online niche market model. In his work, Oscar explains why social recommender systems, like those we know from Amazon and Netflix, have trouble reaching deep enough into the long tail, not realizing the tail’s full economic potential.
Why are Amazon and Netflix more likely to recommend already popular books and movies, and why does Last.fm claim that so many bands sound like U2, Radiohead or Coldplay? Part of the answer can be found in the illustration below. Can you spot what all the named artists have in common?
If you say that they are all very popular rock artists, I would say you’re right. This is the Long Tail of Last.fm, a popular social music recommendation service. A small collection of artists with very high play counts make up the head, while the large majority of artist with lower play counts are found in the tail. This type of distribution is typical for all services that do the Amazon style of recommendation, the “people who bought this also bought that.” For Last.fm, the riddle goes more like “people who listen to this also listen to that”.
When a lot of people (who may otherwise have very diverse tastes in music) listen to Coldplay, Coldplay becomes very well connected with a lot of other artists, and also becomes a hub in what is known as a small-world network. Such networks are the basis for social recommendations. Oscar shows that these hubs are indeed the most popular artists, who again gets recommended more often than others. That is why all roads lead to Radiohead.
Using more formulas and correlation diagrams than most people can comfortably cope with (including me), Oscar explains why social recommenders (like Amazon) should be combined with content-based recommenders, which finds music that actually sounds the same. Such hybrid recommenders are much better at recommending novel and relevant items from The Long Tail. Recommender systems can generally be divided into three groups, based on how the recommendations are generated. These are 1) expert, 2) social and 3) content-based recommender systems. Hybrid recommenders, like Oscar proposes, combine elements mainly from the social and content-based.
Social recommendations reflect the big trends, and capture in a sense the social aspects of music. It works because most of us tend to like the same things as other people like us do. It may be difficult to break out of the demographic enclosures of this gigantic mix-tape party, though, and to discover something truly novel. That, combined with popularity bias (rich get richer effect), cold-start problems and early raters feedback loops, puts some limitations on social recommenders in terms of long tail reach.
Expert recommendations capture more of the cultural aspects of music, things that aren’t evident from our listening habits. A music expert may compile lists of great songs produced by Bob Rock, or top love songs from the 80′s. Such recommendations captures many beautiful and odd aspects of music, but the major drawback is the lack of scalability. No expert can know and recommend all the music in the world.
Content-based (CB) recommendations are often produced by sophisticated music analysis software, comparing the actual sound of the music itself. This “objective” focus on stylistic similarity may surprisingly cross over established culture-dependent genres. CB recommendations scale well to millions of songs, but data sparsity may also produce odd and incorrect recommendations.
A particularly interesting nuggets of information from Oscar’s thesis work is that it for social recommenders take 5 links/clicks/jumps to reach from the head to the long tail, while it takes just 2 for expert and CB recommenders. Wow!
If you like to know more, head over to Oscar’s PhD thesis web page. And I’ll keep writing about recommender systems, so make sure to come back for more.
If you think that everything does actually sound like Coldplay, this song is for you