If you have a feeling that all roads somehow lead to Radiohead, you’re not alone. Oscar Celma, my friend and former colleague at the Pompeu Fabra University in Barcelona, successfully defended his PhD thesis on music recommender systems the other day. The thesis, titled “Music Recommendation and Discovery In The Long Tail”, sheds new light on Chris Anderson’s famous online niche market model. In his work, Oscar explains why social recommender systems, like those we know from Amazon and Netflix, have trouble reaching deep enough into the long tail, not realizing the tail’s full economic potential.
Why are Amazon and Netflix more likely to recommend already popular books and movies, and why does Last.fm claim that so many bands sound like U2, Radiohead or Coldplay? Part of the answer can be found in the illustration below. Can you spot what all the named artists have in common?
Last.fm Artist Play Counts
If you say that they are all very popular rock artists, I would say you’re right. This is the Long Tail of Last.fm, a popular social music recommendation service. A small collection of artists with very high play counts make up the head, while the large majority of artist with lower play counts are found in the tail. This type of distribution is typical for all services that do the Amazon style of recommendation, the “people who bought this also bought that.” For Last.fm, the riddle goes more like “people who listen to this also listen to that”.
When a lot of people (who may otherwise have very diverse tastes in music) listen to Coldplay, Coldplay becomes very well connected with a lot of other artists, and also becomes a hub in what is known as a small-world network. Such networks are the basis for social recommendations. Oscar shows that these hubs are indeed the most popular artists, who again gets recommended more often than others. That is why all roads lead to Radiohead.
Using more formulas and correlation diagrams than most people can comfortably cope with (including me), Oscar explains why social recommenders (like Amazon) should be combined with content-based recommenders, which finds music that actually sounds the same. Such hybrid recommenders are much better at recommending novel and relevant items from The Long Tail. Recommender systems can generally be divided into three groups, based on how the recommendations are generated. These are 1) expert, 2) social and 3) content-based recommender systems. Hybrid recommenders, like Oscar proposes, combine elements mainly from the social and content-based.
Social recommendations reflect the big trends, and capture in a sense the social aspects of music. It works because most of us tend to like the same things as other people like us do. It may be difficult to break out of the demographic enclosures of this gigantic mix-tape party, though, and to discover something truly novel. That, combined with popularity bias (rich get richer effect), cold-start problems and early raters feedback loops, puts some limitations on social recommenders in terms of long tail reach.
Expert recommendations capture more of the cultural aspects of music, things that aren’t evident from our listening habits. A music expert may compile lists of great songs produced by Bob Rock, or top love songs from the 80′s. Such recommendations captures many beautiful and odd aspects of music, but the major drawback is the lack of scalability. No expert can know and recommend all the music in the world.
Content-based (CB) recommendations are often produced by sophisticated music analysis software, comparing the actual sound of the music itself. This “objective” focus on stylistic similarity may surprisingly cross over established culture-dependent genres. CB recommendations scale well to millions of songs, but data sparsity may also produce odd and incorrect recommendations.
A particularly interesting nuggets of information from Oscar’s thesis work is that it for social recommenders take 5 links/clicks/jumps to reach from the head to the long tail, while it takes just 2 for expert and CB recommenders. Wow!
If you like to know more, head over to Oscar’s PhD thesis web page. And I’ll keep writing about recommender systems, so make sure to come back for more.
If you’re considering to implement a recommender system yourself, have a look at this presentation that was given at http://recked.org/ (via Paul Lamere):
If you think that everything does actually sound like Coldplay, this song is for you 
Hi Vegard.
While content-based recommendations is decidedly a good idea, is it really a necessity for getting rid of the Coldplay-problem[TM]?
It should be trivial (with the caveat that I have considered this for about 5 minutes) to set up a probabilistic model that takes into account the variance in connectivity. So that a co-occurrence of purchases is only considered meaningful if that co-occurrence is significant given the potentially huge sales of one or both of the items. Anyway, maybe the futility of this approach is explained in the thesis, but laziness prevailed over 200+ pages.
Of course the assumption is that vendors are actually interested in this. The current system might be deliberate as people might be more likely to purchase something they recognize (and that everyone else has) than try something new.
Hi Eivind!
I find it difficult to say how your suggestion would work. I’ll admit to my own limited understanding of social recommenders, and pass the question on to Oscar himself.
Familiarity matters when it comes to recommendations. It’s true that we’re more likely to purchase something we recognize. But it would be a shame if all we end up doing is skimming the surface of popularity. Popularity should be balanced with novelty.
Hi again, got drunk and forgot about this
> I’ll admit to my own limited understanding of social recommenders
I’m sure you know more than me
Let’s assume we have purchased record A from The Totally Extreme Niche Band and want recommendations based on that record. Looking at the purchasing connection network we see that everyone who bought A also bought B. However B is from The Red Hot Chili Beatles which absolutely everyone owns anyway so the co-occurrence of these purchases (A and B) doesn’t really mean anything. There is however another band C: The Supreme Niche Tunes that is highly similar to A, but due to it’s low sales figures is largely ignored.
Now, given this and the overall probabilities of purchases P(A)=low, P(B)=high, P(C)=low (i.e. normalized sales figures) it is possible to calculate how “surprised” one should be at co-occurrences in purchasing patterns. In our example there is likely no dependency between purchasing record A and B (because everyone buys B) so P(A) and P(B) would be independent events and their probability of co-occurrence would be simply P(A)P(B). A and C however, due to their high similarity would be highly dependent (presumably driven by some enthusiasts-savants in the genre) so the probability of their co-occurrence would be much higher than P(A)P(C). The unexpectedness of this dependency can be described in various ways, p-value/mutual information etc. which can be ranked and compared to other dependencies e.g. the dependency of P(A)P(B). The latter would be small and C would consequently be ranked above B when considered as a recommendation linked to A.
Of course there is no doubt that an some musical similarity measure (if implemented properly) could be superior. The obvious reason being that a purely probabilistic purchasing scheme depends on a certain number of enthusiast making the right purchases in the first place.
>> I’ll admit to my own limited understanding of social recommenders
> I’m sure you know more than me
Perhaps I do, but you sure know more about combinatorics than I do
I think you’re describing a system where generally popular items (artists) are penalized for their connectedness (inbound/outbound links). That makes sense, although you lost me with all the P’s. I guess, with my limited understanding of social recommenders, that it is common to implements something similar.
I really recommened checking out the presentation I’ve embedded into this post. Should be a breeze for a wicked number crusher like yourself.