Archive for category Music

Echo Nest Radio on Spotify

Spotify_-_The_Cult_–_Wild_Flower

I work for Spotify now – so for my Sunday morning programming project I thought I’d write a simple Spotify App that uses The Echo Nest API to create playlists based upon a seed song. I’ve done this before, but the last time was a few years ago and the Spotify Apps API has changed quite a bit since then, so I thought I’d use this as an opportunity to freshen my understanding of the Spotify API as well as to demonstrate how to write a Spotify App that uses The Echo Nest API.

I created an Echo Nest Radio app – it is a very simple app – it looks at what song you are currently playing and will generate an Echo Nest playlist based upon that seed song. The code is pretty straightforward. It grabs the Now Playing track from Spotify, gets the track’s ID and uses that as a seed for The Echo Nest song-radio static playlist API. This call returns Spotify track IDs (thanks to our Rosetta Stone ID mapping layer) that are then tossed into a temporary playlist, which is used to build a List view which is then displayed in the app. All told it is just over 100 lines of Javascript.

It did take me a bit of time to get the hang of the newer Spotify Apps API. It has changed quite a bit since I last used it and many of the examples that I relied on in the past, like Peter Watt’s excellent Kitchen Sink app, use an older version of the API. The new version has some significant changes including a nifty new module system along with a new approach to managing long-running queries that relies on promises. Once I got the hang of it, I decided that I like the new version – it makes for cleaner code and a much more efficient app since much less data needs to be moved around.

The app is on github – to use it you need to sign up for a developer account on Spotify and follow the rest of the Getting Started instructions (this means if you are not a developer, you’ll probably not be able to use the app).

The Spotify Apps API makes it super easy to be able to create apps that run inside Spotify. Its a very familiar programming environment for anyone who has done web programming. You can use all of your favorite libraries from jQuery to Lo-Dash to create really compelling apps that sit on top of the millions and millions of tracks in the Spotify catalog. However, unlike a web app where anyone can publish their app on the web, to publish a Spotify App you have to submit your app to the Spotify App approval process and only apps that Spotify approves are published. Spotify sets a high bar for what ultimately gets approved – which keeps the quality of the apps high, but also means that hacks and experiments built on the Spotify Apps platform will likely never be approved for release to the general public.  It’s a difficult balancing act for Spotify. They’ve built the ultimate music hacking platform with this API, but if they open the doors to every music hack, they will ultimately dilute the listening experience of the user – like so other many App stores that are flooded with garbage apps,  if they publish every app and hack then Spotify listeners would be inundated with the musical equivalent of flashlight and fart apps.  With the approval process, Spotify essentially says ‘the listener comes first’ which is the right choice.   Still, as a music hacker I do wish it was easier to publish rich music apps built on the Spotify platform. Luckily Spotify is committed to building an active and vibrant developer ecosystem so I don’t expect they we will be standing still.

Update 3/24/14: – I’ve added the ability to save these playlists back to Spotify, so you can take the Echo Nest radio playlists with you.

Second update 3/24/14 – note that Spotify’s recent announcement that they are closing app submissions means that you won’t be able to submit apps for publishing anymore, but you should be able to still create your own.

1 Comment

Beyond the Play Button – the future of listening

I’ve created a page with all the supporting info for my SXSW Talk called Beyond The Play Button – the future of listening.

Beyond the Play Button – the future of listening

The page contains slides, links to all supporting data and links to all apps demoed.

Leave a comment

The Echo Nest Is Joining Spotify: What It Means To Me, and To Developers

I love writing music apps and playing with music data. How much? Every weekend I wake up at 6am and spend a long morning writing a new music app or playing with music data. That’s after a work week at The Echo Nest, where I spend much of my time writing music apps and playing with music data. The holiday break is even more fun, because I get to spend a whole week working on a longer project like Six Degrees of Black Sabbath or the 3D Music Maze.

That’s why I love working for The Echo Nest. We have built an amazing open API, which anyone can use, for free, to create music apps. With this API, you can create phenomenal music discovery apps like Boil the Frog that take you on a journey between any two artists, or Music Popcorn, which guides you through hundreds of music genres, from the arcane to the mainstream. It also powers apps that create new music, like Girl Talk in a Box, which lets you play with your favorite song in the browser, or The Swinger, which creates a Swing version of any song:

One of my roles at The Echo Nest is to be the developer evangelist for our API — to teach developers what our API can do, and get them excited about building things with it. This is the best job ever. I get to go to all parts of the world to attend events like Music Hack Day and show a new crop of developers what they can do with The Echo Nest API. In that regard, it’s also the easiest job ever, because when I show apps that developers have built with The Echo Nest API, like The Bonhamizer and The Infinite Jukebox, or show how to create a Pandora-like radio experience with just a few lines of Python, developers can’t help but get excited about our API.

The_Echo_Nest_workshop___Flickr_-_Photo_Sharing_

Photo by Thomas Bonte

At The Echo Nest we’ve built what I think is the best, most complete music API — and now our API is about to get so much better.

spotify-logo-primary-vertical-light-background-rgbToday, we’ve announced that The Echo Nest has been acquired by Spotify, the award-winning digital music service. As part of Spotify, The Echo Nest will use our deep understanding of music to give Spotify listeners the best possible personalized music listening experience.

Spotify has long been committed to fostering a music app developer ecosystem. They have a number of APIs for creating apps on the web, on mobile devices, and within the Spotify application. They’ve been a sponsor and active participant in Music Hack Days for years now. Developers love Spotify, because it makes it easy to add music to an app without any licensing fuss. It has an incredibly huge music catalog that is available in countries around the world.

Spotify and The Echo Nest APIs already work well together. The Echo Nest already knows Spotify’s music catalog. All of our artist, song, and playlisting APIs can return Spotify IDs, making it easy to build a smart app that plays music from Spotify. Developers have been building Spotify / Echo Nest apps for years. If you go to a Music Hack Day, one of the most common phrases you’ll hear is, “This Spotify app uses The Echo Nest API”.

I am incredibly excited about becoming part of Spotify, especially because of what it means for The Echo Nest API. First, to be clear, The Echo Nest API isn’t going to go away. We are committed to continuing to support our open API. Second, although we haven’t sorted through all the details, you can imagine that there’s a whole lot of data that Spotify has that we can potentially use to enhance our API.  Perhaps the first visible change you’ll see in The Echo Nest API as a result of our becoming part of Spotify is that we will be able to keep our view of the Spotify ID space in our Project Rosetta Stone ID mapping layer incredibly fresh. No more lag between when an item appears in Spotify and when its ID appears in The Echo Nest.

The Echo Nest and the Spotify APIs are complementary. Spotify’s API provides everything a developer needs to play music and facilitate interaction with the listener, while The Echo Nest provides deep data on the music itself — what it sounds like, what people are saying about it, similar music its fans should listen to, and too much more to mention here. These two APIs together provide everything you need to create just about any music app you can think of.

I am a longtime fan of Spotify. I’ve been following them now for over seven years. I first blogged about Spotify way back in January of 2007 while they were still in stealth mode. I blogged about the Spotify haircuts, and their serious demeanor:

Those Crazy Spotify Guys in 2007

Those Crazy Spotify Guys in 2007

I blogged about the Spotify application when it was released to private beta (“Woah – Spotify is pretty cool”), and continued to blog about them every time they added another cool feature.

Last month, on a very cold and snowy day in Boston, I sat across a conference table from of bunch of really smart folks with Swedish accents. As they described their vision of a music platform, it became clear to me that they share the same obsession that we do with trying to find the best way to bring fans and music together.

Together, we can create the best music listening experience in history.

I’m totally excited to be working for Spotify now. Still, as with any new beginning, there’s an ending as well. We’ve worked hard, for many years, to build The Echo Nest — and as anyone who’s spent time here knows, we have a very special culture of people totally obsessed with building the best music platform. Of course this will continue, but as we become part of Spotify things will necessarily change, and the time when The Echo Nest was a scrappy music startup when no one in their right mind wanted to be a music startup will be just a sweet memory. To me, this is a bit like graduation day in high school — it is exciting to be moving on to bigger things, but also sad to say goodbye to that crazy time in life.

There is a big difference: When I graduated from high school and went off to college, I had to say goodbye to all my friends, but now as I go off to the college of Spotify, all my friends and classmates from high school are coming with me. How exciting is that!

en_logo_250x200_ltspotify-logo-primary-vertical-light-background-rgb

, , ,

14 Comments

Exploring regional listening preferences

In previous posts we looked at how gender and age can affect listening preferences. Today we take a look at how the location of a listener may affect their listening preferences.

distinctive_artist_map-2

For this study, I sampled the listening preferences of about a quarter million listeners that have a zip code associated with their account. update: Listener data is drawn from a variety of music streaming services that are powered by The Echo Nest. I aggregated these listeners into regions (state, regional and all-US). To compare regions I look at the top-N most popular artists  based upon listener plays in each region and look for artists that have a substantial change in rank between the two regions. These artists are the artists that define the taste for the region.

As an example let’s compare Tennessee to New England. If we look at the top 100 artists listened to in Tennessee and see which artists fall the furthest on the New England chart we find these 10 artists:

# Artist Rank in
Tennessee
Rank in
New England
Delta
1 Hillsong United 97 1097 -1000
2 Juicy J 47 664 -617
3 Young Jeezy 78 579 -501
4 R. Kelly 26 259 -233
5 2 Chainz 63 288 -225
6 Eric Church 79 286 -207
7 The 1975 87 268 -181
8 T.I. 40 211 -171
9 The Civil Wars 22 150 -128
10 Blake Shelton 58 184 -126

The first artist, Hillsong United is the 97th most popular artist in Tennessee for our sample of listeners. In New England, Hillsong United drops all the way to the 1097th artist, a drop in rank of 1,000. Hillsong United is a Christian rock worship band. Tennessee listeners, living in the heart of the U.S. Bible Belt, listen to Hillsong United much more than their New England counterparts. Second on the Tennessee list is Juicy J,a rapper, songwriter and record producer from Memphis, Tennessee. Third is Young Jeezy from nearby Atlanta Georgia.  The rest of the top 10 is dominated by southern rappers, country, and folk artists.

Now let’s look at top 100 New England artists and see which artists fall furthest on the Tennessee charts:

# Artist Rank in
New England
Rank in
Tennessee
Delta
1 Phish 89 431 -342
2 Grateful Dead 45 222 -177
3 Talking Heads 86 254 -168
4 Neil Young 53 182 -129
5 Bonobo 60 189 -129
6 Billy Joel 90 218 -128
7 Bruce Springsteen 48 175 -127
8 The XX 44 158 -114
9 Modest Mouse 85 179 -94
10 Bob Marley 30 123 -93

New England tastes run to jam bands, indie and classic rock.

Exploring Region differences on your own

I built an app that makes it easy to explore the regional differences in music. With the app you can select two regions and it will show you which artists are distinctive for each region.

Regionalisms_in_U_S__Listening_Preferences

Using this approach we can pick a single signature artist for each state. I do this by finding the top most distinctive popular artist for a state that hasn’t already been selected for a more populous state. Here’s the list of signature state artists:

State Name State Rank US Rank Change
AK Ginger Kwan 33 12062 -12029
AL The Civil Wars 34 110 -76
AR Wiz Khalifa 15 69 -54
AZ Linkin Park 47 98 -51
CA Bonobo 28 85 -57
CO The Naked And Famous 48 125 -77
CT David Guetta 43 80 -37
DE Rush 38 385 -347
FL Rick Ross 44 103 -59
GA Future 33 226 -193
HI J Boog 40 4703 -4663
IA B.o.B 32 82 -50
ID Tegan and Sara 31 107 -76
IL Sufjan Stevens 38 68 -30
IN Blake Shelton 31 91 -60
KS Eric Church 48 162 -114
KY Fall Out Boy 17 59 -42
LA Kevin Gates 15 1359 -1344
MA Neil Young 49 128 -79
MD Kelly Rowland 45 217 -172
ME R.E.M. 28 197 -169
MI Young Jeezy 34 138 -104
MN Metric 41 211 -170
MO The Shins 29 67 -38
MS August Alsina 42 909 -867
MT Tim McGraw 36 187 -151
NC Miguel 37 88 -51
ND Stone Sour 47 735 -688
NE Bastille 48 89 -41
NH Grateful Dead 29 191 -162
NJ Bruce Springsteen 23 109 -86
NM Alan Jackson 45 425 -380
NV Ciara 34 106 -72
NY James Blake 47 130 -83
OH Florida Georgia Line 26 75 -49
OK Jason Aldean 24 84 -60
OR Kurt Vile 49 212 -163
PA Edward Sharpe & the Magnetic Zeros 43 77 -34
RI Nirvana 48 126 -78
SC Hillsong United 24 262 -238
SD Hinder 42 1154 -1112
TN Juicy J 47 151 -104
TX George Strait 16 199 -183
UT AWOLNATION 31 111 -80
VA Dave Matthews Band 48 113 -65
VT Phish 5 353 -348
WA The Head and the Heart 22 83 -61
WI Jack Johnson 45 99 -54
WV matchbox twenty 48 282 -234
WY Dirty Heads 48 1334 -1286

distinctive_artist_map-2

It is pretty clear that people in different parts of the US listen to different kinds of music. These regionalisms can be used to help recommend music for people when you otherwise might not know anything about their music taste.

Note – the base map in the visualizations is shaded by population density.

302 Comments

The Tokyo Tune Train

I spent this weekend in the Harajuku neighborhood of Tokyo at Music Hack Day Tokyo. Inspired by the Tokyo Subway, and the love many people in Japan have for music games I thought I would build a Tokyo Subway-themed music game for my hack.  The result is the Tokyo Tune Train.

The Tokyo Tune Train

The Tokyo Tune Train

The goal of the game is to drive a Tokyo Subway car through the maze of routes and make it to the final stop with the fewest deviations from the track (in my world, the trains don’t stay on the tracks unless you steer them). It’s a bit like Centipede or Snake, but with a train track and music integrated into the game play.

Tokyo_Tune_TrainFirst_Of_The_Year__Equinox__by_Skrillex

Headlights help you see up coming turns

The movement of the train is synchronized to a song – with every beat of the song, the train advances a hundred feet down the track.  As long as you stay on the track, the song will play perfectly, but if you miss a turn, not only will you veer off the tracks you will also veer into a completely different part of the song.  If you stop the train, the song will stop and continually play that one beat until you start again. If you go backwards the song will play backwards.  You can speed the train up and slow it down (which, of course, also speeds up or slows down the song). A faster train is harder to drive but gets you to the destination quicker.  You get points for good driving and getting to the destination in the quickest time. There’s an adjustable headlight on the train – the brighter the headlight, the further ahead you can see, making the drive easier, but the headlight uses lots of energy, so a brighter headlight will cost you a number of points. You can also drive during the day letting you see the whole line stretched out in front of you. It’s much easier to navigate, but driving during the day costs more points than (access to the tracks during the day is very expensive in my train system). There’s even an autopilot that will take over and drive the train perfectly for you, but it chews through points very rapidly, so use it sparingly.

This is the first game I’ve built in a long time. My goal was to make something that was fun to play in which music was an integral part of the game.

The Technical Bits

At the core of the game is the Echo Nest remix library. I essentially ripped Girl Talk in a Box apart and put in a new play engine that was tied to the game mechanic. The harder bit was figuring out how to build good labyrinth mazes that would get progressively twistier and would fill a 2-D plane.  I played around with a number of algorithms, but none were satisfactory, so instead, I built a maze editor that lets you hand draw mazes and then export the data as json to be used by the app.

Maze_Draw

I enlisted the help of Elissa and Eric to build the mazes. They created 15 awesomely creative mazes that met the needs of the game much better than any maze algorithm, again confirming the superiority of human curators over the algorithm.

Professional maze builders  @elissab and @eswens conducting research at The Scramble

Professional labyrinth builders @elissab and @eswens conducting maze research at The Scramble

I briefly took a look at using Google Play Game service to manage the high-scores, but it looked like it would take at least an hour of precious hacking time to integrate and since it is a feature that wouldn’t appear in the demo, I decided it was not worth the effort.

The Demo

It was a great demo session at Music Hack Day Tokyo. There were easily 150 people crammed into the demo area – it was standing room only.  The demo session was conducted in Japanese, so Taishi provided simultaneous translation for me.  I was nervous giving the demo, so I let the train autopilot do most of the driving.  The crowd seemed to really enjoy the demo, the hack got lots of votes for ‘favorite hack‘ on the hacker league page of hacks, and I got many compliments. I was pleased with the response and feedback

Photo by @timfalls

Taishi translated the demo while I try to drive a train in front of 150 hackers. Photo by @timfalls

A day later I see that a number of people have played the game and have posted their scores to twitter which is great to see (way better response than I received for my dark horse hack).

twitter feedback

twitter feedback

Give the game a try and let me know what you think:

Ride the Equinox line of the Tokyo Tune Train

The code is on github.

1 Comment

Exploring age-specific preferences in listening

Earlier this week we looked at how gender can affect music listening preferences. In this post, we continue the tour through demographic data and explore how the age of the listener tells us something about their music taste.

Where does the age data come from? As part of the enrollment process for most music services, the user is asked for a few pieces of demographic data, including gender and year-of-birth.  As an example, here’s a typical user-enrollment screen from a popular music subscription service:

iPhoto-4

 

Is this age data any good? The first thing we need to do before we get too far with analyzing the age data is to get a feel for how accurate it is. If new users are entering random values for their date of birth then we won’t be able to use the listener’s age for anything useful. For this study, I looked at the age data submitted by several million listeners. This histogram shows the relative number of users by year of birth.

Gnuplot_and_1__gnuplot-2

The first thing I notice is the curve has the shape one would expect. The number of listeners in each age bucket increases as the listener gets younger until around age 21 or so, at which points it drops off rapidly.  The shape of the curve aligns with the data from this study by EMI in 2011 that shows the penetration of music streaming service by age demographic. This is a good indicator that our age data is an accurate representation of reality.

With_Streaming_and_Sharing__Teens_Find_Ways_Around_Paying_for_Music_-_eMarketer-2

However, there are a few anomalies in the age data. There are unexpected peaks at each decade – likely due to people rounding their birth year to the nearest decade. A very small percentage (0.01 %) indicate that they are over 120 years old, which is quite unlikely. Despite this noise, the age data looks to be a valid and fairly accurate representation, in the aggregate, of the age of listeners.  We should be able to use this data to understand how age impacts listening.

Does a 64-year-old listen to different music than a 13-year-old?
One would expect that people of different ages would have different music tastes. Let’s see if we can confirm this with our data.  For starters, lets compare the average listening habits of 64-year-old listeners to that of the aggregate listening habits of the 13-year-old listener. For this experiment I selected 5,000 listeners in each age category, and aggregated their normalized artist plays to find the most-frequently-played artists.  As expected, you can see that 64-year-old listeners have different tastes than 13-year-old listeners.

Untitled_2

The top artists for the average 64-year-old listener include a mix of currently popular artists along with a number of artists from years gone by. While the top artists for the average 13-year-old includes only the most current artists.  Still, there are seven artists (shown in bold) that overlap in the top 20 – an overlap rate of about 35%. This 35% overlap is consistent across all ranges of top artists for the two groups.  No matter if we look at the top 100 or the top 1000 artists – there’s about a 35% overlap between the listening of 13- and 64-year-olds.

Untitled_2

I suspect that 35% overlap is actually an overstatement of the real overlap between 13- and 64-year-olds.  There are a few potential confounding effects:

  • There’s a built-in popularity bias in music services.  If you go to any popular music service you will see that they all feature a number of playlists filled with popular music. Playlists like The Billboard Top 100, The Viral 50, The Top Tracks, Popular New Releases etc. populate the home page or starting screen for most music services. This popularity bias inflates the apparent interest in popular music so, for instance, it may look like a 64-year-old is more interested in popular music than they really are because they are curious about what’s on all of those featured playlists.
  • The age data isn’t perfect – for instance, there are certainly a number of people that we think are 64-years-old but are not. This will skew the results to artists that are more generally popular. We don’t really know how big this affect is, but it is certainly non-zero.
  • People share listening accounts – this is perhaps the biggest confounding factor – that 64-year-old listener may be listening to music with their kids, their grand-kids, their neighbors and friends which means that not all of those plays should count as plays by a 64-year-old.  Again, we don’t know how big this effect is, but it is certainly non-zero.

Let’s pause and have a listen to some music. First the favorite music of a typical 64-year-old listener:

Rdio Version

And now the favorite music of a typical 13-year-old listener:

Rdio Version

Finding the most distinctive artists

Perhaps more interesting than looking at how the two ages overlap in listening, is to look at how they differ – what are the artists that a 64-year-old will listen to that are rarely, if ever, listened to by a 13-year-old and vice versa.  These are the most distinctive artists.

Edit_Post_‹_Music_Machinery_—_WordPress

We can find the distinctive artists by identifying the artists in the top 100 of one group that fall the furthest in ranking in the other group. For example Skrillex is the 40th most listened to artist for the typical 13-year-old listener, but for 64-year-old listeners, Skrillex falls all the way to the 3,937 most listened to artist, making Skrillex one of the most distinguishing artist between the two groups of listeners. Likewise, Roy Orbison is the 42nd most listened to artist among 64-year-olds. He drops to position 4,673 among 13-year-olds making him one of the distinguishing artists that separate the 64-year-old from the 13-year-old.

We can use this technique to create playlists of artists that separate the 13-year-old from the 64-year-olds.   Are you a 13-year-old, having a party and really wish that grandma would go to another room? Try this playlist:

Rdio version

Are you a 64-year-old and you want all of those 13 year-olds at the party to go home? Try this playlist:

Rdio version

We can also use this data to bring these two groups together. We can find the music that is liked the most among the two groups.  We can do this by ordering artists by their worst ranking among the two groups. Artists like Skrillex and Roy Orbison fall to the bottom of the list since each is poorly ranked by one of the groups, while artists like Katy Perry and Bruno Mars rise to the top because they are favored by both groups.

Again, the confounding factors mentioned previously will bias the shared lists to more popular music. Nevertheless, if you are trying to make a playlist of music that will please both a 64-year-old and a 13-year-old, and you know nothing else about their music taste, this is probably your best bet.

Artists that are favored by both 64-year-old and 13-year-old listeners are: Bruno Mars, Taylor Swift, P!nk, Rihanna, Justin Timberlake, Katy Perry, Robin Thicke, Maroon 5, Lana Del Rey, Daft Punk, Beyoncé, Drake, Luke Bryan, Adele, Macklemore & Ryan Lewis, Miley Cyrus, David Guetta, Lorde, Jay-Z, Usher

Rdio version

We can sum up the differences between the two groups in this graphic:

age-differences-13-64-2

Broadening our view

We’ve shown that, as expected, 13-year-olds and 64-year-olds have different listening preferences. We can apply the same techniques across the range of age demographics typically used by marketers. We can find the most distinctive artists for each demographic bucket.  It is interesting to see the progression of music taste over time. For instance, it is clear that something happens to a music listener between the 25 to 34 and 35 to 44 age buckets. The typical listener goes from hipster (Lumineers, Vampire Weekend, The National), to old (Pearl Jam, U2, Bon Jovi).

Untitled

It is interesting to look at the starting year for artists in each of these buckets to get a sense of how the artist’s own age relates to the age of their fans:

Untitled_2

My take-way from this is that no matter how old you are, you don’t like the music from the 70s and the 80s so much.

Most homogenous Artists

We can also find the artists that are most acceptable across all demographics. These are the artists that are liked by more listeners in all of the groups. Like in the 13/64-year-old example, we can find these artists by ordering them by their worst ranking among all the demographic groups.

Most homogeneous artists: Bruno Mars, Rihanna, Katy Perry, Lana Del Rey, Beyoncé, P!nk, Jay-Z, Macklemore & Ryan Lewis, Daft Punk, Maroon 5, Justin Timberlake, Robin Thicke, David Guetta, Luke Bryan, Taylor Swift, Drake, Adele, Imagine Dragons, Miley Cyrus, Lorde

This is essentially the list of the most popular artists but with the most polarizing artists from any one demographic removed.  If you don’t know the age of your listener, and you want to give the listener a low risk listening experience, these artists are a good place to start.  And yes … this results in a somewhat bland, non-adventurous listening session – that’s the point. But as soon as you know a bit about the true listening preference of a new listener, you can pivot away from the bland and give them something much more in line with their music taste.

Rdio version

Rounding out the stats

There are a few more interesting bits of data we an pull out related to the age of the listener

Average number of artists in listening rotation

The typical 25- to 34-year old listener has more artists in active rotation than any other age group, while the 65+ listeners have the least.

Untitled_2

Relative number of plays per user by age group

Likewise, the typical 25- to 34-year-old listener plays more music than any other category.

Untitled_2

Tying it all up …

This quick tour through the ages confirms our thinking that the age of a listener plays a significant role in the type of music that they listen to.  We can use this information to find music that is distinctive for a particular demographic.  We can also use this information to help find artists that may be acceptable to a wide range of listeners. But we should be careful to consider how popularity bias may affect our view of the world. And perhaps most important of all, people don’t like music from the 70s or 80s so much.

,

7 Comments

Gender Specific Listening

One of the challenges faced by a music streaming service is to figure out what music to play for the brand-new listener.  The first listening experience of a new listener can be critical to gaining that listener as a long time subscriber. However, figuring out what to play for that new listener is very difficult because often there’s absolutely no data available about what kind of music that listener likes. Some music services will interview the new listener to get an idea of their music tastes.

beats-enrollment

Selecting your favorite genres is part of the nifty user interview for Beat’s music

However, we’ve seen that for many listeners, especially the casual and indifferent listeners, this type of enrollment may be too complicated. Some listeners don’t know or care about the differences between Blues, R&B and Americana and thus won’t be able to tell you which they prefer. A listener whose only experience in starting a listening session is to turn on the radio may not be ready for a multi-screen interview about their music taste.

So what can a music service play for a listener when they have absolutely no data about that listener? A good place to start is to play music by the most popular artists.  Given no other data,  playing what’s popular is better than nothing. But perhaps we can do better than that. The key is in looking at the little bit of data that a new listener will give you.

For most music services, there’s a short user enrollment process that gets some basic info from the listener including their email address and some basic demographic information. Here’s the enrollment box for Spotify:

Music_for_everyone__-_Spotify_-__Private_Browsing_

Included in this information is the date of birth and the gender of the listener. Perhaps we can use basic demographic data to generate a slightly more refined set of artists. For starters, lets consider gender.  Let’s try to answer the question: If we know that a listener is male or female does that increase our understanding of what kind of music they might like?  Let’s take a look.

Exploring Gender Differences in Listening
Do men listen to different music than women do? Anecdotally, we can think of lots of examples that point to yes – it seems like more of One Direction’s fans are female, while more heavy metal fans are male, but lets take a look at some data to see if this is really the case.

The Data – For this study,  I looked at the recent listening of about 200 thousand randomly selected listeners that have self-identified as either male or female.  From this set of listeners, I tallied up the number of male and female listeners for each artist and then simply ranked the artists in order or listeners. Here’s a quick look at the top 5 artists by gender.

Top 5 artists by gender

Rank All Male Female
1 Rihanna Eminem Rihanna
2 Bruno Mars Daft Punk Bruno Mars
3 Eminem Jay-Z Beyoncé
4 Katy Perry Bruno Mars Katy Perry
5 Justin Timberlake Drake P!nk

Among the top 5 we see that the Male and Female listeners only share one artist in common:Bruno Mars.  This trend continues as we look at the top 40 artists. Comparing lists by eye can be a bit difficult, so I created a slopegraph visualization to make it easier to compare. Click on this image to see the whole slopegraph:

click for full chart

click for full chart

Looking at the top 40 charts artists we see that more than a quarter of the artists are gender specific. Artists that top the female listener chart but are missing on the male listener chart include: Justin Bieber, Demi Lovato, Shakira, Britney Spears, One Direction, Christina Aguilera, Ke$ha, Ciara, Jennifer Lopez, Avril Lavigne and Nicki Minaj. Conversely, artists that top the male listener chart but are missing on the top 40 female listener chart include: Bob Marley, Kendrick Lamar, Wiz Khalifa, Avicii, T.I. Queen, J.Cole, Linkin Park, Kid Cudi and 50 Cent. While some artists seem to more easily cross gender lines like Rihanna, Justin Timberlake, Lana Del Rey and Robin Thicke.

No matter what size chart we look at – whether it is the top 40, top 200 or the top 1000 artists – about 30% of artists on a gender-specific chart don’t appear on the corresponding chart for the opposite gender.  Similarly, about 15% of the artists that appear on a general chart of top artists will be of low relevance to a typical listener based on these gender-listening differences.

What does this all mean?  If you don’t know anything about a listener except for their gender, you can reduce the listener WTFs by 15% for a typical listener by restricting plays to artists from the gender specific charts.  But perhaps even more importantly, we can use this data to improve the listening experience for a listener even if we don’t know a listener’s gender at all.  Looking at the data we see that there are a number of gender-polarizing artists on any chart. These are artists that are extremely popular for one gender, but not popular at all for the other.  Chances are that if you play one of these polarizing artists for a listener that you know absolutely nothing about, 50% of the time you will get it wrong.  Play One Direction and 50% of the time the listener won’t like it, just because 50% of the time the listener is male.  This means that we can improve the listening experience for a listener, even if we don’t know their gender by eliminating the gender skewing artists and replacing them with more gender neutral artists.

Let’s see how this would affect our charts.  Here are the new Top 40 artists when we account for gender differences.

Rank Old Rank Artist
1 2 Bruno Mars
2 1 Rihanna
3 5 Justin Timberlake
4 4 Katy Perry
5 6 Drake
6 15 Chris Brown
7 3 Eminem
8 8 P!nk
9 11 David Guetta
10 14 Usher
11 17 Maroon 5
12 7 Jay-Z
13 13 Adele
14 9 Beyoncé
15 12 Lil Wayne
16 23 Lana Del Rey
17 25 Robin Thicke
18 24 Pitbull
19 27 The Black Eyed Peas
20 19 Lady Gaga
21 20 Michael Jackson
22 10 Daft Punk
23 18 Miley Cyrus
24 22 Macklemore & Ryan Lewis
25 28 Coldplay
26 16 Taylor Swift
27 26 Calvin Harris
28 21 Alicia Keys
29 29 Imagine Dragons
30 30 Britney Spears
31 44 Ellie Goulding
32 31 Kanye West
33 42 J. Cole
34 41 T.I.
35 52 LMFAO
36 32 Shakira
37 35 Bob Marley
38 54 will.i.am
39 36 Ke$ha
40 39 Wiz Khalifa

Artists promoted to the chart due to replace gender-skewed artists are in bold. Artists that were dropped from the top 40 are:

  • Avicii – skews male
  • Justin Bieber – skews female
  • Christina Aguilera – skews female
  • One Direction – skews female
  • Demi Lovato – skews female

Who are the most gender skewed artists?

The Top 40  is a fairly narrow slice of music. It is much more interesting to look at how listening can skew across a much broader range of music.  Here I look at the top 1,000 artists listened to by males and the top 1,000 artists listened to by females and find the artists that have the largest change in rank as they move from the male chart to the female chart. Artists that lose the most rank are artists that skew male the most, while artists that gain the most rank skew female.

Top male-skewed artists:
artists that skew towards male fans

  • Iron Maiden
  • Rage Against the Machine
  • Van Halen
  • N.W.A
  • Jimi Hendrix
  • Limp Bizkit
  • Wu-Tang Clan
  • Xzibit
  • The Who
  • Moby
  • Alice in Chains
  • Soundgarden
  • Black Sabbath
  • Stone Temple Pilots
  • Mobb Deep
  • Queens of the Stone Age
  • Ice Cube
  • Kavinsky
  • Audioslave
  • Pantera

Top female-skewed artists:
artists that skew towards female fans

  • Danity Kane
  • Cody Simpson
  • Hannah Montana
  • Emily Osment
  • Playa LImbo
  • Vanessa Hudgens
  • Sandoval
  • Miranda Lambert
  • Sugarland
  • Aly & AJ
  • Christina Milian
  • Noel Schajris
  • Maria José
  • Jesse McCartney
  • Bridgit Mendler
  • Ashanti
  • Luis Fonsi
  • La Oreja de Van Gogh
  • Michelle Williams
  • Lindsay Lohan

Gender-skewed Genres

By looking at the genres of the most gender skewed artists we can also get a sense of which genres are most gender skewed as well.  Looking at the genres of the top 1000 artists listened to by male listeners and the top 1000 artists with female listeners we identify the most skewed genres:

Genres most skewed to female listeners:

  • Pop
  • Dance Pop
  • Contemporary  Hit Radio
  • Urban Contemporary
  • R&B
  • Hot Adult Contemporary
  • Latin Pop
  • Teen Pop
  • Neo soul
  • Latin
  • Pop rock
  • Contemporary country

Genres most skewed to male listeners:

  • Rock
  • Hip Hop
  • House
  • Album Rock
  • Rap
  • Pop Rap
  • Indie Rock
  • Funk Rock
  • Gangster Rap
  • Electro house
  • Classic rock
  • Nu metal

Summary

This study confirms what we expected – that there are differences in gender listening. For mainstream listening about 30% of the artists in a typical male’s listening rotation won’t be found in a typical female listening rotation and vice versa. If we happen to know a listener’s gender and nothing else, we can improve their listening experience somewhat by replacing artists that skew to the opposite gender with more neutral artists.  We can even improve the listening experience for a listener that we know absolutely nothing about – not even their gender – by replacing gender-polarized artists with artists that are more accepted by both genders.

Of course when we talk about gender differences in listening, we are talking about probabilities and statistics averaged over a large number of people. Yes, the typical One Direction fan is female, but that doesn’t mean that all One Direction fans are female.  We can use gender to help us improve the listening experience for a brand new user, even if we don’t know the gender of that new user. But I suspect the benefits of using gender for music scheduling is limited to helping with the cold start problem. After a new user has listened to a dozen or so songs, we’ll have a much richer picture of the type of music they listen to – and we may discover that the new male listener really does like to listen to One Direction and Justin Bieber and that new female listener is a big classic rock fan that especially likes Jimi Hendrix.

update – 2/13 – commenter AW suggested that the word ‘bias’ was too loaded a term. I agree and have changed the post replacing ‘bias’ with ‘difference’

24 Comments

Follow

Get every new post delivered to your Inbox.

Join 1,399 other followers