Archive for category data
The Playlist Miner
The Spotify Web API team pushed out a new feature recently that extends the search API to include playlist search. With this new feature it is now possible to search across all of the popular public playlists created by Spotify users. To try out the new search capability I created a new web app called The Playlist Miner.
The Playlist Miner is a web app that will create a Spotify playlist for you by finding the top songs in all of the playlists that match your criteria. Say, for example, that I want to create a dinner party playlist. First, I find the top playlists that match ‘dinner party’ with The Playlist Miner:
The Playlist Miner will find up to the top 1,000 most popular playlists that match dinner party. It shows them to me, giving me a chance to refine my query to focus in on the exact type of playlist that I am interested in.
For this first try, I see lots of Christmas-oriented playlists (‘Tis the Season after all), but since I’m looking for music for a post-holiday dinner party, I’d rather not have holiday music in the playlist. So I refine my query to find non-Christmas oriented dinner party playlists like so:
The resulting playlists are suitably non-Christmasy.
I like the look of these playlists so I hit the Find Top Tracks button and The Playlist Miner will scour through all of the matching playlists (290 of them in this case) and find the most frequently appearing tracks.
Once the top 100 tracks are found, I can save them to Spotify as my own playlist.
Selecting Prefer more distinctive labor and delivery tracks adjusts the track order for popularity so that tracks that are more distinctive to the particular playlist context will rise to the top. You can also use logical operators to focus in on the exact type of playlist you want to. You can search for “work out” OR workout NOT running to find workout playlists without running in their titles/descriptions.
Under the hood – The Playlist Miner uses lots of bits of the Spotify API – user authentication, playlist search, playlist reading, playlist saving and more. The app is a an API calling beast – aggregating all the tracks from a thousand playlists requires 1,000 API calls. It’s a testament to the Spotify Web API that it doesn’t even blink under the load. You can play with the code on github.
It’s fun to use The Playlist Miner to explore the quirkier aspects of how people listen to music. There are ironing playlists and sleeping baby playlists. There are playlists for getting psyched and playlists for Labor and Delivery. With the Playlist Miner you can pull from all the playlists created for a particular purpose and build your own. Give it a try.
The Million Songs of Christmas
No other holiday dominates our listening like Christmas. During this season, we are exposed to a seemingly never ending playlist of Christmas music. So its no surprise that there’s a huge amount of Christmas music available on Spotify. How much? Let’s take a look.
How much Christmas music is there?
It is actually quite hard to pinpoint the exact number of Christmas songs. First, every week during the holiday season thousands more Christmas songs are added to the set. Second, some songs are seasonal – is Frosty The Snowman a Christmas song? Not literally, but it gets a lot of play at this time of year, even by the antipodes. Finally, there are a number of other holidays and celebrations at this time of year such as Hanukkah, Boxing Day, New Years, Kwanzaa, the Winter Solstice, and Festivus that we want to include in this category. So when I say “Christmas Music” I’m referring to western music that is played primarily during December. There’s probably a better term to describe this music, but terms like seasonal, and holiday have their own special baggage – perhaps something like music coincident with the northern hemispheric winter solstice is the most precise description, but lets stick with Christmas music just to keep things simple. So how much Christmas music is there? In early December 2014, crack music + data nerd Aaron Daubman dove into the Spotify + Echo Nest music catalog and found 914,047 Christmas tracks – that’s just under a million Christmas tracks. Let’s unwrap this dataset to see what we can find.
First, some basic stats: Those 914,047 tracks represent 180,660 unique songs and were created by 63,711 unique artists – from Aaron Neville to Zuma the King. The top 20 artists with the most Christmas tracks in the Spotify catalog are all pre-Beatles artists:
Artists with the most Christmas Tracks
| # | Name | Count |
|---|---|---|
| 1 | Bing Crosby | 22382 |
| 2 | Frank Sinatra | 17979 |
| 3 | Elvis Presley | 12381 |
| 4 | Nat King Cole | 11613 |
| 5 | Johann Sebastian Bach | 8958 |
| 6 | Dean Martin | 8000 |
| 7 | Perry Como | 7529 |
| 8 | Ella Fitzgerald | 6428 |
| 9 | Mahalia Jackson | 5883 |
| 10 | Mario Lanza | 5377 |
| 11 | Johnny Mathis | 5036 |
| 12 | Rosemary Clooney | 4538 |
| 13 | Peggy Lee | 4450 |
| 14 | Harry Belafonte | 4054 |
| 15 | The Andrews Sisters | 3567 |
| 16 | Louis Armstrong | 3481 |
| 17 | Gene Autry | 3411 |
| 18 | Doris Day | 2985 |
| 19 | Pat Boone | 2767 |
| 20 | Connie Francis | 2500 |
Yes, that’s right, Bing Crosby has 22,382 different Christmas tracks (!) in the Spotify catalog. Now, a little digression on what we consider to be a unique track. Music, especially popular music, is released in many forms. A very popular song, such as Bing Crosby’s White Christmas, may appear on a wide range of albums – from the original studio release to a plethora of Christmas Compilations and artist ‘best of’ albums. Each of these track releases may have different album art, different rights holders and regional licenses. Thus, even though the audio for White Christmas may be the same on each of the release, we consider each release as a different track.
White Christmas
Let’s take a closer look at Bing Crosby’s White Christmas. In our catalog of nearly a million Christmas tracks, 2,196 of them are Bing Crosby’s classic. I’ll say that again, just because it is a rather phenomenal fact – there are 2,196 different albums on Spotify that contain Bing’s White Christmas. It is hard to believe, so I created a web page that contains all 2,196 of the albums so you can see them all. Click on the image below to load them all up (warning – with 2000+ album covers it’s a bit of a browser buster).
White Christmas isn’t the only uber-track of the holidays. Here are the top 25 Christmas tracks based upon the number of times they have been released on an album:
The most released Christmas tracks
| # | Name | Count |
| 1 | Bing Crosby – White Christmas | 2196 |
| 2 | Eartha Kitt – Santa Baby | 1286 |
| 3 | Elvis Presley – Blue Christmas | 1285 |
| 4 | Frank Sinatra – Jingle Bells | 1121 |
| 5 | Harry Belafonte – Mary’s Boy Child | 904 |
| 6 | Bing Crosby – Silver Bells | 881 |
| 7 | Nat King Cole – The Christmas Song | 870 |
| 8 | Frank Sinatra – The Christmas Waltz | 811 |
| 9 | Rosemary Clooney – Suzy Snowflake | 788 |
| 10 | Bobby Helms – Jingle Bell Rock | 779 |
| 11 | Elvis Presley – White Christmas | 738 |
| 12 | Judy Garland – Have Yourself a Merry Little Christmas | 735 |
| 13 | Frank Sinatra – White Christmas | 703 |
| 14 | Frank Sinatra – Christmas Dreaming | 696 |
| 15 | Frank Sinatra – Have Yourself a Merry Little Christmas | 695 |
| 16 | Elvis Presley – Silent Night | 688 |
| 17 | Elvis Presley – I Believe | 664 |
| 18 | Frank Sinatra – Santa Claus Is Coming to Town | 660 |
| 19 | Louis Armstrong – Zat You Santa Claus | 598 |
| 20 | Dean Martin – The Christmas Blues | 575 |
| 21 | Frank Sinatra – Mistletoe and Holly | 568 |
| 22 | Louis Armstrong – Cool Yule | 566 |
| 23 | Frank Sinatra – Silent Night | 563 |
| 24 | Bing Crosby – Jingle Bells | 560 |
| 25 | Elvis Presley – Santa Claus Is Back in Town | 559 |
You can see all of the releases for Elvis’s Blue Christmas and Eartha Kitt’s Santa Baby here:
So there are lots of copies of Bing Crosby’s White Christmas and Eartha Kitt’s Santa Baby out there – but what are the most common Christmas songs overall? Which ones have been recorded the most by any artist? The following table shows the top 25:
Most recorded songs
| # | Name | Recordings |
| 1 | Silent Night | 19041 |
| 2 | White Christmas | 15928 |
| 3 | Jingle Bells | 14521 |
| 4 | Winter Wonderland | 9524 |
| 5 | Joy to the World | 9093 |
| 6 | The First Noel | 8731 |
| 7 | Have Yourself a Merry Little Christmas | 8511 |
| 8 | O Holy Night | 7925 |
| 9 | Hark The Herald Angels Sing | 7727 |
| 10 | The Christmas Song | 7673 |
| 11 | Away in a Manger | 7544 |
| 12 | God Rest Ye Merry Gentlemen | 7524 |
| 13 | O Little Town of Bethlehem | 7480 |
| 14 | Santa Claus Is Coming To Town | 6851 |
| 15 | I’ll Be Home for Christmas | 6844 |
| 16 | O Come All Ye Faithful | 6273 |
| 17 | Deck The Halls | 6057 |
| 18 | Silver Bells | 6044 |
| 19 | Ave Maria | 5847 |
| 20 | What Child Is This? | 5755 |
| 21 | We Wish You A Merry Christmas | 5619 |
| 22 | It Came Upon A Midnight Clear | 5019 |
| 23 | Sleigh Ride | 5004 |
| 24 | Blue Christmas | 4688 |
| 25 | Let It Snow! Let It Snow! Let It Snow! | 4598 |
Of course this data may be confounded by the uber-tracks like White Christmas that have thousands of versions by a single artist, so lets look at the most recorded songs by unique artists – that is, we only count Bing Crosby once for White Christmas instead of 2,196 times. When we do that the top 25 changes a bit:
Most recorded Christmas songs (Unique Artists)
| # | Name | Recordings |
| 1 | Silent Night | 7406 |
| 2 | Jingle Bells | 4485 |
| 3 | Joy to the World | 3593 |
| 4 | White Christmas | 3592 |
| 5 | O Holy Night | 3536 |
| 6 | The First Noel | 3181 |
| 7 | What Child Is This? | 3150 |
| 8 | Away in a Manger | 3140 |
| 9 | God Rest Ye Merry Gentlemen | 2871 |
| 10 | Have Yourself a Merry Little Christmas | 2823 |
| 11 | O Come All Ye Faithful | 2675 |
| 12 | Hark The Herald Angels Sing | 2638 |
| 13 | Angels We Have Heard on High | 2494 |
| 14 | Winter Wonderland | 2489 |
| 15 | The Christmas Song | 2398 |
| 16 | We Wish You A Merry Christmas | 2281 |
| 17 | Deck The Halls | 2274 |
| 18 | O Little Town of Bethlehem | 2197 |
| 19 | We Three Kings | 2048 |
| 20 | Santa Claus Is Coming To Town | 1837 |
| 21 | It Came Upon A Midnight Clear | 1768 |
| 22 | Ave Maria | 1705 |
| 23 | Auld Lang Syne | 1603 |
| 24 | Silver Bells | 1599 |
| 25 | I’ll Be Home for Christmas | 1577 |
The songs in green are the songs that are unique to each list.
Artists with the most number of unique songs
Bing Crosby is at the top of the Most Christmasy artists mainly because of the widespread re-issuing of White Christmas. But if we look at unique songs (i.e. White Christmas only counts once for Bing Crosby), the top Christmas artists look very different – with classical composers, Karaoke ‘artists’ and music factories topping the charts:
Artists with the most number of unique songs
| 1 | Johann Sebastian Bach | 3681 |
| 2 | Bing Crosby | 1462 |
| 3 | The Karaoke Channel | 1098 |
| 4 | George Frideric Handel | 903 |
| 5 | A-Type Player | 835 |
| 6 | Frank Sinatra | 816 |
| 7 | ProSound Karaoke Band | 762 |
| 8 | Pyotr Ilyich Tchaikovsky | 691 |
| 9 | SBI Audio Karaoke | 641 |
| 10 | Mega Tracks Karaoke Band | 577 |
| 11 | ProSource Karaoke | 539 |
| 12 | Ameritz Karaoke Entertainment | 508 |
| 13 | Tbilisi Symphony Orchestra | 506 |
| 14 | Elvis Presley | 472 |
| 15 | Perry Como | 440 |
| 16 | Karaoke – Ameritz | 428 |
| 17 | Nat King Cole | 413 |
| 18 | Ameritz Karaoke Band | 397 |
| 19 | Merry Tune Makers | 385 |
| 20 | Christmas Songs | 370 |
Current popular Christmas crooner Michael Bublé, with 31 unique Christmas songs has a way to go before he makes it on to the most-unique-songs-recorded chart.
Speaking of Karaoke – there’s lots of Christmas Karaoke – 23,472 tracks to be precise. The top 25 Karaoke songs are the classics:
Top Karaoke Christmas Songs
| # | Name | Count |
| 1 | White Christmas | 345 |
| 2 | Winter Wonderland | 333 |
| 3 | Silent Night | 312 |
| 4 | Jingle Bells | 309 |
| 5 | Last Christmas | 258 |
| 6 | Silver Bells | 219 |
| 7 | Blue Christmas | 204 |
| 8 | Santa Baby | 189 |
| 9 | The Christmas Song | 185 |
| 10 | Jingle Bell Rock | 172 |
| 11 | Have Yourself a Merry Little Christmas | 171 |
| 12 | Please Come Home for Christmas | 163 |
| 13 | Little Drummer Boy | 163 |
| 14 | Sleigh Ride | 156 |
| 15 | O Come All Ye Faithful | 154 |
| 16 | Here Comes Santa Claus | 150 |
| 17 | Feliz Navidad | 146 |
| 18 | All I Want for Christmas Is You | 146 |
| 19 | O Holy Night | 144 |
| 20 | I Saw Mommy Kissing Santa Claus | 143 |
| 21 | Rockin’ Around the Christmas Tree | 135 |
| 22 | Santa Claus Is Coming to Town | 126 |
| 23 | Frosty the Snowman | 125 |
| 24 | Rudolph the Red Nosed Reindeer | 121 |
| 25 | We Wish You a Merry Christmas | 118 |
Top Terms
We can build a good list of seasonal terms by finding the most frequently occurring words in song titles. Here are the top 75 or so, as a word cloud created by wordle (stop words are removed of course).
Longest Christmas song name
There are lots of very long song names in the set of Christmas songs – the longest is this Christmas medly.
A great song for testing how well your music player UI deals with unusual titles.
Conclusion
One would think that with a million Christmas tracks we’d already have more than enough Christmas music – but, it seems, we still like new Christmas music. Ariana Grande’s recently released Santa Tell Me is climbing the streaming charts (currently #44 at charts.spotify.com).
Plus, there’s seemingly no-end to the variety of Christmas Music. If White Christmas with Bing Crosby is not your style, then there’s Blue Christmas by Elvis.
And If that’s not your thing, maybe you’ll enjoy Red Christmas by Insane Clown Posse.
The Mobile Infinite Jukebox Survey results
Over the last six months or so The Infinite Jukebox had a link to a survey about features peopled would like to see in a mobile version of The Infinite Jukebox. Since then, over 10,000 people have taken the survey. Here are the results.
The survey was linked to directly from the Infinite Jukebox. The questions asked were:
Since the text in link to the survey was “Interested in a mobile version of the Infinite Jukebox? Then take this one minute survey” it is no surprise that 99% of all respondents are interested in a mobile version of the app.
The split between Android and iOS aligns with other iOS vs Android metrics out there on the webs.
As for how much people would be willing to pay, 64% would be willing to pay something for the app.
This was a bit surprising – 70% of folks want to play music from their own collection, and only 11% are interested in playing music from a streaming service like Spotify or Rdio.
The final question was an open-ended question asking about what other features would you like to see in the Infinite Jukebox. Many of the responses were about what features would like to see in the current web version, while many were about what features should be in a mobile version. Some of the more common results are here:
Common new feature suggestions
- Background playing
- Offline playling
- No Ads
- Simple tuning options
- Playlist support
- Choose song length
- Time limits per song
- Infinitise multiple songs
- Color schemes
- Volume controls
- Social features (voting on best tunings)
So, you may be wondering where is the mobile version of the Infinite Jukebox? It is coming along, all the hard coding bits are done, but it has been very much a spare time project. I do hope to release it sometime in the near future. Here’s a short clip of the app in action:
[youtube http://youtu.be/jTTYIunVK1I]Thanks to everyone who took the survey, its been quite informative.
The Ultimate Thanksgiving Playlist
On the annual drive to Thanksgiving dinner I’ve tortured my family with Alice’s Restaurant too many times over the years. Arlo Guthrie’s classic is still, in my mind, the classic Thanksgiving song, but there has to be more. So this year, I set out to expand my repertoire of Thanksgiving music – to build the ultimate Thanksgiving playlist. To do so, I looked through the top 300 or so most listened to Thanksgiving playlists on Spotify and found the top 100 songs that most frequently appear in all of these playlists, after discounting for popularity. Here are the results: The Ultimate Thanksgiving Playlist:
[spotify spotify:user:plamere:playlist:7EOwcqcYhCDsrvoutSVP9E]This is six hours of Thanksgiving music. All the classics are there, from Alice’s Restaurant to We are going to be Friends by the White Stripes. It should get you through the Thanksgiving drive, the meal, dessert and maybe even an after dinner snack.
However, if you want to synchronize your cooking and your music listening, there’s no better way then to hop on over to Time For Turkey for your basting+music needs.
And since the Christmas season starts immediately after the last piece of pumpkin pie has been consumed, lets not waste time breaking out the Christmas playlist. Here are the top 100 songs appearing across the most popular 1,000 Christmas playlists: Top Christmas Songs
[spotify spotify:user:plamere:playlist:26fUkb0ZQdg62IY497OdNL]How Students Listen
The Spotify Insights team took a deep dive into some of the listening data of college students to see if there were any differences in how students at different schools listen. We looked at a wide range of data including what artists were played, what songs were played and when, what playlists played, what genres were played and so on. We focused mostly on looking for distinctive listening patterns and behaviors at the different schools. The results were a set of infographic style visualizations that summarize the distinctive listening patterns for each school.
It was a fun study to do and really shows how much we learn about listening behavior based upon music streaming behavior. Read about the study on the Spotify Insights Blog: Top 40 Musical Universities in America:How Students Listen
Outside Lands lineup
I’m on my way to Outside Hacks – a hackathon tied in with the Outside Lands music festival. Since many hacks at the hackathon will be related to the festival it is pretty important to have a machine-readable version of the artist lineup for the festival. However, I couldn’t find any online. Since I had an hour in the airport lounge, and the airport actually has decent WiFi, I thought I would try to be a good hacker citizen and generate an easily parseable lineup.
A little python + some BeautifulSoup and a bit of Echo Nest Rosetta Data and I have created an Outside Lands lineup JSON that includes links to artist pages, plus Echo Nest, Spotify and Rdio IDs. The JSON is hosted online at:
http://static.echonest.com/OutsideLands/lineup_2014.json
Here’s the code:
| import sys | |
| from bs4 import BeautifulSoup | |
| import json | |
| import pyen | |
| en = pyen.Pyen() | |
| def get_fid(artist, idspace): | |
| if 'foreign_ids' in artist and len(artist['foreign_ids']) > 0: | |
| for fids in artist['foreign_ids']: | |
| if fids['catalog'] == idspace: | |
| return fids['foreign_id'] | |
| return None | |
| def en_artist_lookup(name): | |
| response = en.get('artist/search', name=name, | |
| bucket=['id:spotify', 'id:rdio-US']) | |
| artists = response['artists'] | |
| if len(artists) > 0: | |
| artist = artists[0] | |
| print artist | |
| enid = artist['id'] | |
| spid = get_fid(artist, 'spotify') | |
| rdio = get_fid(artist, 'rdio-US') | |
| if rdio: | |
| rdio = rdio.split(':')[2] | |
| print name, '/', artist['name'], enid, spid, rdio | |
| ids = { | |
| 'echonest':enid, | |
| 'spotify':spid, | |
| 'rdio':rdio | |
| } | |
| return ids | |
| return {} | |
| if __name__ == '__main__': | |
| lineup = [] | |
| f = open(sys.argv[1]) | |
| html_doc = f.read() | |
| soup = BeautifulSoup(html_doc) | |
| for a in soup.find_all('a', class_='band'): | |
| name = a.text.strip() | |
| ids = en_artist_lookup(name) | |
| artist = { | |
| 'artist' : name, | |
| 'link' : a['href'], | |
| 'ids' : ids | |
| } | |
| lineup.append(artist) | |
| out = open('lineup_2014.json', 'w') | |
| print >> out, json.dumps(lineup, indent=4) | |
| out.close() |
It’s about time to get on the plane. If you can think of other interesting data to add to the lineup json let me know and I’ll try to add it before the hackathon.
Most Distinctive Fourth of July Songs
Independence day is just a few days away so I spent a bit of time over the weekend digging into the data to see what are the most listened to songs on the Fourth of July. Eliot wrote a great piece about the data for the Spotify Blog: The Most Distinctive Fourth of July Songs in the 50 U.S. States. Here I dig in a bit deeper.
From a musical perspective, The Fourth of July is a very interesting holiday. It’s a big summer outdoor party – and the music reflects that. On the Fourth, people listen to patriotic songs, BBQ songs, popular songs, party songs, songs about place and history. People listen to wide range of genres, from rock and pop to folk, country, marches, and new weird america. To see what music people listen to on the Fourth I went through about 5,000 playlists that people have created that have ‘fourth of july’ in the title. I aggregated the songs across all of these playlists and created a playlist of the top 100 or songs.
As you can see, the playlist is a mix of US-centric music and summer party music – which is a pretty good reflection of what people actually listen to on the Fourth of July. But I wanted to go a bit further and see what music was particularly distinctive for the Fourth of July – as compared to any other summer outdoor party playlist. To do this, I collected the top most frequently appearing songs on the Fourth of July playlists and scored each song by calculating the ratio of the play counts that occurred on July 4, 2013 in the U.S. vs. play counts for the song during the following weeks. Songs that were played much more frequently on the Fourth than during the rest of July get a much higher score. Ranking songs by this ratio yields the list of distinctive Fourth of July Songs:
I think it is a pretty good list of the music that we listen to more on the Fourth of July than on any other day. There’s a John Philip Sousa march, Woody Guthrie’s This Land is Your Land, Katy Perry’s Firework, lots and lots of country music. Some artists appear more frequently than others – Bruce Springsteen, John Mellencamp and Lee Greenwood are Fourth of July favorites.
I’m always interested in regional differences in how we listen to music, so I looked at the listening in each state in the U.S. to see which of the core Fourth of July songs was listened to more. If you look just at the most popular Fourth of July song in each state the results are pretty boring – the most popular song in 46 out of 50 states for July 2013 was Party in the USA by Miley Cyrus – but if you look at the distinctive score (the ratio of plays on the fourth to plays during the following weeks), you get a more nuanced view of how people listen to music. Of course we represent this as a map:
So the real question is, are these playlists generated by data mining any good? As actual playlists to listen to, I don’t think so. They are too incoherent, with jumps that take you from a sappy country song, to a teen pop song, to a march that will give you iPod whiplash. Compare the playlists above to a human-curated Fourth of July Playlist:
This is a much more listenable playlist. However, I think the data mined playlists do provide an excellent starting point – a big pool of Fourth of July songs. With a genre and popularity filter these could be turned into decent, listenable playlists. Similarly, give this pool of Fourth of July songs to a team of music curators and they can build some pretty fantastic Fourth of July playlists for every type of music fan.
Fetching My Starred Items from Spotify
I wrote a quick demo that shows how to create a web app that fetches the starred items for a Spotify user via the nifty new Spotify Web API. The demo will first solicit permission from the user, and if the user grants such permissions, the app will then retrieved the starred list and show the tracks in a simple list.
The online demo is here: My Starred Items. The source is on github.
The Set Listener
Going to a show? Not totally familiar with an artist’s catalog? Give The Set Listener a try. The Set Listener is a web app that will create a Spotify playlist of an artist’s most recent show.
To use The Set Listener just type in the artist name, and hit the search button, you’ll be presented with a playlist of songs from that artist’s most recent show. Hit the ‘Save this playlist to Spotify’ button and you’ll have a Spotify playlist that you can listen to on your desktop or on your mobile phone.
The app relies on the SetList.fm API and the brand new and super spiffy Spotify Web API. Now that the Spotify Web API supports the creation and saving of playlists creating these types of apps is quite straightforward – just a few hours of coding. This was my first time using the SetList.fm API – its a super resource for setlists from concerts by thousands of artists.
Check out the app online at The Set Listener. The code is online on github.
