Archive for category The Echo Nest

Wicked smart playlists

Over the past few weekends I’ve been working on a little side project called the Playlist Builder Library (or PBL for short). The Playlist Builder Library is a Python library for creating and manipulating playlists. It’s sort of like remix for playlists. With PBL you can take songs from playlists, albums, artists, genres and flexibly combined them,  rearrange them, filter them and sort them into new playlists.

For example, here’s a PBL program that creates radio station of today’s top hits but guarantees that every 4th song is either by Sia or Katy Perry:

[gist https://gist.github.com/plamere/2fa839150815f040450d]

Here’s the resulting playlist:

[spotify spotify:user:plamere:playlist:6TIeQMve7pVBLCAY8WUX3L]

That’s 5 lines a code to create a non-trivial playlist.

PBL supports all sorts of sources for tracks such as Spotify playlists, top tracks from artists,  albums, genres, the extremely flexible and powerful Echo Nest playlisting API. These sources can be manipulated in all sorts of interesting ways.  Here are a couple more examples:

You can filter all the songs in ‘Your favorite coffeehouse’ to get just the lowest energy songs:

coffee = PlaylistSource('coffeehouse', ucoffee_house) 
low_energy_coffee = AttributeRangeFilter(coffee, 'echonest.energy', max_val=.5)

You an combine your favorite playlists in a single one:

playlist_names = ['Your Favorite Coffeehouse', 'Acoustic Summer','Acoustic Covers', 'Rainy Day']
all = DeDup(Alternate([Sample(PlaylistSource(n), 10) for n in playlist_names]))

Even sophisticated tasks are really easy. For instance, imagine dad is on a roadtrip with daughter. They agree to alternate between dad’s music and daughter’s music. Dad is selfish, so he makes a playlist that alternates the longest cool jazz tracks with the shortest teen party playlists with this 3 line script:

teen_party = First(Sorter(PlaylistSource('Teen Party'), 'duration'), 10) 

jazz_classics = Last(Sorter(PlaylistSource('Jazz Classics'), 'duration'), 10) 

both = Alternate([teen_party, Reverse(jazz_classics)])

Here’s the result

[spotify spotify:user:plamere:playlist:0VKGTR6eCPe55bBjezi5z3]

Note that the average duration of Teen Party songs is much less than 3 minutes, while the average duration of Jazz Classics is above 6 minutes. Selfish dad gets to listen to his music twice as long with this jazz-skewing playlist.

There’s a whole lot of nifty things that can be done with PBL.  If you are a Python programmer with an itch for creating new playlists check it out.  The docs are online at http://pbl.readthedocs.org/ and the source is at https://github.com/plamere/pbl.

PBL is pretty modular so it is easy to add new sources and manipulators, so if you have an idea or two for changes let me know or just send me a pull request.

, ,

1 Comment

The Million Songs of Christmas

No other holiday dominates our listening like Christmas. During this season, we are exposed to a seemingly never ending playlist of Christmas music. So its no surprise that there’s a huge amount of Christmas music available on Spotify.  How much? Let’s take a look.

How much Christmas music is there?
It is actually quite hard to pinpoint the exact number of Christmas songs. First, every week during the holiday season thousands more Christmas songs are added to the set.  Second, some songs are seasonal – is Frosty The Snowman a Christmas song? Not literally, but it gets a lot of play at this time of year, even by the antipodes. Finally, there are a number of other holidays and celebrations at this time of year such as Hanukkah, Boxing DayNew Years, Kwanzaa, the Winter Solstice, and Festivus that we want to include in this category.  So when I say “Christmas Music” I’m referring to western music that is played primarily during December. There’s probably a better term to describe this music, but terms like seasonal, and holiday have their own special baggage – perhaps something like music coincident with the northern hemispheric winter solstice is the most precise description, but lets stick with Christmas music just to keep things simple. So how much Christmas music is there?  In early December 2014, crack music + data nerd Aaron Daubman  dove into the Spotify + Echo Nest music catalog and found 914,047 Christmas tracks – that’s just under a million Christmas tracks. Let’s unwrap this dataset to see what we can find.

First, some basic stats: Those 914,047 tracks represent 180,660 unique songs and were created by 63,711 unique artists – from Aaron Neville to Zuma the King. The top 20 artists with the most Christmas tracks in the Spotify catalog are all pre-Beatles artists:

Artists with the most Christmas Tracks

# Name Count
1 Bing Crosby 22382
2 Frank Sinatra 17979
3 Elvis Presley 12381
4 Nat King Cole 11613
5 Johann Sebastian Bach 8958
6 Dean Martin 8000
7 Perry Como 7529
8 Ella Fitzgerald 6428
9 Mahalia Jackson 5883
10 Mario Lanza 5377
11 Johnny Mathis 5036
12 Rosemary Clooney 4538
13 Peggy Lee 4450
14 Harry Belafonte 4054
15 The Andrews Sisters 3567
16 Louis Armstrong 3481
17 Gene Autry 3411
18 Doris Day 2985
19 Pat Boone 2767
20 Connie Francis 2500

Yes, that’s right, Bing Crosby has 22,382 different Christmas tracks (!) in the Spotify catalog. Now, a little digression on what we consider to be a unique track.  Music, especially popular music, is released in many forms. A very popular song, such as Bing Crosby’s White Christmas, may appear on a wide range of albums – from the original studio release to a plethora of Christmas Compilations and artist ‘best of’ albums. Each of these track releases may have different album art, different rights holders and regional licenses. Thus, even though the audio for White Christmas may be the same on each of the release, we consider each release as a different track.

White Christmas
Let’s take a closer look at Bing Crosby’s White Christmas. In our catalog of nearly a million Christmas tracks, 2,196 of them are Bing Crosby’s classic. I’ll say that again, just because it is a rather phenomenal fact – there are 2,196 different albums on Spotify that contain Bing’s White Christmas. It is hard to believe, so I created a web page that contains all 2,196 of the albums so you can see them all.  Click on the image below to load them all up (warning – with 2000+ album covers it’s a bit of a browser buster).

static_echonest_com_insights_christmas_whitechristmas_html 

White Christmas isn’t the only uber-track of the holidays. Here are the top 25 Christmas tracks based upon the number of times they have been released on an album:

The most released Christmas tracks

# Name Count
1 Bing Crosby – White Christmas 2196
2 Eartha Kitt – Santa Baby 1286
3 Elvis Presley – Blue Christmas 1285
4 Frank Sinatra – Jingle Bells 1121
5 Harry Belafonte – Mary’s Boy Child 904
6 Bing Crosby – Silver Bells 881
7 Nat King Cole – The Christmas Song 870
8 Frank Sinatra – The Christmas Waltz 811
9 Rosemary Clooney – Suzy Snowflake 788
10 Bobby Helms – Jingle Bell Rock 779
11 Elvis Presley – White Christmas 738
12 Judy Garland – Have Yourself a Merry Little Christmas 735
13 Frank Sinatra – White Christmas 703
14 Frank Sinatra – Christmas Dreaming 696
15 Frank Sinatra – Have Yourself a Merry Little Christmas 695
16 Elvis Presley – Silent Night 688
17 Elvis Presley – I Believe 664
18 Frank Sinatra – Santa Claus Is Coming to Town 660
19 Louis Armstrong – Zat You Santa Claus 598
20 Dean Martin – The Christmas Blues 575
21 Frank Sinatra – Mistletoe and Holly 568
22 Louis Armstrong – Cool Yule 566
23 Frank Sinatra – Silent Night 563
24 Bing Crosby – Jingle Bells 560
25 Elvis Presley – Santa Claus Is Back in Town 559

You can see all of the releases for Elvis’s Blue Christmas and Eartha Kitt’s Santa Baby  here:

static_echonest_com_insights_christmas_BlueChristmas_html

static_echonest_com_insights_christmas_SantaBaby_html

So there are lots of copies of Bing Crosby’s White Christmas and Eartha Kitt’s Santa Baby out there – but what are the most common Christmas songs overall? Which ones have been recorded the most by any artist?  The following table shows the top 25:

Most recorded songs 

# Name Recordings
1 Silent Night 19041
2 White Christmas 15928
3 Jingle Bells 14521
4 Winter Wonderland 9524
5 Joy to the World 9093
6 The First Noel 8731
7 Have Yourself a Merry Little Christmas 8511
8 O Holy Night 7925
9 Hark The Herald Angels Sing 7727
10 The Christmas Song 7673
11 Away in a Manger 7544
12 God Rest Ye Merry Gentlemen 7524
13 O Little Town of Bethlehem 7480
14 Santa Claus Is Coming To Town 6851
15 I’ll Be Home for Christmas 6844
16 O Come All Ye Faithful 6273
17 Deck The Halls 6057
18 Silver Bells 6044
19 Ave Maria 5847
20 What Child Is This? 5755
21 We Wish You A Merry Christmas 5619
22 It Came Upon A Midnight Clear 5019
23 Sleigh Ride 5004
24 Blue Christmas 4688
25 Let It Snow! Let It Snow! Let It Snow! 4598

Of course this data may be confounded by the uber-tracks like White Christmas that have thousands of versions by a single artist, so lets look at the most recorded songs by unique artists – that is, we only count Bing Crosby once for White Christmas instead of 2,196 times. When we do that the top 25 changes a bit:

Most recorded Christmas songs (Unique Artists)

# Name Recordings
1 Silent Night 7406
2 Jingle Bells 4485
3 Joy to the World 3593
4 White Christmas 3592
5 O Holy Night 3536
6 The First Noel 3181
7 What Child Is This? 3150
8 Away in a Manger 3140
9 God Rest Ye Merry Gentlemen 2871
10 Have Yourself a Merry Little Christmas 2823
11 O Come All Ye Faithful 2675
12 Hark The Herald Angels Sing 2638
13 Angels We Have Heard on High 2494
14 Winter Wonderland 2489
15 The Christmas Song 2398
16 We Wish You A Merry Christmas 2281
17 Deck The Halls 2274
18 O Little Town of Bethlehem 2197
19 We Three Kings 2048
20 Santa Claus Is Coming To Town 1837
21 It Came Upon A Midnight Clear 1768
22 Ave Maria 1705
23 Auld Lang Syne 1603
24 Silver Bells 1599
25 I’ll Be Home for Christmas 1577

The songs in green are the songs that are unique to each list.

Artists with the most number of unique songs
Bing Crosby is at the top of the Most Christmasy artists mainly because of the widespread re-issuing of White Christmas. But if we look at unique songs (i.e. White Christmas only counts once for Bing Crosby), the top Christmas artists look very different – with classical composers, Karaoke ‘artists’ and music factories topping the charts:

Artists with the most number of unique songs

1 Johann Sebastian Bach 3681
2 Bing Crosby 1462
3 The Karaoke Channel 1098
4 George Frideric Handel 903
5 A-Type Player 835
6 Frank Sinatra 816
7 ProSound Karaoke Band 762
8 Pyotr Ilyich Tchaikovsky 691
9 SBI Audio Karaoke 641
10 Mega Tracks Karaoke Band 577
11 ProSource Karaoke 539
12 Ameritz Karaoke Entertainment 508
13 Tbilisi Symphony Orchestra 506
14 Elvis Presley 472
15 Perry Como 440
16 Karaoke – Ameritz 428
17 Nat King Cole 413
18 Ameritz Karaoke Band 397
19 Merry Tune Makers 385
20 Christmas Songs 370

Current popular Christmas crooner Michael Bublé, with 31 unique Christmas songs has a way to go before he makes it on to the most-unique-songs-recorded chart.

Speaking of Karaoke – there’s lots of Christmas Karaoke – 23,472 tracks to be precise.  The top 25 Karaoke songs are the classics:

Top Karaoke Christmas Songs

# Name Count
1 White Christmas 345
2 Winter Wonderland 333
3 Silent Night 312
4 Jingle Bells 309
5 Last Christmas 258
6 Silver Bells 219
7 Blue Christmas 204
8 Santa Baby 189
9 The Christmas Song 185
10 Jingle Bell Rock 172
11 Have Yourself a Merry Little Christmas 171
12 Please Come Home for Christmas 163
13 Little Drummer Boy 163
14 Sleigh Ride 156
15 O Come All Ye Faithful 154
16 Here Comes Santa Claus 150
17 Feliz Navidad 146
18 All I Want for Christmas Is You 146
19 O Holy Night 144
20 I Saw Mommy Kissing Santa Claus 143
21 Rockin’ Around the Christmas Tree 135
22 Santa Claus Is Coming to Town 126
23 Frosty the Snowman 125
24 Rudolph the Red Nosed Reindeer 121
25 We Wish You a Merry Christmas 118

Top Terms

We can build a good list of seasonal terms by finding the most frequently occurring words in song titles. Here are the top 75 or so, as a word cloud created by wordle (stop words are removed of course).

Banners_and_Alerts_and_Wordle_Applet

Longest Christmas song name
There are lots of very long song names in the set of Christmas songs – the longest is this Christmas medly.

Andrea und Manuela – Morgen kommt der Weohnachtsmann – Medley / Morgen kommt der Weihnachtsmann/Leise rieselt der Schnee/Oh du Fröhliche/Ihr Kinderlein kommet/Süßer die Glocken nie klingen/Oh Tannenbaum/Kling Glöckchen/Stille Nacht, heilige Nacht/Alle Jahre wieder – Morgen kommt der Weihnachtsmann/Leise rieselt der Schnee/Oh du Fröhliche/Ihr Kinderlein kommet/Süßer die Glocken nie klingen/Oh Tannenbaum/Kling Glöckchen/Stille Nacht, heilige Nacht/Alle Jahre wieder

A great song for testing how well your music player UI deals with unusual titles.

Conclusion

One would think that with a million Christmas tracks we’d already have more than enough Christmas music – but, it seems, we still like new Christmas music. Ariana Grande’s recently released Santa Tell Me is climbing the streaming charts (currently #44 at charts.spotify.com).

Plus, there’s seemingly no-end to the variety of Christmas Music. If White Christmas with Bing Crosby is not your style, then there’s Blue Christmas by Elvis.

And If that’s not your thing, maybe you’ll enjoy Red Christmas by Insane Clown Posse.

Leave a comment

More on “Where’s the Drama?”

My Music Hack Day Berlin hack was “Where’s the Drama?” – a web app that automatically identifies the most dramatic moment in any song and plays it for you. I’ve been having lots of fun playing with it … and even though (or perhaps because) I know how it works, I’m often surprised at how well it does at finding the most dramatic moments.  Here are some examples:

How does it work? The app grabs the detailed audio analysis for the song from The Echo Nest.  This includes a detailed loudness map of the song. This is the data I use to find the drama.  To do so, I look for the part of the song with the largest rise in volume over the course of a 30 second window (longer songs can have a bit of a longer dramatic window). I give extra weight to crescendos that culminate in louder peaks (so if there are two crescendos that are 20dB in range but one ends at 5dB louder, it will win). Once I identify the most dynamic part of a song, I pad it a bit (so we get to hear a bit of the drop after the build).

The rest is just UI – the song gets plotted as a heavily filtered loudness curve with the dramatic passage highlighted. I plot things with Highcharts, which is a pretty nifty javascript plotting and charting library. I recommend.

 Where_is_the_Drama_

Playing the music – I wanted to use Spotify to play the music, which was a bit problematic since there currently isn’t a way to play full streams with the Spotify Web API, so I did a couple of hacky hacks that got me pretty far. First of all, I discovered that you can add a time offset to a Spotify URI like so:

        

When this URI is opened in Spotify (even when opened via a browser), Spotify will start to play the song a the 1:05 time offset.  

I still needed to be able to stop playing the track – and there’s no way to do that directly – so instead, I just open the URI:

      

which happens to be the URI for John Cage’s 4’33.  In other words, to stop playing one track, I just start playing another (that happens to be silent).  The awesome side effect of this is that I’ll be slowly turning anyone who uses “Where’s the Drama?” into experimental music listeners as the Spotify recommendation system responds to all of those John Cage ‘plays’. This should win some sort of ‘hackiest hack of the year’ award.

It was a fun hack to make, and great fun to demo. And now that I have the app, I am no longer wasting time listening to song intros and outros, I can just get to the bit of the song that matters the most. 

, , , , , ,

3 Comments

Acrostify – make Spotify playlists with embedded secret messages

For my summer vacation early-morning coding for fun project I revamped my old Acrostic Playlist Maker to work with Spotify.  The app, called Acrostify, will generate acrostic playlists with the first letter of each song in the playlist spelling out a secret message.  With the app, you can create acrostic playlists and save them to Spotify.

 

Screen Shot 2014-08-07 at 6.38.41 AM

The app was built using The Echo Nest and Spotify APIs. The source is on github.

Give it a try at Acrostify.

 

,

1 Comment

Outside Lands Recommendations

I am at Outside Hacks this weekend – A hackathon associated with the Outside Lands music festival. For this hack I thought it would be fun to try out the brand new Your Music Library endpoints in the Spotify Web API. These endpoints let you inspect and manipulate the tracks that a user has saved to their music. Since the hackathon is all about building apps for a music festival, it seems natural to create a web app that gives you festival artist recommendations based upon your Spotify saved tracks. The result is the Outside Lands Recommender:

 

Your_Outside_Lands_Festival_Recommendations

 

The Recommender works by pulling in all the saved tracks from your Spotify ‘Your Music’ collection, aggregating the artists and then using the Echo Nest Artist Similar API to find festival artists that match or are similar to those artists. The Spotify API is then used to retrieve artist images and audio samples for the recommendations where they are presented in all of their bootstrap glory.

This was a pretty straight forward app, which was good since I only had about half the normal hacking time for a weekend hackathon. I spent the other half of the time building a festival dataset for hackers to use (as well as answering lots of questions about both the Spotify and Echo Nest APIs).

It has been a very fun hackathon. It is extremely well organized, the Weebly location is fantastic, and the quality of hackers is very high. I’ve already seen some fantastic looking hacks and we are still a few hours from demo time.  Plus, this happened.

Screenshot_7_27_14__12_32_PM

Two more victims

,

Leave a comment

Set Listener update – now listen to specific shows in Spotify

Last month I released the Set Listener –  a web app that lets you create a Spotify playlist for an artist’s most recent show.   A frequent request by users has been to allow the creation of a Spotify playlist for any show, not just the most recent one. However, I didn’t want to have to implement an elaborate event search and browse feature (this was supposed to be a quick hack) especially since SetList.fm already implements this browsing. So instead I did a bit of hack to implement this feature – in addition to accepting an artist name, the Set Listener will accept the url of any setlist.fm show. If you enter a setlist.fm show URL, the Set Listener will grab the songs for that show and generate the playlist. Here’s the flow:

Go to setlist.fm and browse until you find your show of interest. Grab the URL

God_Is_an_Astronaut_Concert_Setlist_at_Melnā_Piektdiena__Riga_on_May_19__2014___setlist_fm

 

Paste the URL into the Set Listener and you should be good to go:

The_Set_Listener

,

3 Comments

The Echo Nest + Spotify Sandbox

I am wearing my International Executive Music Hacker hat today. I’m writing this blog post at 5AM somewhere over the Atlantic Ocean, on my way to the Barcelona Music Hack Day, where I’ll be representing both The Echo Nest and Spotify. I’m pretty excited about the hack event – first, because it’s in freaking Barcelona, and second, because I get to talk about what’s been going on with the Spotify and Echo Nest APIs.

The_Echo_Nest___Spotify_Developer

It has been just about 100 days since The Echo Nest and Spotify have joined forces. In that time we’ve been working hard to build the best music platform for listeners and for developers. This week we are releasing some of the very first fruits of our labors.

First up, we are releasing a new Spotify Web API.

This is a complete revamp of the Spotify Metadata API (the old version has now been deprecated). The Spotify Web API gives you access to all sorts of information about the Spotify catalog including details about artists, albums and tracks. Want to know the top tracks for an artist? There’s an API for that. Looking for high quality album art, artist images and 30 second audio previews? There are APIs for that too. Best of all, the new API includes perhaps the most requested Spotify API feature of all time With the Spotify Web API you can now create and modify playlists on behalf of authenticated users. Yes – you can now create a Spotify web app that creates playlists. (I personally requested this feature way back in 2008, here’s my begging plea for the feature in 2009).

static_echonest_com_SpotifyPopcorn_I’ve been using the beta version of this new API for a couple months now and I must say I am quite impressed. The API is fast, super easy to use, and provides all sorts of great data for building apps. In the past weeks I’ve had fun converting a number of my favorite apps to use the Spotify API. First there’s the Road Trip Mix Tape that lets you create a Spotify playlist of music by artists that are from the very towns you are driving through. Then there’s Music Popcorn, a visual interface for exploring genres. For the less visual, there’s the Genre Browser that gives you lots of details about the different music genres including playlists that help give you a gentle introduction to any of the thousands of Echo Nest genres. Next there’s Boil the Frog, an app that creates seamless playlists between any two artists. Finally there’s the 3D Music Maze, an app that lets you explore for music by wandering through a 3 dimensional music world.

Next up, a freshly minted Echo Nest + Spotify Sandbox — a new Spotify ID space.

180px-Rosetta_Stone_BW.jpegThese apps are possible because of the second thing we are releasing this week – a spiffy, shiny new Spotify Rosetta Stone catalog that ensures that the Echo Nest API has the freshest, and most up-to-date view of the Spotify universe of music. For those who might be new to The Echo Nest, Project Rosetta Stone is something we’ve been working on here at the Nest for many years. The goal of Project Rosetta Stone is to solve one of the most common problems that nearly every music app developer faces. The problem is that every music service has its own set of IDs – a music subscription service like Spotify has its own artist, album and track IDs. A lyric service has its own (and very different) IDs for those same artists, albums and tracks and a concert ticketing API has yet a third set of IDs. This is quite problematic for app developers that want to build an app that combines information from multiple services. Without a common ID system, the app developer has to resort to metadata searching and matching – which is slow and quite error prone – this results in a poor app.

Project Rosetta Stone solves this problem by providing ID mappings between as many music services as we can. With this mapping you can easily translate IDs from one ID space to another. With Rosetta Stone, if you have the Spotify track ID you can get Lyricfind and/or Musixmatch IDs making it easy to use those respective APIs to retrieve lyrics for that song. You can easily map the Spotify artist ID to a Songkick or Eventful ID to get ticket and touring information from those APIs. And of course you can use the Spotify track ID to get detailed Echo Nest information about the song such as its tempo, energy, danceability, along with detailed Echo Nest artist data such as latest artist news, blog posts and similar artists.

We have had Spotify IDs in Rosetta Stone for many years, but this particular mapping has in the past been problematic for us. Spotify has a huge catalog and keeping the mapping fresh and up to date between Spotify and The Echo Nest has always been a big challenge. There’s a huge back catalog with millions of tracks to deal with plus thousands of new tracks are being added every week. The result was that there was always a bit of a lag between when updates to the Spotify catalog were reflected in the Rosetta Stone mapping. This meant that if you built a Rosetta Stone-based app you could find that The Echo Nest wouldn’t always know about a Spotify track, especially if a track was very new. The result would be a less-than-perfect app.

This week we are releasing a new Spotify ID space. Our engineers have been working hard over the last 100 days to set up all sorts of infrastructure and plumbing to ensure that we have the most up-to-date view of the Spotify catalog. No more lag between when a new track appears in Spotify and when you can get Echo Nest data. Plus, all of our APIs that take IDs as inputs will now also take Spotify IDs as input as well. If you have a Spotify artist ID you can use it with any Echo Nest artist API method. Likewise, if you have a Spotify track ID you can use it with any Echo Nest song or track API method that takes a track ID as input. This makes it **really** easy for developers to use The Echo Nest and Spotify Apps together. For example, here’s an API call that returns detailed audio properties for a Spotify track given its ID.

http://developer.echonest.com/api/v4/track/profile?api_key=FILDTEOIK2HBORODV&format=json&id=spotify:track:3L7BcXHCG8uT92viO6Tikl&bucket=audio_summary

I’ve been having much fun using The Echo Nest API with the brand new Spotify API. I’ve already written some code that you can use. First, I wrote a Python library for Spotify called Spotipy. It’s makes it easy to write Python programs that use the new Spotify Web API, and it works well with my Echo Nest Python library called Pyen. Here’s an example of using the two libraries together:


import pyen
import spotipy
import sys
'''
get a set of images for artists that are similar to
the given seed artist
'''
en = pyen.Pyen()
sp = spotipy.Spotify()
spids = []
response = en.get('artist/similar', name='weezer', bucket='id:spotify', limit=True)
for artist in response['artists']:
spids.append(artist['foreign_ids'][0]['foreign_id'])
for artist in sp.artists(spids)['artists']:
print artist['images'][0]['url'], artist['name']

I’ve also put together a number of Javascript example apps that use both APIs. These are simple apps intended to help new developers (or at least new to music apps) use the APIs together to do common things like create chillout playlists, browse through the web of similar artists, and more.

So yes, I’m pretty jazzed about this trip to Barcelona. I get to create a music hack, I get to spend a few days with some of the best music hackers in the world (The Barcelona Music Hack Day, as part of the Sonar Festival tends to attract the top music hackers). I get to spend a few days on the Mediterranean in one of the most beautiful cities in the world. Best of all, I get to talk about the new Spotify and Echo Nest developer platform and help music hackers build cool stuff on top of the newly combined platform.

I’ve put together a page that talks in detail about the new Spotify / Echo Nest platform. It has links to all of the API docs, libraries, examples, github repos, demos and details on how you can use The Echo Nest / Spotify Platform. Check it out here:

The_Echo_Nest___Spotify_Developer

 

http://static.echonest.com/enspex

Keep an eye on this space for I’ll be updating it as we continue to integrate our developer APIs. There’s lots more coming, so stay tuned!

5 Comments

10 Years of blogging

Ten years ago, May 3, 2004 I typed my very first blog post. At the time I was a researcher at Sun Microsystem’s Lab in the speech group working on speech synthesis and speech recognition systems, and so that was what I blogged about. Over the next year, my blog,  first called “Duke Speaks!” and then called “Duke Listens!”, (Duke is the name of the Java mascot) slowly morphed into a blog that was focused on music technology, and in particular, music information retrieval, music recommendation, music playlisting and discovery.  Five years later, when I left Sun to join The Echo Nest, I relaunched the blog as Music Machinery where I’ve been writing ever since.

I’ve written lots of blog posts. Some of my favorite from the Duke Listens! days are:

Some of my more popular posts from the Music Machinery days are:

I’m really looking forward to the next decade of blogging. Now that I work  for Spotify, there’s a seemingly unlimited supply of music data that will be prime blogging material. It will be great fun.

Thanks to you my reader, for stopping by, and  for all of your great comments, and feedback over the years.

2 Comments

The Skip

skipAt the time when I was coming of age musically, when we listened to music on LPs, the listening experience was very different than it is today. For one, if you didn’t like the currently playing song you had to get out of your chair, walk over to the turntable, carefully pick up the tone arm and advance the needle to the next track.  That was a lot of work to avoid three minutes of bad music. You really had to really dislike a song to make skipping it worth the effort. Today, with our fancy iPhones and our digital streaming music subscription services, skipping a song couldn’t be easier. Just tap a button and you are on to the next song.  The skip button is now a big part of the overall listening experience. Don’t like a song? Skip it. Never heard a song? Skip it. Just heard a song? Skip it. The Skip even plays a role in how we we pay for music. For most music subscription services if you want the freedom to skip a song whenever you want, you’ll need to be a premium subscriber, otherwise you’ll be limited to a half-dozen or so skips per hour.

I am interested in how people are using the skip button when listening to music so I spent a bit of time taking a closer look at skip data. This and the next blog post or two will be all about the skipping behavior of music listeners. We’ll take a look at how often people skip, whether different listener demographics have different skipping behavior, what artists and genres are most and least likely to trigger skips and more!

The Data
This is my first deep dive into Spotify data. The Spotify team has built up a fantastic big data infrastructure making it easy to extract insights from the billions and billions of music plays. For this study I’ve processed several billions of plays from many million unique listeners from all around the world.

What is a skip?
For this study, I define a skip as any time the listener abandons a song before the song finishes. It could be because the listener explicitly presses the skip button, or it could be that they searched for and started another song before the current song finished, or they clicked on a different song in the playlist. For whatever reason, if the listener doesn’t make it to the end of the song, I call it a skip.

How often do people skip?
The first and most basic question to answer is:  How often do people skip?. Given that skipping is so easy how big of a part does skipping play in our listening. The answer: A lot!  

Here are the numbers.  First, lets look at how often a song is skipped within the first five seconds of play.  I call these quick skips. The likelihood that a song will be skipped within the first five seconds is an astounding 24.14%. Nearly one quarter of all song plays are abandoned in the first 5 seconds. The likelihood that a song will be skipped within the first thirty seconds rises to 35.05%. The chance that a song is skipped before it ends is a whopping 48.6%. Yes, the odds are only slightly better than 50/50 that a song will be played all the way to the end.

Skipped in Likelihood of skip
First 5 seconds 24.14 %
First 10 seconds 28.97 %
First 30 seconds 35.05 %
Before song finishes 48.6 %

The following plot shows the average skipping behavior for millions of listeners and billions of plays. The plot shows the rather steep drop off in listeners in the early part of a song when most listeners are deciding whether or not to skip the song.  Then there’s a slow but steady decline in listeners until we reach the end of the song where only about 50% of the listeners remain.

all_songs

The next plot shows the average skipping behavior within in the first 60 seconds of a song. It shows that most of the song skips happen within the first 20 seconds or so of the song, and after that there’s a relatively small but steady skipping rate.

all_songs_detail

We can also calculate an overall skip rate per listener – that is, the average number of times a listener skips a song per hour.

Average listener/skips per hour:  14.65

On average a listener is skipping a song once every four minutes. That’s a whole lot of skipping.

Who is doing all that skipping?

Do different types of listeners skip music at different rates? Lets take a look.

By Gender

Skipping rate of male listeners:     44.75%
Skipping rate of female listeners:  45.23%

There seems to be little difference as to how often men and women skip.

By Platform:

Desktop skipping rate:   40.1%
Mobile skipping rate:      51.1%

When we are at our desktops, we tend to settle into longer listening sessions and skip less, while when we are mobile we spend much more time interacting with our music.

By age:

Skipping_behavior_by_age

This plot shows the skipping rate as a function of the age of the listener.  It shows that young teenagers have the highest skipping rate – well above 50%, but as the listener gets older their skipping rate drops rather dramatically, to reach the skipping nadir of about 35%.  Interestingly, the skipping rate rises again for people in their late 40s and early 50s.  I have a couple of theories about why this might be.  The first theory is that the skipping rate is a indication of how much free time a person has time. Teenagers skip more because they have more time to devote to editing their music stream, whereas thirty-somethings, with their little kids and demanding jobs, have no time to pay attention to  their music players.  The second theory, suggested by Spotify über-analyst Chris Tynan, is that the late-forties skipping resurgence is caused by teenagers that use their parent’s account.

When do people skip the most?

The following plot shows the skipping behavior over a 24 hour period.  To create the plot, I analyzed the listening behavior for UK residents (which are conveniently confined to a single timezone) over several weeks.

Skipping_behavior_by_hour_of_the_day

The plot shows that the skipping rate is lowest when people are paying less attention to music – like when they are asleep, or at work. Skipping behavior peaks in the morning hour as people start they day and start to head into work and again at the end of the day when they are at home or out socializing with their friends.  The plot shows the time of day when people tend to have the most attention to devote to hand-curating their music stream. When people are sleeping or working, their skip rate goes down.

In the next plot, below, the skipping rate is overlaid with normalized song plays.  It is interesting to see that the highest skipping rates do not coincide with the peak music playing times of the day, but instead is aligned with the times of day when rate of change in plays is the most.

Skipping_behavior_compared_to_song_plays_by_hour_of_the_day

 

Skipping behavior by Day of the Week

The following plot shows the average skipping rate per day of the week.  The skipping rate is higher on weekends, showing, once again, that when people have more spare time, they are more apt to curate their listening sessions by skipping tracks.

Skipping_behavior_by_day_of_the_week_and_2__ssh

Take away
The Skip really has changed how we listened to music.  It plays a significant role in how we interacts with our music stream. When we are more engaged with our music – we skip more, and when music is in the background such as when we are working or relaxing, we skip less. When we have more free time, such as when we are young, or on the weekends, or home after a day of work, we skip more. That’s when we have more time to pay attention to our music. The big surprise for me is how often we skip.  On average, we skip nearly every other song that we play.

Skipping has become an important part of how we listen to music.  It is no surprise then, that ‘unlimited skipping’ is a feature used to entice people to upgrade to a premium paid account. And it may be one of the reasons why people would switch from a service that doesn’t offer unlimited skips even on their premium service to one that does.

Coming soon: Look for my next post that will look at which genres, songs and artists get skipped the most and the least.

59 Comments

Echo Nest Radio on Spotify

Spotify_-_The_Cult_–_Wild_Flower

I work for Spotify now – so for my Sunday morning programming project I thought I’d write a simple Spotify App that uses The Echo Nest API to create playlists based upon a seed song. I’ve done this before, but the last time was a few years ago and the Spotify Apps API has changed quite a bit since then, so I thought I’d use this as an opportunity to freshen my understanding of the Spotify API as well as to demonstrate how to write a Spotify App that uses The Echo Nest API.

I created an Echo Nest Radio app – it is a very simple app – it looks at what song you are currently playing and will generate an Echo Nest playlist based upon that seed song. The code is pretty straightforward. It grabs the Now Playing track from Spotify, gets the track’s ID and uses that as a seed for The Echo Nest song-radio static playlist API. This call returns Spotify track IDs (thanks to our Rosetta Stone ID mapping layer) that are then tossed into a temporary playlist, which is used to build a List view which is then displayed in the app. All told it is just over 100 lines of Javascript.

It did take me a bit of time to get the hang of the newer Spotify Apps API. It has changed quite a bit since I last used it and many of the examples that I relied on in the past, like Peter Watt’s excellent Kitchen Sink app, use an older version of the API. The new version has some significant changes including a nifty new module system along with a new approach to managing long-running queries that relies on promises. Once I got the hang of it, I decided that I like the new version – it makes for cleaner code and a much more efficient app since much less data needs to be moved around.

The app is on github – to use it you need to sign up for a developer account on Spotify and follow the rest of the Getting Started instructions (this means if you are not a developer, you’ll probably not be able to use the app).

The Spotify Apps API makes it super easy to be able to create apps that run inside Spotify. Its a very familiar programming environment for anyone who has done web programming. You can use all of your favorite libraries from jQuery to Lo-Dash to create really compelling apps that sit on top of the millions and millions of tracks in the Spotify catalog. However, unlike a web app where anyone can publish their app on the web, to publish a Spotify App you have to submit your app to the Spotify App approval process and only apps that Spotify approves are published. Spotify sets a high bar for what ultimately gets approved – which keeps the quality of the apps high, but also means that hacks and experiments built on the Spotify Apps platform will likely never be approved for release to the general public.  It’s a difficult balancing act for Spotify. They’ve built the ultimate music hacking platform with this API, but if they open the doors to every music hack, they will ultimately dilute the listening experience of the user – like so other many App stores that are flooded with garbage apps,  if they publish every app and hack then Spotify listeners would be inundated with the musical equivalent of flashlight and fart apps.  With the approval process, Spotify essentially says ‘the listener comes first’ which is the right choice.   Still, as a music hacker I do wish it was easier to publish rich music apps built on the Spotify platform. Luckily Spotify is committed to building an active and vibrant developer ecosystem so I don’t expect they we will be standing still.

Update 3/24/14: – I’ve added the ability to save these playlists back to Spotify, so you can take the Echo Nest radio playlists with you.

Second update 3/24/14 – note that Spotify’s recent announcement that they are closing app submissions means that you won’t be able to submit apps for publishing anymore, but you should be able to still create your own.

1 Comment