Archive for category code
Outside Lands lineup
I’m on my way to Outside Hacks – a hackathon tied in with the Outside Lands music festival. Since many hacks at the hackathon will be related to the festival it is pretty important to have a machine-readable version of the artist lineup for the festival. However, I couldn’t find any online. Since I had an hour in the airport lounge, and the airport actually has decent WiFi, I thought I would try to be a good hacker citizen and generate an easily parseable lineup.
A little python + some BeautifulSoup and a bit of Echo Nest Rosetta Data and I have created an Outside Lands lineup JSON that includes links to artist pages, plus Echo Nest, Spotify and Rdio IDs. The JSON is hosted online at:
http://static.echonest.com/OutsideLands/lineup_2014.json
Here’s the code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| from bs4 import BeautifulSoup | |
| import json | |
| import pyen | |
| en = pyen.Pyen() | |
| def get_fid(artist, idspace): | |
| if 'foreign_ids' in artist and len(artist['foreign_ids']) > 0: | |
| for fids in artist['foreign_ids']: | |
| if fids['catalog'] == idspace: | |
| return fids['foreign_id'] | |
| return None | |
| def en_artist_lookup(name): | |
| response = en.get('artist/search', name=name, | |
| bucket=['id:spotify', 'id:rdio-US']) | |
| artists = response['artists'] | |
| if len(artists) > 0: | |
| artist = artists[0] | |
| print artist | |
| enid = artist['id'] | |
| spid = get_fid(artist, 'spotify') | |
| rdio = get_fid(artist, 'rdio-US') | |
| if rdio: | |
| rdio = rdio.split(':')[2] | |
| print name, '/', artist['name'], enid, spid, rdio | |
| ids = { | |
| 'echonest':enid, | |
| 'spotify':spid, | |
| 'rdio':rdio | |
| } | |
| return ids | |
| return {} | |
| if __name__ == '__main__': | |
| lineup = [] | |
| f = open(sys.argv[1]) | |
| html_doc = f.read() | |
| soup = BeautifulSoup(html_doc) | |
| for a in soup.find_all('a', class_='band'): | |
| name = a.text.strip() | |
| ids = en_artist_lookup(name) | |
| artist = { | |
| 'artist' : name, | |
| 'link' : a['href'], | |
| 'ids' : ids | |
| } | |
| lineup.append(artist) | |
| out = open('lineup_2014.json', 'w') | |
| print >> out, json.dumps(lineup, indent=4) | |
| out.close() |
It’s about time to get on the plane. If you can think of other interesting data to add to the lineup json let me know and I’ll try to add it before the hackathon.
Fetching my saved tracks from Spotify
The @SpotifyPlatform team just pushed out an update to the Spotify Web API that lets a developer retrieve and manipulate the tracks that a user has saved in ‘Your Music’. To show off this new functionality I wrote a quick demo that shows how to fetch the saved tracks for a Spotify user via the nifty new API. The demo will first solicit permission from the user, and if the user grants such permissions, the app will then retrieved the saved tracks and show them as a simple list.
The demo is on line here: My Saved Tracks and the source is on github.
Set Listener update – now listen to specific shows in Spotify
Posted by Paul in code, Spotify, The Echo Nest on July 21, 2014
Last month I released the Set Listener – a web app that lets you create a Spotify playlist for an artist’s most recent show. A frequent request by users has been to allow the creation of a Spotify playlist for any show, not just the most recent one. However, I didn’t want to have to implement an elaborate event search and browse feature (this was supposed to be a quick hack) especially since SetList.fm already implements this browsing. So instead I did a bit of hack to implement this feature – in addition to accepting an artist name, the Set Listener will accept the url of any setlist.fm show. If you enter a setlist.fm show URL, the Set Listener will grab the songs for that show and generate the playlist. Here’s the flow:
Go to setlist.fm and browse until you find your show of interest. Grab the URL
Paste the URL into the Set Listener and you should be good to go:
Most Distinctive Fourth of July Songs
Independence day is just a few days away so I spent a bit of time over the weekend digging into the data to see what are the most listened to songs on the Fourth of July. Eliot wrote a great piece about the data for the Spotify Blog: The Most Distinctive Fourth of July Songs in the 50 U.S. States. Here I dig in a bit deeper.
From a musical perspective, The Fourth of July is a very interesting holiday. It’s a big summer outdoor party – and the music reflects that. On the Fourth, people listen to patriotic songs, BBQ songs, popular songs, party songs, songs about place and history. People listen to wide range of genres, from rock and pop to folk, country, marches, and new weird america. To see what music people listen to on the Fourth I went through about 5,000 playlists that people have created that have ‘fourth of july’ in the title. I aggregated the songs across all of these playlists and created a playlist of the top 100 or songs.
As you can see, the playlist is a mix of US-centric music and summer party music – which is a pretty good reflection of what people actually listen to on the Fourth of July. But I wanted to go a bit further and see what music was particularly distinctive for the Fourth of July – as compared to any other summer outdoor party playlist. To do this, I collected the top most frequently appearing songs on the Fourth of July playlists and scored each song by calculating the ratio of the play counts that occurred on July 4, 2013 in the U.S. vs. play counts for the song during the following weeks. Songs that were played much more frequently on the Fourth than during the rest of July get a much higher score. Ranking songs by this ratio yields the list of distinctive Fourth of July Songs:
I think it is a pretty good list of the music that we listen to more on the Fourth of July than on any other day. There’s a John Philip Sousa march, Woody Guthrie’s This Land is Your Land, Katy Perry’s Firework, lots and lots of country music. Some artists appear more frequently than others – Bruce Springsteen, John Mellencamp and Lee Greenwood are Fourth of July favorites.
I’m always interested in regional differences in how we listen to music, so I looked at the listening in each state in the U.S. to see which of the core Fourth of July songs was listened to more. If you look just at the most popular Fourth of July song in each state the results are pretty boring – the most popular song in 46 out of 50 states for July 2013 was Party in the USA by Miley Cyrus – but if you look at the distinctive score (the ratio of plays on the fourth to plays during the following weeks), you get a more nuanced view of how people listen to music. Of course we represent this as a map:
So the real question is, are these playlists generated by data mining any good? As actual playlists to listen to, I don’t think so. They are too incoherent, with jumps that take you from a sappy country song, to a teen pop song, to a march that will give you iPod whiplash. Compare the playlists above to a human-curated Fourth of July Playlist:
This is a much more listenable playlist. However, I think the data mined playlists do provide an excellent starting point – a big pool of Fourth of July songs. With a genre and popularity filter these could be turned into decent, listenable playlists. Similarly, give this pool of Fourth of July songs to a team of music curators and they can build some pretty fantastic Fourth of July playlists for every type of music fan.
Fetching My Starred Items from Spotify
I wrote a quick demo that shows how to create a web app that fetches the starred items for a Spotify user via the nifty new Spotify Web API. The demo will first solicit permission from the user, and if the user grants such permissions, the app will then retrieved the starred list and show the tracks in a simple list.
The online demo is here: My Starred Items. The source is on github.
The Set Listener
Going to a show? Not totally familiar with an artist’s catalog? Give The Set Listener a try. The Set Listener is a web app that will create a Spotify playlist of an artist’s most recent show.
To use The Set Listener just type in the artist name, and hit the search button, you’ll be presented with a playlist of songs from that artist’s most recent show. Hit the ‘Save this playlist to Spotify’ button and you’ll have a Spotify playlist that you can listen to on your desktop or on your mobile phone.
The app relies on the SetList.fm API and the brand new and super spiffy Spotify Web API. Now that the Spotify Web API supports the creation and saving of playlists creating these types of apps is quite straightforward – just a few hours of coding. This was my first time using the SetList.fm API – its a super resource for setlists from concerts by thousands of artists.
Check out the app online at The Set Listener. The code is online on github.
My new superpower – creating Spotify playlists from a web app
The new Spotify Web API allows the developer to create and add tracks to a playlist on behalf of a listener. This is a pretty powerful feature, opening the door for a whole range of apps. For instance, this weekend, I added the ability to save a Roadtrip Mixtape playlist, so you can now actually take your mixtapes on the road. The code is on github if you are interested in seeing how it is done.
The Echo Nest + Spotify Sandbox
I am wearing my International Executive Music Hacker hat today. I’m writing this blog post at 5AM somewhere over the Atlantic Ocean, on my way to the Barcelona Music Hack Day, where I’ll be representing both The Echo Nest and Spotify. I’m pretty excited about the hack event – first, because it’s in freaking Barcelona, and second, because I get to talk about what’s been going on with the Spotify and Echo Nest APIs.
It has been just about 100 days since The Echo Nest and Spotify have joined forces. In that time we’ve been working hard to build the best music platform for listeners and for developers. This week we are releasing some of the very first fruits of our labors.
First up, we are releasing a new Spotify Web API.
This is a complete revamp of the Spotify Metadata API (the old version has now been deprecated). The Spotify Web API gives you access to all sorts of information about the Spotify catalog including details about artists, albums and tracks. Want to know the top tracks for an artist? There’s an API for that. Looking for high quality album art, artist images and 30 second audio previews? There are APIs for that too. Best of all, the new API includes perhaps the most requested Spotify API feature of all time With the Spotify Web API you can now create and modify playlists on behalf of authenticated users. Yes – you can now create a Spotify web app that creates playlists. (I personally requested this feature way back in 2008, here’s my begging plea for the feature in 2009).
I’ve been using the beta version of this new API for a couple months now and I must say I am quite impressed. The API is fast, super easy to use, and provides all sorts of great data for building apps. In the past weeks I’ve had fun converting a number of my favorite apps to use the Spotify API. First there’s the Road Trip Mix Tape that lets you create a Spotify playlist of music by artists that are from the very towns you are driving through. Then there’s Music Popcorn, a visual interface for exploring genres. For the less visual, there’s the Genre Browser that gives you lots of details about the different music genres including playlists that help give you a gentle introduction to any of the thousands of Echo Nest genres. Next there’s Boil the Frog, an app that creates seamless playlists between any two artists. Finally there’s the 3D Music Maze, an app that lets you explore for music by wandering through a 3 dimensional music world.
Next up, a freshly minted Echo Nest + Spotify Sandbox — a new Spotify ID space.
These apps are possible because of the second thing we are releasing this week – a spiffy, shiny new Spotify Rosetta Stone catalog that ensures that the Echo Nest API has the freshest, and most up-to-date view of the Spotify universe of music. For those who might be new to The Echo Nest, Project Rosetta Stone is something we’ve been working on here at the Nest for many years. The goal of Project Rosetta Stone is to solve one of the most common problems that nearly every music app developer faces. The problem is that every music service has its own set of IDs – a music subscription service like Spotify has its own artist, album and track IDs. A lyric service has its own (and very different) IDs for those same artists, albums and tracks and a concert ticketing API has yet a third set of IDs. This is quite problematic for app developers that want to build an app that combines information from multiple services. Without a common ID system, the app developer has to resort to metadata searching and matching – which is slow and quite error prone – this results in a poor app.
Project Rosetta Stone solves this problem by providing ID mappings between as many music services as we can. With this mapping you can easily translate IDs from one ID space to another. With Rosetta Stone, if you have the Spotify track ID you can get Lyricfind and/or Musixmatch IDs making it easy to use those respective APIs to retrieve lyrics for that song. You can easily map the Spotify artist ID to a Songkick or Eventful ID to get ticket and touring information from those APIs. And of course you can use the Spotify track ID to get detailed Echo Nest information about the song such as its tempo, energy, danceability, along with detailed Echo Nest artist data such as latest artist news, blog posts and similar artists.
We have had Spotify IDs in Rosetta Stone for many years, but this particular mapping has in the past been problematic for us. Spotify has a huge catalog and keeping the mapping fresh and up to date between Spotify and The Echo Nest has always been a big challenge. There’s a huge back catalog with millions of tracks to deal with plus thousands of new tracks are being added every week. The result was that there was always a bit of a lag between when updates to the Spotify catalog were reflected in the Rosetta Stone mapping. This meant that if you built a Rosetta Stone-based app you could find that The Echo Nest wouldn’t always know about a Spotify track, especially if a track was very new. The result would be a less-than-perfect app.
This week we are releasing a new Spotify ID space. Our engineers have been working hard over the last 100 days to set up all sorts of infrastructure and plumbing to ensure that we have the most up-to-date view of the Spotify catalog. No more lag between when a new track appears in Spotify and when you can get Echo Nest data. Plus, all of our APIs that take IDs as inputs will now also take Spotify IDs as input as well. If you have a Spotify artist ID you can use it with any Echo Nest artist API method. Likewise, if you have a Spotify track ID you can use it with any Echo Nest song or track API method that takes a track ID as input. This makes it **really** easy for developers to use The Echo Nest and Spotify Apps together. For example, here’s an API call that returns detailed audio properties for a Spotify track given its ID.
I’ve been having much fun using The Echo Nest API with the brand new Spotify API. I’ve already written some code that you can use. First, I wrote a Python library for Spotify called Spotipy. It’s makes it easy to write Python programs that use the new Spotify Web API, and it works well with my Echo Nest Python library called Pyen. Here’s an example of using the two libraries together:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pyen | |
| import spotipy | |
| import sys | |
| ''' | |
| get a set of images for artists that are similar to | |
| the given seed artist | |
| ''' | |
| en = pyen.Pyen() | |
| sp = spotipy.Spotify() | |
| spids = [] | |
| response = en.get('artist/similar', name='weezer', bucket='id:spotify', limit=True) | |
| for artist in response['artists']: | |
| spids.append(artist['foreign_ids'][0]['foreign_id']) | |
| for artist in sp.artists(spids)['artists']: | |
| print artist['images'][0]['url'], artist['name'] |
I’ve also put together a number of Javascript example apps that use both APIs. These are simple apps intended to help new developers (or at least new to music apps) use the APIs together to do common things like create chillout playlists, browse through the web of similar artists, and more.
So yes, I’m pretty jazzed about this trip to Barcelona. I get to create a music hack, I get to spend a few days with some of the best music hackers in the world (The Barcelona Music Hack Day, as part of the Sonar Festival tends to attract the top music hackers). I get to spend a few days on the Mediterranean in one of the most beautiful cities in the world. Best of all, I get to talk about the new Spotify and Echo Nest developer platform and help music hackers build cool stuff on top of the newly combined platform.
I’ve put together a page that talks in detail about the new Spotify / Echo Nest platform. It has links to all of the API docs, libraries, examples, github repos, demos and details on how you can use The Echo Nest / Spotify Platform. Check it out here:
http://static.echonest.com/enspex
Keep an eye on this space for I’ll be updating it as we continue to integrate our developer APIs. There’s lots more coming, so stay tuned!
Minimizing my Karaoke pain
Rumor has it from some of the Echo Nest gang that went to Stockholm last week for new employee orientation that there is some sort of mandatory Karaoke requirement. Now for some, I’m sure this is great fun, but for others, like myself, not so much. I thought it would be best to prepare for my own mandatory Karaoke by finding some very short songs in order to minimize my time on stage. To do this I went through a database of the top Billboard songs of the last 60 years to find the shortest songs. Here are some of the top shortest popular songs of the last 60 years:
| Length(Seconds) | Artist/Title | Date |
| 76 | Anna Kendrick Cups | 2013-01-14 |
| 78 | Zac Efron What I’ve Been Looking For (Reprise) | 2006-02-13 |
| 83 | Buchanan & Goodman Santa And The Satellite (Part I) | 1957-12-25 |
| 92 | Audrey Dear Elvis (Page 1) | 1956-09-24 |
| 96 | Fats Domino Whole Lotta Loving | 1958-11-19 |
| 98 | Glee Cast Isn’t She Lovely | 2011-05-30 |
| 99 | Maurice Williams & The Zodiacs Stay | 1960-10-05 |
| 101 | Swinging Blue Jeans, The Hippy Hippy Shake | 1964-03-09 |
| 103 | Peter, Paul & Mary Settle Down (Goin’ Down That Highway) | 1963-01-21 |
| 105 | Four Tops Ain’t That Love | 1965-08-02 |
| 105 | Fats Domino Shu Rah | 1961-03-22 |
| 105 | Chuck Berry Let It Rock | 1960-02-03 |
| 107 | Lucas Gabreel & Ashley Tisdale Bop To The Top | 2006-02-13 |
| 107 | Beach Boys, The Little Deuce Coupe | 1963-08-19 |
| 107 | Clyde McPhatter Lover Please | 1962-03-05 |
| 108 | Ventures, The Hawaii Five-O | 1969-03-10 |
| 110 | Glee Cast Sing! | 2010-11-01 |
| 110 | Glee Cast It’s My Life / Confessions Part II | 2009-10-26 |
| 110 | Ricky Nelson If You Can’t Rock Me | 1963-04-22 |
So it looks like my minimum possible karaoke pain will be 76 seconds if I go with Anna Kendrick’s Cups. Certainly better than Gun’s in Roses November Rain at 8:57 seconds or Don Mclean’s American Pie at 6:49. But better yet, I can go with Hawaii Five-O . That song is not only short, but has no vocals. With that song I’m sure to be pitch perfect!
Spotify iOS token exchange service in python
On the very same day that Spotify announced its acquisition of The Echo Nest they released a brand new Spotify iOS SDK. Trying this new SDK out has been high on my priority list, and finally after a few crazy weeks I’ve had a bit of time to take it for a test drive. I walked through the beginner’s tutorial and was up and running with an iOS app running in the simulator in about 30 minutes. Easy Peazy! The bit that took the longest was setting up the token exchange service. This is a service that you need to run on your own server as part of the authentication process. The tutorial provides such a sample service written in ruby, however I’m not a ruby programmer so I had to go through all the gyrations of installing ruby, figuring out how to install gems and getting the required gems installed. Once I had everything installed it worked fine and I was able to get the tutorial running. However, I figure that I’ll be working with the iOS SDK a great deal in my future, and I’d rather not have to deal with a ruby server every time I create a new app, and so for my Sunday morning programming project I’ve re-written the ruby swap service in python. The service is on github here: spotify_token_swap
If you are going to be using the new Spotify iOS SDK to create apps and you’d rather deal with python than ruby, then you might find it useful.
Music Machinery 