Archive for category data
Going Undercover
My Music Hack Day Stockholm hack is ‘Going Undercover‘. This hack uses the extensive cover song data from SecondHandSongs to construct paths between artists by following chains of cover songs. Type in the name of a couple of your favorite artists and Going Undercover will try to find a chain of cover songs that connects the two artists. The resulting playlist will likely contain familiar songs played by artists that you never heard of before. Here’s a Going Undercover playlist from Carly Rae Jepsen to Johnny Cash:
For this hack I stole a lot of code from my recent Boil the Frog hack, and good thing I could do that otherwise I would never have finished the hack in time. I spent many hours working to reconcile the Second Hand Songs data with The Echo Nest and Rdio data (Second Hand Songs is not part of Rosetta stone, so I had to write lots of code to align all the IDs up). Even with leveraging the Boil the Frog code, I had a very late night trying to get all the pieces working (and of course, the bug that I spent 2 hours banging my head on at 3AM was 5 minutes of work after a bit of sleep).
I am pretty pleased with the results of the hack. It is fun to build a path between a couple of artists and listen to a really interesting mix of music. Cover songs are great for music discovery, they give you something familiar to hold on to while listening to a new artist.
The Music We Lost in 2012
Posted by Paul in data, The Echo Nest on December 19, 2012
It’s the time of the year when music critics make their lists of artists that we passed away or bands that broke up in the last twelve months. To help them write their retrospectives I’ve put together two lists: One of well-known musicians that died during 2012, and one of well-known bands that called it quits during the year.
I made the list using the Echo Nest Artist Search API, restricting the results to artists that had a year ending in 2012.
Lots of great music left us in 2012. See the lists here: 2012 Music Memoriam
TimesOpen 2012 Hack Day Wrap-Up
I spent Saturday at the New York Times attending the TimesOpen Hack Day. It was great fun, with lots of really smart folks creating neat (and not always musical stuff). Some great shots here:
Check out the wrap up at the Times: TimesOpen 2012 Hack Day Wrap-Up
Visualizing the Structure of Pop Music
Posted by Paul in code, data, ismir, Music, music information retrieval, The Echo Nest, visualization on November 19, 2012
The Infinite Jukebox generates plots of songs in which the most similar beats are connected by arcs. I call these plots cantograms. For instance, below is a labeled cantogram for the song Rolling in the Deep by Adele. The song starts at 3:00 on the circle and proceeds clockwise, beat by beat completely around the circle. I’ve labeled the plot so you can see how it aligns with the music. There’s an intro, a first verse, a chorus, a second verse, etc. until the outro and the end of the song.
One thing that’s interesting is that most of the beat similarity connections occur between the beats in the three instances of the chorus. This certainly makes intuitive sense. The verses have different lyrics, so for the most part they won’t be too similar to each other, but the choruses have the same lyrics, the same harmony, the same instrumentation. They may even be, for all we know may even be exactly the same audio, that perfect performance, cut and pasted three times by the audio engineer to make the best sounding version of the song.
Now take a look at the cantogram for another popular song. The plot below shows the beat similarities for the song Tik Tok by Ke$ha. What strikes me the most about this plot is how similar it looks to the plot for Rolling in the Deep. It has the characteristic longer intro+first verse, some minor inter-verse similarities and the very strong similarities between the three choruses.
As we look at more plots for modern pop music we see the same pattern over and over again. In this plot for Lady Gag’s Paparazzi a cantogram we again see the same pattern.
We see it in the plot for Justin Bieber’s Baby:
Taylor Swift’s Fearless has a two verses before the first chorus, shifting it further around the circle, but other than that the pattern holds:
Now compare and contrast the pop cantograms with those from other styles of music. First up is Led Zeppelin’s Stairway to heaven. There’s no discernable repeating chorus, or global song repetition, the only real long-arc repetition occurs during the guitar solo for the last quarter of the song.
Here’s another style of music. Deadmau5’s Raise your weapon. This is electronica (and maybe some dubstep). Clearly from the cantogram we can see that is is not a traditional pop song. Very little long arc repetition, with the densest cluster being the final dubstep break.
Dave Brubeck’s Take Five has a very different pattern, with lots of short term repetition during the first half of the song, while during the second half with Joe Morello’s drum solo there’s a very different pattern.
Green Grass and High Tides has yet a different pattern – no three choruses and out here. (By the way, the final guitar solo is well worth listening to in the Infinite Jukebox. It is the guitar solo that never ends).
The progressive rock anthem Roundabout doesn’t have the Pop Pattern
Nor does Yo-Yo Ma’s performance of the Cello suite No. 1.
Looking at the pop plots one begins to understand that pop music really could be made in a factory. Each song is cut from the same mold. In fact, one of the most successful pop songs in recent years, was produced by a label with factory in its name. Looking at Rebecca Black’s Friday we can tell right away that it is a pop song:
Compare that plot to this years Youtube breakout, Thanksgiving by Nicole Westbrook, (another Ark Music Factory assembly):
The plot has all the makings of the standard pop song for the 2010s.
In the music information retrieval research community there has been quite a bit of research into algorithmically extracting song structure, and visualizations are often part of this work. If you are interested in learning more about this research, I suggest looking at some of the publications by Meinard Müller and Craig Sapp.
Of course, not every pop song will follow the pattern that I’ve shown here. Nevertheless, I find it interesting that this very simple visualization is able to show us something about the structure of the modern pop song, and how similar this structure is across many of the top pop songs.
update: since publishing this post I’ve updated the layout algorithm in the Infinite Jukebox so that songs start and end at 12 Noon and not 3PM, so the plots you see in this post are rotated 90degrees clockwise from what you would see in the jukebox.
Hear Here Version 0.3
More weekend programming on ‘Hear Here’ the mobile version of the Road Trip mixtape. I’ve added a familiarity filter so you can chose whether you want to listen to only the most recognizable artists in a region, or you want to listen to everything. Screenshots:
I’ve taken the app on a few road tests. There were a few crashes (of the app kind, not the car kind, luckily). Stability and responsiveness with unreliable networking is a fun challenge. I’m taking a 1000 mile roadtrip in a week or two, hopefully will have it 100% ready by then.
Roadtrip Mixtape
Posted by Paul in code, data, Music, The Echo Nest on June 17, 2012
Over the last few years I’ve made a number of 1,000+ mile road trips as I shuttle kids to colleges in far away places. Listening to music has always been a big part of these trips. I thought it’d be nice to be able to listen to music by local artists when driving through a particular region, so I spent a few weekends creating an app called Roadtrip Mixtape that populates a roadtrip playlist with artists that are from the region you are driving through.
To create a playlist, type your starting and ending cities for your roadtrip. The app will use Google’s directions to plan the best route between the two cities. The route will then be broken into 15 minute playlist legs. Each playlist leg is populated by 15 minutes worth of music by nearby artists.
The beginning of each leg is represented by a green ball. You can click on the ball to see what artists will be played during that leg. The app plays music via Rdio using their nifty Web Player API. If you are an Rdio subscriber you can listen to full streams, and if not you get to hear 30 second samples. One bit of interesting info that I show for a route is the ‘Avg distance’. This shows the average distance to each artist on the roadtrip. If this number is low, you are traveling through a musically dense part of the world, and if it is high, you are traveling in a sparse musical region. For instance, for a roadtrip from Boston to New York the average artist distance is 3 miles (about as low as it goes). However, if you are traveling from Omaha to Denver, the average artist distance is 81 miles.
You can also click anywhere on the map to see and listen to nearby artists. For example, if you click on Shreveport you’ll see something like this:
When you click the ‘Hear here’ button, you’ll get a playlist of the hotttest artists from Shreveport.
Listening to nearby artists is quite fun. There’s potential from some extreme sonic whiplash as you drive near a brutal death metal band and then a pop vocalist from the 1950s
The Technical Bits
To build the app I used the new artist location data from The Echo Nest. This (still in beta) feature, allows you to retrieve the location of any artist. Here’s an example API call that retrieves the artist location for Radiohead:
For this app, I collected the locations for the top 100,000 or so most popular artists in the Rdio catalog. These artists were from about 15,000 different cities. I used geopy along with the Yahoo Placefinder geocoder to find the latitude and longitude for each of these cities. For the mapping and route finding, I used version 3.9 of the Google maps API. For music playback I used the Rdio Web Playback API. With the tight integration between the Echo Nest and Rdio ID spaces it was easy to go from a geolocated Echo Nest artist to a list of Rdio track IDs for songs by that artist.
The Bad Bits
As a web app that relies on the flash-based Rdio web player, Roadtrip Mixtape is not really a mobile app. It won’t play music on an iPhone or iPad, so the best way to actually use this app on the road is probably to bring along your tethered laptop. Not the best user experience. Thus, my next weekend project will be to learn a little bit of iOS programming a make a version of this app that runs on an iPhone and an iPad. Stay tuned for the next version.
Austin climbing the Most Musical City chart with a bullet!
I’ve received quite a bit of feedback on my recent Most Musical City post, especially from folks from Austin that didn’t like Austin’s 14th place ranking. This reddit/austin comment thread was rather brutal, and this Austinist article Wait, What?! Austin Not Ranked In Top 10 Musical Cities List even closed with this appeal: any data analysts out there up for the challenge to get Austin closer to the top?
Well, John Rees, the Director of Community & Economic Development at Capital Area Council of Governments in Austin is just the data analyst that the Austinist was looking for. He re-ran the analysis but instead of using city populations he calculated the rankings based upon metropolitan statistical areas. In the May issue of Data Points Newsletter John reports on this analysis:
When data from The Echo Nest is adjusted to include metropolitan statistical area population data, the rankings of America’s most musical places changes significantly. Topping the list is Nashville, San Francisco and Los Angeles (which includes Beverly Hills). The Austin region jumps ten places from the original list to become America’s forth [sic] most musical region.
John goes on to point out some of the non-quantifiable aspects of the Austin music scene such as the diversity of music as well as the presence of events such as SXSW and Austin City Limits. John makes a strong argument that Austin is one of the country’s premier music destinations. Even the reaction of Austin’s residents to my post says a lot about Austin as a music city. People from Austin really care about music and don’t take it kindly when they are not at the top of the most musical city list. So congrats to Austin, not just for moving up the chart but also for demonstrating that Austin is the city that is most passionate about music
What is the most musical city in the United States?
Posted by Paul in code, data, fun, The Echo Nest on May 20, 2012
There are many cities in the United States that are known for their music. Cities like Nashville, Detroit, Seattle and New Orleans have played a major part in the musical history and development of this country. But what is the most musical city? Which city has spawned the most musical artists? To answer this question I used the soon-to-be-released artist location data from The Echo Nest artist API. I gathered up the top 50,000 or so U.S. artists, found their city of origin and tallied the number of artists per city. From this tally I calculated the number of artists per 1,000 inhabitants in each city. The more artists per 1000 inhabitants, the more musical the city.
Using the artists per 1k inhabitants, we can easily find the top 25 most musical cities in the United States:
| # | Artists per 1,000 inhabitants | Artists | Population | City |
|---|---|---|---|---|
| 1 | 3.14 | 111 | 35355 | Beverly Hills, CA |
| 2 | 2.26 | 1651 | 732072 | San Francisco, CA |
| 3 | 1.68 | 894 | 530852 | Nashville, TN |
| 4 | 1.64 | 936 | 571281 | Boston, MA |
| 5 | 1.54 | 651 | 422908 | Atlanta, GA |
| 6 | 1.53 | 53 | 34703 | Charlottesville, VA |
| 7 | 1.48 | 817 | 552433 | Washington, DC |
| 8 | 1.39 | 513 | 367773 | Minneapolis, MN |
| 9 | 1.37 | 740 | 540513 | Portland, OR |
| 10 | 1.32 | 51 | 38601 | Burlington, VT |
| 11 | 1.24 | 4789 | 3877129 | Los Angeles, CA |
| 12 | 1.22 | 15 | 12314 | Muscle Shoals, AL |
| 13 | 1.20 | 683 | 569369 | Seattle, WA |
| 14 | 1.11 | 755 | 678368 | Austin, TX |
| 15 | 1.05 | 75 | 71253 | Bloomington, IN |
| 16 | 1.05 | 50 | 47529 | Chapel Hill, NC |
| 17 | 1.05 | 47 | 44916 | Olympia, WA |
| 18 | 1.00 | 13 | 12945 | Princeton, NJ |
| 19 | 0.95 | 182 | 190886 | Richmond, VA |
| 20 | 0.94 | 11 | 11678 | Hendersonville, NC |
| 21 | 0.87 | 12 | 13769 | Malibu, CA |
| 22 | 0.87 | 88 | 100975 | Denton, TX |
| 23 | 0.86 | 179 | 207970 | Orlando, FL |
| 24 | 0.86 | 86 | 100158 | Berkeley, CA |
| 25 | 0.85 | 114 | 133874 | Orange, CA |
I find the results to be pretty interesting. Beverly Hills, the tiny city at the heart of the entertainment world is #1. San Francisco is the most musical of all large cities, followed closely by Nashville. Among, the most musical of small cities is Muscle Shoals AL which, according to Wikipedia, is famous for its contributions to American popular music. Less musical than expected are New Orleans (rank 36), NYC (rank 37), Detroit (rank 52).
Among the least musical cities in the U.S. are my hometown (Manchester NH), with only one artist in the top 50,000 U.S. based artist for the 100K inhabitants. The least musical large city in the U.S. is Kansas City KS, with only 7 top-50k artists for their nearly half million inhabitants. Luckily Kansas City residents can drive a few miles to Kansas city Missouri (with its 194 musicians for its 442k inhabitants) when they get tired of their own seven artists.
You can see the full list of cities with population greater than 5,000 ordered by their musicality here: The Most Musical Cities in the United States. I’d love to do this for all the cities in the world, but I can’t find a good source of city population data for world cities. If you know of one let me know.
I’m rather exited about this upcoming release of artist location data in our API. It will open the doors for a whole bunch of interesting applications, such as road trip playlisters that play music by artists local to the city you are near, contextual playlisters that will favor artists from your home town, or music exploration apps that will let you explore music from a particular region of the world. I can’t wait to see what people build with this data. Stay tuned, I’ll post when the API is released.
WordPress now supports embedding of Spotify, Rdio and Gists.
WordPress now supports direct embedding of Spotify songs:
And direct embedding of Rdio songs:
http://www.rdio.com/#/artist/deadmau5/album/4×4=12_1/track/Raise_Your_Weapon/
And best of all Gist embeddimg!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| import urllib | |
| import json | |
| from pyechonest import playlist | |
| def show_playlist(seed_artist): | |
| for s in playlist.basic(artist=seed_artist, type='artist-radio', ] | |
| buckets=['id:lyricfind-US'], results=10, limit=True): | |
| print '==================================================================' | |
| print s.title, 'by', s.artist_name | |
| print '==================================================================' | |
| show_lyrics(s) | |
| def show_lyrics(s): | |
| lfid = s.get_foreign_id('lyricfind-US').replace('lyricfind-US:song:', '') | |
| url = 'http://test.lyricfind.com/api_service/lyric.do' + \ | |
| '?apikey=your_api_key' + \ | |
| '&reqtype=default&output=json&trackid=elid:' + lfid | |
| f = urllib.urlopen(url) | |
| js = f.read() | |
| f.close() | |
| dict = json.loads(js) | |
| try: | |
| lyrics = dict['track']['lyrics'] | |
| for line in lyrics.split('\r\n'): | |
| print line | |
| except: | |
| print '(no lyrics)' | |
| if __name__ == '__main__': | |
| if len(sys.argv) > 1: | |
| show_playlist(' '.join(sys.argv[1:])) | |
| else: | |
| print 'usage: %s artist name' % (sys.argv[0],) |
Life is good! Thanks WordPress. All the details are here in this blog post.
Map of Music Styles
Posted by Paul in code, data, events, fun, tags, The Echo Nest, visualization on April 22, 2012
I spent this weekend at Rethink Music Hackers’ Weekend building a music hack called Map of Music Styles (aka MOMS). This hack presents a visualization of over 1000 music styles. You can pan and zoom through the music space just like you can with Google maps. When you see an interesting style of music you can click on it to hear some samples of music of that style.
It is fun to explore all the different neighborhoods of music styles. Here’s the Asian corner:
Here’s the Hip-Hop neighborhood:
And a mega-cluster of ambient/chill-out music:
To build the app, I collected the top 2,000 or so terms via The Echo Nest API. For each term I calculated the most similar terms based upon artist overlap (for instance, the term ‘metal’ and ‘heavy metal’ are often applied to the same artists and so can be considered similar, where as ‘metal’ and ‘new age’ are rarely applied to the same artist and are, therefore, not similar). To layout the graph I used Gephi (Its like Photoshop for graphs) and exported the graph to SVG. After that it was just a bit of Javascript, HTML, and CSS to create the web page that will let you pan and zoom. When you click on a term, I fetch audio that matches the style via the Echo Nest and 7Digital APIs.
There are a few non-styles that snuck through – the occasional band name, or mood, but they don’t hurt anything so I let them hang out with the real styles. The app works best in Chrome. There’s a bug in the Firefox version that I need to work out.
Give it a try and let me know how you like it: Map of Music Styles

























