Music Machinery

I spent this weekend at Music Hack Day San Franciso. Music Hack Day is an event where people who are passionate about music and technology get together to build cool, music-related stuff. This San Francisco event was the 30th Music Hack Day with around 200 hackers in attendance. Many cool hacks were built. Check out this Projects page on Hacker League for the full list.

My weekend hack is called The Bonhamizer. It takes just about any song and re-renders it as if John Bonham was playing the drums on the track. For many tracks it works really well (and others not so much). I thought I’d give a bit more detail on how the hack works.

Data Sources

For starters, I needed some good clean recordings of John Bonham playing the drums. Luckily, there’s
set of 23 drum out takes from the “In through the out door” recording session. For instance, there’s
this awesome recording of the drums for “All of my Love”. Not only do you get the sound of Bonhmam
pounding the crap out of the drums, you can hear him grunting and groaning. Super stuff. This particular
recording would become the ‘Hammer of the Gods’ in the Bonhamizer.

[audio http://static.echonest.com/audio2/1360068024326/BonhamFile14.mp3]

From this set of audio, I picked four really solid drum patterns for use in the Bonhamizer.

Of course you can’t just play the drum recording and any arbitrary song recording at the same time and expect the drums to align with the music. To make the drums play well with any song a number of things need to be done (1) Align the tempos of the recordings, (2) Align the beats of the recordings, (3) Play the drums at the right times to enhance the impact of the drums.

To guide this process I used the Echo Nest Analyzer to analyze the song being Bonhamized as well as the Bonham beats. The analyzer provides a detailed map for the beat structure and timing for the song and the beats. In particular, the analyzer provides a detailed map of where each beat starts and stops, the loudness of the audio at each beat, and a measure of the confidence of the beat (which can be interpreted as how likely the logical beat represents a real physical beat in the song).

Aligning the tempos  (We don’t need no stinkin’ time stretcher)

Perhaps the biggest challenge of implementing the Bonhamizer is to align the tempo of the song and the Bonham beats. The tempo of the ‘Hammer of the Gods’ Bonham beat is 93 beats per minute. If fun.’s Some Nights is at 107 beats per minute we have to either speed Bonham up or slow fun. down to get the tempos to match.

Since the application is written in Javascript and intended to run entirely in the browser,  I don’t have access to server side time stretching/shifting libraries like SoundTouch or Dirac. Any time shifting needs to be done in Javascript. Writing a real-time (or even an ahead-of-time) time-shifter in Javascript is beyond what I could do in a 24 coding session. But with the help of the Echo Nest beat data and the Web Audio API there’s a really hacky (but neat) way to adjust song tempos without too many audio artifacts.

The Echo Nest analysis data includes the complete beat timing info. I know where every beat starts and how long it is. With the Web Audio API I can play any snippet of audio from an MP3, with millisecond timing accuracy. Using this data I can speed a song up by playing a song beat-by-beat, but adjusting the starting time and duration of each beat to correspond to the desired tempo. If I want to speed up the Bonham beats by a dozen beats per minute, I just play each beat about 200 ms shorter than it should be. Likewise, if I need to slow down a song, I can just let each beat play for longer than it should before the next beat plays. Yes, this can lead to horrible audio artifacts, but many of the sins of this hacky tempo adjust will be covered up by those big fat drum hits that will be landing on top of them.

I first used this hacky tempo adjustment in the Girl Talk in a Box hack.  Here’s a few examples of songs with large tempo adjustments.  In this  example,  AWOLNation’s SAIL is sped up by 25% you can hear the shortened beats.

And in this example with Sail slowed down by 25% you can hear double strikes for each beat.

I use this hacky tempo adjust to align the tempo of the song and the Bonham beats, but since I imagine that if a band had John Bonham playing the drums they would be driven to play a little faster, so I make sure that I speed on the song a little bit when I can.

Speeding up the drum beats by playing each drum beat a bit shorter than it is supposed to play gives pretty good audio results. However, using the technique to slow beats down can get a bit touchy. If a beat is played for too long, we will leak into the next beat. Audibly this sounds like a drummer’s flam. For the Bonhamizer this weakness is really an advantage. Here’s an example track with the Bonham beats slowed down enough so that you can easily hear the flam.

When to play (and not play) the drums

If the Bonhamizer overlaid every track with Bonham drums from start to finish, it would not be a very good sounding app.  Songs have loud parts and soft parts, they have breaks and drops, they have dramatic tempo changes.  It is important for the Bonhamizer to adapt to the song. It is much more dramatic for drums to refrain from striking during the quiet a capella solos and then landing like the Hammer of the Gods when the full band comes in as in this fun. track.

There are a couple of pieces of data from the Echo Nest analysis that I can use to help decide when to the drums should play.  First, there’s detailed loudness info.  For each beat in the song I retrieve information on how loud the beat is.  I can use this info to find the loud and the quiet parts of the song and react accordingly.   The second piece of information is  the beat confidence.  Each beat is tagged with a confidence level by the Echo Nest analyzer. This is an indication of how sure the analyzer is that a beat really happened there.  If a song has a very strong and steady beat, most of the beat confidences will be high. If a song is frequently changing tempo, or has weak beat onsets then many of the beat confidence levels will be low, especially during the tempo transition periods.  We can use both the loudness and the confidence levels to help us decide when the drums should play.

For example, here’s a plot that shows the beat confidence level for the first few hundred beats of Some Nights.

The red curve shows the confidence level of the beats.  You can see that the confidence ebbs and flows in the song.  Likewise we can look at the normalized loudness levels at the beat level, as shown by the green curve in the plot below:

We want to play the drums during the louder and high confidence sections of the song, and not play them otherwise.  However, if just simply match the loudness and confidence curves we will have drums that stutter start and stop, yielding a very unnatural sound.  Instead, we need to filter these levels so that we get a more natural starting and stopping of the drums.    The filter needs to respond quickly to changes – so for instance, if the song suddenly gets loud, we’d like the drums to come in right away, but the filter also needs to reject spurious changes – so if the song is loud and confident for only a few beats, the drums should hold back.  To implement this I used a simple forward looking runlength filter.   The filter looks for state changes in the confidence and loudness levels. If it sees one and it looks like the new state is going to be stable for the near future, then the new state is accepted, otherwise it is rejected.  The code is simple:

When the filter is applied we get a nice drum transitions – they start when the music gets louder and the tempo is regular and they stop when its soft or the tempo is changing or irregular:

This above plot shows the confidence and loudness levels for the Some Nights by Fun. The blue curve  shows when the drums play.  Here’s a detailed view of the first 200 beats:

You can see how transient changes in loudness and confidence are ignored, while longer term changes trigger state changes in the drums.

Note that some songs are always too soft or most of the beats are not confident enough to warrent adding drums.  If the filter rejects 80% of the song, then we tell the user that ‘Bonzo doesn’t really want to play this song’.

Aligning the song and the beats

At this point we now have tempo matched the song and the drums and have figured out when to and when not to play the drums. There’s just a few things left to do.  First thing, we need to make sure that each of the song and drum beats in a bar lines up so the both the song and drums are playing the same beat within a bar (beat 1, beat 2, beat 3,  beat 4, beat 1 …).  This is easy to do since the Echo Nest analysis provides bar info along with the beat info.  We align the beats by finding the bar position of the starting beat in the song and shifting the drum beats so that the bar position of the drum beats align with the song.

A seemingly more challenging issue is to deal with mismatched time signatures.  All of the Bonham beats are in 4-4 time, but not all songs are in 4.  Luckily, the Echo Nest analysis can tell us the time signature of the song.  I use a simple but fairly effective strategy. If a song is in 3/4 time I just drop one of the beats in the Bonham beat pattern to turn the drumming into a matching 3/4 time pattern.  It is 4 lines of code:

if (timeSignature == 3) {
    if (drumInfo.cur % 4 == 1) {
        drumInfo.cur+=1;
    }
 }

Here’s an example with Norwegian Wood (which is in 3/4 time).  This approach can be used for more exotic time signatures (such as 5/4 or 7/4) as well. For songs in 5/4, an extra Bonham beat can be inserted into each measure. (Note that I never got around to implementing this bit, so songs that are not in 4/4 or 3/4 will not sound very good).

Playing it all

Once we have aligned tempos, filtered the drums and aligned the beats, it is time to play the song. This is all done with the Web Audio API, which makes it possible for us to play the song beat by beat with fine grained control of the timing.

Issues + To DOs

There are certainly a number of issues with the Bonhamizer that I may address some day.

I’d like to be able to pick the best Bonham beat pattern automatically instead of relying on the user to pick the beat.  It may be interesting to try to vary the beat patterns within a song too, to make things a bit more interesting.  The overall loudness of the songs vs. the drums could be dynamically matched. Right now, the songs and drums play at their natural volume. Most times that’s fine, but sometimes one will overwhelm the other.   As noted above, I still need to make it work with more exotic time signatures too.

Reception

The Bonhamizer has been out on the web for about 3 days but in that time about 100K songs have been Bonhamized. There have been positive articles in The Verge,  Consequence of Sound, MetalSucks, Bad Ass Digest and Drummer Zone.  It is big in Japan. Despite all its flaws, people seem to like the Bonhamizer.  That makes me happy.  One interesting bit of feedback I received was from Jonathan Foote (father of MIR). He pointed me to Frank Zappa and his experiments with xenochrony The mind boggles.  My biggest regret is the name. I really should have called it the autobonhamator. But it is too late now … sigh.

All the code for the Bonhamizer is on github. Feel free to explore it, fork and it make your own Peartifier or Moonimator or Palmerizer.