Posts Tagged gwap

Some NameDropper stats

The NameDropper has been live for less than a day and already I ‘ve collected some good data from the game play.   Here are some stats:

Total games played:  1841
Total unique players:  462
Total play time:  30hrs, 20mins, 36 seconds

The artists that were most frequently confused with fake artists were:

93 Erik Satie
82 Ennio Morricone
66 Edvard Grieg
49 Nightwish
46 Yann Tiersen
40 Maurice Ravel
35 Porcupine Tree
34 Felix Mendelssohn
32 Dinah Washington
30 Antonio Vivaldi
This is showing that the game (and the underlying familiarity algorithm that drives it) may have a bias toward classical musicians.  One of the knobs we have in our familiarity algorithm is a boost for age of the artist (even though Beethoven doesn’t get played on the radio very often, most people have heard of him, so he gets a boost for being old).  But this early data suggests that we are giving a bit too much of an age boost.  Ideally, the most frequently confused artists would be spread evenly across genre. The fact that there are no Hip-Hop or Pop artists on this list suggests that our familiarity algorithm skews away from those artists.  Looks like we’ll have a few knobs to tweak.
Game-with-a-purpose are a great way to get data from humans that would otherwise be difficult to get.  The challenge in creating such a game though is to make one that is actually fun to play.  If your game is no fun, you won’t get any data at all.   Looking at player stats can give us an idea of how fun the game is.    Some NameDropper player stats:

Games per player:
Average:  4
Median: 2
Max: 182    (woah, someone played 182 times!)
Time spent playing:
Average: 236 secs
Median: 90 secs
Max:  10,995 secs (over 3 hours)
Scores:
Average: 9139
Median: 4054
Min: -4656
Max: 54671

Yep, someone scored -4656 points.  If you take too long to answer you get dinged (to avoid the Wikipedia winner).
I was a bit concerned as to whether or not someone could ‘learn’ the game (my daughter Cari suggested as much), so I looked at the scores over time of one of the frequent players to see if the score trended upwards.
There doesn’t seem to be any noticeable trend for this user, so it seems that the game is perhaps a good measure of general music knowledge and not a measure of how may times you’ve played the game.  People are still playing the game – and there seems to be a good fight for the top spot on the leaderboard which means that I should continue to get good data.  Next step is to tweak the familiarity knobs based on the data collected so far and update the game to see if the average scores go up.

, ,

6 Comments

The Name Dropper

[tweetmeme source= ‘plamere’ only_single=false]

TL;DR;  I built a game called Name Dropper that tests your knowledge of music artists.

One bit of data that we provide via our web APIs is Artist Familiarity.  This is a number between 0 and 1 that indicates how likely it is that someone has heard of that artists.    There’s no absolute right answer of course – who can really tell if Lady Gaga is more well known than Barbara Streisand or  whether Elvis is more well known than Madonna. But we can certainly say that The Beatles are more well known, in general, than Justin Bieber.

To make sure our familiarity scores are good, we have a Q/A process where a person knowledgeable in music ranks our familiarity score by scanning through a list of artists ordered in descending familiarity until they start finding artists that they don’t recognize.  The further they get into the list, the better the list is.  We can use this scoring technique to rank multiple different familiarity algorithms quickly and accurately.

One thing I noticed, is that not only could we tell how good our familiarity score was with this technique, this also gives a good indication of how well the tester  knows music.  The further a tester gets into a list before they can’t recognize artists, the more they tend to know about music.   This insight led me to create a new game:  The Name Dropper.

The Name Dropper is a simple game.  You are presented with a list of dozen artist names.  One name is a fake, the rest are real.

If you find the fake, you go onto the next round, but if you get fooled, the game is over.    At first, it is pretty easy to spot the fakes, but each round gets a little harder,  and sooner or later you’ll reach the point where you are not sure, and you’ll have to guess.  I think a person’s score is fairly representative of how broad their knowledge of music artists are.

The biggest technical challenge in building the application was coming up with a credible fake artist name generator.  I could have used Brian’s list of fake names – but it was more fun trying to build one myself.  I think it works pretty well.  I really can’t share how it works since that could give folks a hint as to what a fake name might look like and skew scores (I’m sure it helps boost my own scores by a few points).  The really nifty thing about this game is it is a game-with-a-purpose.  With this game I can collect all sorts of data about artist familiarity and use the data to help improve our algorithms.

So go ahead, give the Name Dropper a try and see if you can push me out of the top spot on the leaderboard:

Play the Name Dropper


, , , , ,

6 Comments

Herd it on Facebook

UCSD Researcher Gert Lanckriet announced today that “Herd It” – the game-with-a-purpose for collecting audio annotations has been officially launched on Facebook.  Following in the footsteps of other gwaps such as Major Miner, Tag-a-tune and the Listen Game.

On the music-ir mailing list Gert explains ‘Herd it’:  “The scientific goal of this experiment is to investigate whether a human computation game integrated with the Facebook social network will allow the collection of high quality tags for audio clips. The quality of the tags will be tested by using them to train an automatic music tagging system (based on statistical models). Its predictive accuracy will be compared to a system trained on high quality tags collected through controlled human surveys (such as, e.g., the CAL500 data set). The central question we want to answer is whether the “game tags” can train an auto-tagging system as (or more) accurately than “survey tags” and, if yes, under what conditions (amount of tags needed, etc.). The results will be reported once enough data has been collected.”

I’ve played a few rounds of the game and enjoyed myself.  I recognized all of the music that they played (it seemed to be drawn from top 100 artists like Nirvana, Led Zeppelin, Maria Carey and John Lennon).  The timed rounds made the game move quickly. Overall, the game was fun.  But I did miss the feeling of close collaboration that I would get from some other Gwaps where I would have to try to guess how my partner would try to describe a song.  Despite this, I found the games to be fun and I could easily see spending a few hours trying to get a top score.  The team at UCSD clearly has put lots of time into making the games highly interactive and fun.  Animations, sound and transparent game play all add to the gaming experience.  Once glitch, even though I was logged into Facebook, the Herd It game didn’t seem to know who I was, it just called me ‘Herd It’.  So my awesome highscore is anonymous.

Here are some screen shots from the game.  For this round, I had to chose the most prominent sound (this was for the song ‘Heart of Gold’), I chose slide guitar, but most people chose acoustic guitar (what do they know!).

For this round, I had to chose the genre for a song. easy enough.

For this round I had to position a song on a Thayer mood model scale.

Here’s the game kick off screen … as you can see, I’m “Herd it” and not Paul

I hope the Herd It game attracts lots of attention.  It could be a great source of music metadata.

, , , ,

2 Comments

TagatuneJam

TagATune, the music-oriented ‘game with a purpose’ is now serving music from Jamendo.com.   TagATune has already been an excellent source of high quality music labels.  Now they will be getting gamers to apply music labels to popular music.  A new dataset  will be forthcoming.  Also, adding to the excitement of this release, is the announcment of a contest.  The highest scoring Jammer will be formally acknowledged as a contributor to this dataset as well as receive a special mytery prize.  (I think it might be jam). Sweet.

, , ,

Leave a comment