Music Machinery

This morning Time.com published the final result for their annual TIME 100 Poll.  Time reports  that the new owner of the title ‘Worlds’s most influential person, is moot’. What TIME doesn’t say is that their poll was so totally manipulated that the results of the poll are not an indication of who is the most influential, but instead they stand as a monument to Time’s incompetence.

Looking at the poll results we see clear evidence of the hack.  The first letters of the top 21 finalists in the poll spell out ‘Marblecake, also the game’. Evidence of precision hackery for anyone to see.  And yet, Time says they rebuffed all attempts to hack the poll. Quoting from the time article:  “TIME.com’s technical team did detect and extinguish several attempts to hack the vote”.    Which leads me to wonder whether Time.com is being dishonest or is just plain incompetent. Considering Hanlon’s razor , I have to go with incompetence.  (And if you have any doubt about Time’s incompetence, take a close look at the Poll.  Notice that Oprah Winfrey and Ratan Tata have the exact same number of votes. That’s because they both shared the same ID in the poll.  A vote for either one was a vote for the other. Same goes for Michael Bloomberg and Gustavo Dudamel. If you vote for one, you vote for the other.)

How did the hack happen? I’ve already described in great detail the steps that the loose collective known as ‘Anonymous’ took to hack the poll. This group (that gathered on an IRC channel at anonnet.org) probed for weaknesses in the poll protocols and wrote autovoters to stuff the ballot box with votes that would put the candidates in the proper order to spell out the Message, adapting as necessary whenever Time adjusted its protocol in a meager attempt to keep the hackers out.  But two weeks ago, Time got serious about poll security.  They modified the poll so that you needed to prove that you were human (via a captcha) in order to vote.

This instantly shut down all of the autovoters.  Anonymous was offline – no longer able to submit thousands of votes per minute.  And what’s worse, when the autovoters were shutdown, the Message ‘Marblecake, also the game’ soon decayed into a meaningless “mablre caelakosteghamm”.  It seemed that Time.com had won – the Message would not survive the next two weeks of voting.  But Anonymous didn’t give up, they considered it a challenge to restore the Message.  Here’s how they did it.

Update -4/29 Professor Luis von Ahn, the project lead for reCAPTCHA,  sent me a very polite email suggesting that I change a few words here to make it clear to a casual reader that reCAPTCHA was not hacked.  I agree that the original post could be easily misinterpreted by a casual reader, so  I’ve changed a couple of words here and there to make it absolutely clear that reCAPTCHA was not compromised for the Time Poll.

First attempt – trying (and failing) to crack reCAPTCHA
The first thing Anonymous tried to do was tried to break reCAPTCHA, the captcha technology used by Time.com.  They built a program that would analyze the images, break the words into characters and apply OCR to the images in an attempt to automate the captcha process.  However, unsurprisingly, it proved to be too difficult of a task – certainly that was a nut that would take more than a week to crack.  So after a few days, they abandoned this approach.

Second Attempt:  trying (and failing) to hack reCAPTCHA –   ‘The Penis Flood’

The next tactic used was to see if they could find a flaw in the reCAPTCHA implementation.  One thing they discovered about reCAPTCHA was that it always presents two words to a user for decoding – one word is a control word known by the reCAPTCHA system, while the other is an unknown  word (reCAPTCHA uses the humans to help correct OCR errors).  Wikipedia describes the process: “Scanned text is subjected to analysis by two different optical character recognition programs; in cases where the programs disagree, the questionable word is converted into a CAPTCHA. The word is displayed along with a control word already known and is labeled by the human.  Those words that are consistently given a single label by human judges are recycled as control words”.  What Anonymous realized was that if they always labeled the unknown scanned text with the same word – and if they did this thousands and thousands of times eventually a large percentage of the unknown words would be mislabeled with their word. All they had to do was look at the two words in the captcha, enter the proper label for the ‘easy’ one (presumably that would be the one that the two optical scanners would agree upon) and enter the word “penis” for the hard one.  If they did this often enough, then soon a significant percentage of the images would be labeled as ‘penis’ and the ability to autovote would be restored (one side effect, that was not lost on Anonymous, was the notion that for years to come there would be a number of  digital books with  the word ‘penis’ randomly inserted throughout the text.    Update: I asked Ben Maurer, chief engineer of reCAPTCHA about this ‘penis flood‘ attack, Ben says that they’ve anticipated this type of attack and they have numerous protections that will keep the penises from penetrating the reCAPTCHA barrier.   Update – 4/29 – Luis von Ahn, the project lead  of reCAPTCHA goes on to say ” about the “penis attack”. We serve over 400 million CAPTCHAs per week, so submitting 200k CAPTCHAS with the word penis doesn’t even come close to poisoning our database — we serve each word to multiple random users, and we require them to be correct on the other word, so to get any traction with this attack, they would have had to submit at least 100 times more CAPTCHAs. And even if they did this, we have many other measures against it. That attack simply doesn’t work.”

Third Attempt: Optimizing  reCAPTCHA entry
As appealing as the notion of sprinkling the word ‘penis’ into texts, the Anonymous team knew that the clock was ticking, and if they were going to restore the Message they didn’t have time to wait for the autovoters to come back online – they were going to have to vote manually, many, many times. And so they needed to be able to enter captcha’s as fast as they could. They developed a set of guidelines that allowed them to quickly decide which reCAPTCHA words they could skip. For example:

You will be given 2 words: 1 real, 1 fake.

For [REAL FAKE] or [FAKE REAL], you can just type in REAL and it should be accepted.

If it’s [LOOKSREAL LOOKSREAL] or [LOOKSFAKE LOOKSFAKE], it’s usually just quicker to just type in both words.  Don’t waste precious time deciding which one of them is real.

Use both the appearance and the type of word to identify a fake
word.  Don’t rely on just one of them.

The whole ruleset is here: fake captcha

By understanding how reCAPTCHA worked – the team was able to double their productivity (since they usually only had to enter one word instead of two).  To further optimize their voting they created a  poll front-end that allowed you to enter votes quickly while giving you an update of the poll status (and since it is a 4chan kind of crowd, they also provided the option to stream some porn just to keep you company while you are subverting one of the largest media companies in the world.

They found that with this version of the manual loader, the thing that was taking the most time was loading the captcha images, so they made a bare bones version that loaded 3 captchas at a time, in the background eliminating this bottleneck, and doubling their manual voting speed once more (and showing them vote per minute stats).

Update – Just to be perfectly clear, anon didn’t hack reCAPTCHA. It did exactly what it was supposed to do. It shut down the auto voters instantly and effectively. The only option left after Time added reCAPTCHA to the poll was a brute force attack.    Ben Maurer,  (chief engineer on reCAPTCHA) comments on the hack: “reCAPTCHA put up a hard to break barrier that forced the attackers to spend hundreds of hours to obtain a relatively small number of votes. reCAPTCHA prevented numerous would-be attackers from engaging in an attack. In any high-profile system, it’s important to implement reCAPTCHA as part of a larger defense-in-depth strategy”.    As Dr. von Ahn points out  “had Time used reCAPTCHA from the beginning, this would have never happened — anon submitted *tens of millions* of votes before Time added reCAPTCHA, but they were only able to submit ~200k afterwards. And to do this, they had to resort to typing the CAPTCHAs by hand!” One thing that Time inc. did that made it much easier for the anonymous hack was to allow leave the door open for cross-site request forgeries which allowed anon to create a streamlined poll  that never had to fetch data from Time.com.

Brute Force

With the streamlined manual voting process, a single, motivated voter could cast 30 votes per minute (perhaps only 20 VPM if they were watching porn).  But some calculations showed that they needed about 200K votes to cast to get everyone in their proper position.  If they were going to succeed they really had to organize their votes.  They churned the numbers and came up with this plan:

TOTAL VOTES NEEDED 191,209

Alexander Levedev (up to 37.5) 6,541 votes
Rick Warren (more than 1,902,130) 7,255 votes
Kobe Bryant (up to 39.50) 109,174 votes
Sheikh Ahmed bin Zayed Al Nahyan (up to 35.50) 5,000 votes
Hu Jintao (up to 31.50) 19,836 votes
Elizabeth Warren (up to 27.50) 43,403 votes

With a sprinkling of help from folks on /b/, the core team of about a dozen got down to manual voting. (To get help from /b/ they put together info on how to streamline the captcha process, how to configure the browser to mask referrals, deal with proxies and provided some other (perhaps not-safe-for work  incentives).  Some of the most hardcore voters  (I call them ‘devoters’) spent  40+ hours voting.  At their peak, they were casting about 200 votes per minute (compared to the many, many thousands per minute that they could cast via autovoter before Time added the captcha).

With 200k votes to cast, they knew it would be close, and they didn’t know exactly when the polls were closing.  In the final days the crew was getting demotivated. But one  boost to their productivity and morale occurred when they sussed out how Time actually did the final ordering (they round the average rating to the nearest rating, and then use the total number of votes to break a tie).  With this little nugget of information, they were able to redistribute how they voted, eliminating the need for about 30K of the 200K votes.  They discovered a few more quirks in how Time.com ranked the candidates which allowed them to shave even more votes off the required total for a total savings of 46k votes.  With these vote savings, the goal was close at hand,  with their boosted morale they were able to push across the finish line.

The End Game
Finally, on Friday, Time closed the poll, but funny thing was they didn’t turn off the polling URLs, so even though you couldn’t vote through the official Time.com website, it was still possible to vote via the streamlined manual voter – and so the ballot stuffing continued.  On Saturday afternoon, the message was restored, but the voting continued – as the team tried  to gain a cushion of safety, should voters for other candidates mess things up at the last minute.  Early morning on April 27th Time.com published the results.  And there, for the whole world to see was the message, completely intact,”mARBLECAKE ALSO THE GAME”.

Celebrations were in order – there was cake

and happy faces

and a general sigh of relief from the group.

It is 12 hours after Time.com poll has been closed.  The mood among Anonymous is high – the hack was completed, it is there for the world to see.  Time.com behaved as  expected – they refused to acknowledge the hack and the Message – but the word is out there.  People are reading about the hack on 4chan, Reddit and Digg – people know that the poll was hacked and they know that Anonymous is responsible.  They started with a goal and despite some rather severe setbacks were able to meet that goal

From where I sit, I really have to wonder about Time.com.  They spent their time  promoting and running this poll that they know (or should know) is a total farce. They give a  wink and nudge to the questionable results by saying “This is an  Internet poll. Doubting the results is kind of the point.” Which is just stupid.  Perhaps the point should be “if you want to maintain any kind of journalistic  integrity, don’t conduct online polls”.

So what’s next for Anonymous? One hacker (knowing the stereotype people have for  an Anonymous hacker) says “we’re going to resume masturbating and being the total failures that we are “.  When I asked Zombocom, the mastermind of the Message , if he had any message for moot – the man that they put on top of the world – Zombocom replied: ‘ “The Game” – but still, enjoy it.’

Update: A mini-interview with moot:
A friend put me in touch with moot so I could ask him about the hack.  Since he’s so influential I kept my questions short and to the point. Here’s the mini-interview:

Time makes a joke a your expense (“To put the magnitude of the upset in perspective, it’s worth noting that everyone Moot beat out actually has a job. “).  Any response to Time magazine about this:

I wasn’t offended by the blurb on TIME.com. To clarify, I never claimed to be unaware of the “concerted plan to influence the poll,” just that I hadn’t instructed anybody to vote for me. They did it all on their own (as you already know).

Time also indicates that they rebuffed the attempts to hack the poll. (“TIME.com’s technical team did detect and extinguish several attempts to hack the vote. “).  This seems to me to be a lie.  Likewise, they ignore the ‘marblecake, also the game’ message completely. Anything to say about this?

Honestly, I think Time had as much fun with the poll as we all did. It drove a lot of traffic to their site, and after the final results were released, generated a lot of buzz about the upcoming issue.

There’s a group of a dozen or so guys who’ve devoted a couple of months to this.  Anything to say to them?

As for a response to the players: “Thanks.”

Update 10/24/2012:  

Ukraine translation by Gmail Archive – http://www.stoodio.org/moot-wins-time-inc-loses.