Posts Tagged hack
This morning Time.com published the final result for their annual TIME 100 Poll. Time reports that the new owner of the title ‘Worlds’s most influential person, is moot’. What TIME doesn’t say is that their poll was so totally manipulated that the results of the poll are not an indication of who is the most influential, but instead they stand as a monument to Time’s incompetence.
Looking at the poll results we see clear evidence of the hack. The first letters of the top 21 finalists in the poll spell out ‘Marblecake, also the game’. Evidence of precision hackery for anyone to see. And yet, Time says they rebuffed all attempts to hack the poll. Quoting from the time article: “TIME.com’s technical team did detect and extinguish several attempts to hack the vote”. Which leads me to wonder whether Time.com is being dishonest or is just plain incompetent. Considering Hanlon’s razor , I have to go with incompetence. (And if you have any doubt about Time’s incompetence, take a close look at the Poll. Notice that Oprah Winfrey and Ratan Tata have the exact same number of votes. That’s because they both shared the same ID in the poll. A vote for either one was a vote for the other. Same goes for Michael Bloomberg and Gustavo Dudamel. If you vote for one, you vote for the other.)
How did the hack happen? I’ve already described in great detail the steps that the loose collective known as ‘Anonymous’ took to hack the poll. This group (that gathered on an IRC channel at anonnet.org) probed for weaknesses in the poll protocols and wrote autovoters to stuff the ballot box with votes that would put the candidates in the proper order to spell out the Message, adapting as necessary whenever Time adjusted its protocol in a meager attempt to keep the hackers out. But two weeks ago, Time got serious about poll security. They modified the poll so that you needed to prove that you were human (via a captcha) in order to vote.
This instantly shut down all of the autovoters. Anonymous was offline – no longer able to submit thousands of votes per minute. And what’s worse, when the autovoters were shutdown, the Message ‘Marblecake, also the game’ soon decayed into a meaningless “mablre caelakosteghamm”. It seemed that Time.com had won – the Message would not survive the next two weeks of voting. But Anonymous didn’t give up, they considered it a challenge to restore the Message. Here’s how they did it.
Update -4/29 Professor Luis von Ahn, the project lead for reCAPTCHA, sent me a very polite email suggesting that I change a few words here to make it clear to a casual reader that reCAPTCHA was not hacked. I agree that the original post could be easily misinterpreted by a casual reader, so I’ve changed a couple of words here and there to make it absolutely clear that reCAPTCHA was not compromised for the Time Poll.
First attempt – trying (and failing) to crack reCAPTCHA
The first thing Anonymous tried to do was tried to break reCAPTCHA, the captcha technology used by Time.com. They built a program that would analyze the images, break the words into characters and apply OCR to the images in an attempt to automate the captcha process. However, unsurprisingly, it proved to be too difficult of a task – certainly that was a nut that would take more than a week to crack. So after a few days, they abandoned this approach.
Second Attempt: trying (and failing) to hack reCAPTCHA – ‘The Penis Flood’
The next tactic used was to see if they could find a flaw in the reCAPTCHA implementation. One thing they discovered about reCAPTCHA was that it always presents two words to a user for decoding – one word is a control word known by the reCAPTCHA system, while the other is an unknown word (reCAPTCHA uses the humans to help correct OCR errors). Wikipedia describes the process: “Scanned text is subjected to analysis by two different optical character recognition programs; in cases where the programs disagree, the questionable word is converted into a CAPTCHA. The word is displayed along with a control word already known and is labeled by the human. Those words that are consistently given a single label by human judges are recycled as control words”. What Anonymous realized was that if they always labeled the unknown scanned text with the same word – and if they did this thousands and thousands of times eventually a large percentage of the unknown words would be mislabeled with their word. All they had to do was look at the two words in the captcha, enter the proper label for the ‘easy’ one (presumably that would be the one that the two optical scanners would agree upon) and enter the word “penis” for the hard one. If they did this often enough, then soon a significant percentage of the images would be labeled as ‘penis’ and the ability to autovote would be restored (one side effect, that was not lost on Anonymous, was the notion that for years to come there would be a number of digital books with the word ‘penis’ randomly inserted throughout the text. Update: I asked Ben Maurer, chief engineer of reCAPTCHA about this ‘penis flood‘ attack, Ben says that they’ve anticipated this type of attack and they have numerous protections that will keep the penises from penetrating the reCAPTCHA barrier. Update – 4/29 – Luis von Ahn, the project lead of reCAPTCHA goes on to say ” about the “penis attack”. We serve over 400 million CAPTCHAs per week, so submitting 200k CAPTCHAS with the word penis doesn’t even come close to poisoning our database — we serve each word to multiple random users, and we require them to be correct on the other word, so to get any traction with this attack, they would have had to submit at least 100 times more CAPTCHAs. And even if they did this, we have many other measures against it. That attack simply doesn’t work.”
Third Attempt: Optimizing reCAPTCHA entry
As appealing as the notion of sprinkling the word ‘penis’ into texts, the Anonymous team knew that the clock was ticking, and if they were going to restore the Message they didn’t have time to wait for the autovoters to come back online – they were going to have to vote manually, many, many times. And so they needed to be able to enter captcha’s as fast as they could. They developed a set of guidelines that allowed them to quickly decide which reCAPTCHA words they could skip. For example:
You will be given 2 words: 1 real, 1 fake.
For [REAL FAKE] or [FAKE REAL], you can just type in REAL and it should be accepted.
If it’s [LOOKSREAL LOOKSREAL] or [LOOKSFAKE LOOKSFAKE], it’s usually just quicker to just type in both words. Don’t waste precious time deciding which one of them is real.
Use both the appearance and the type of word to identify a fake
word. Don’t rely on just one of them.
The whole ruleset is here: fake captcha
By understanding how reCAPTCHA worked – the team was able to double their productivity (since they usually only had to enter one word instead of two). To further optimize their voting they created a poll front-end that allowed you to enter votes quickly while giving you an update of the poll status (and since it is a 4chan kind of crowd, they also provided the option to stream some porn just to keep you company while you are subverting one of the largest media companies in the world.
They found that with this version of the manual loader, the thing that was taking the most time was loading the captcha images, so they made a bare bones version that loaded 3 captchas at a time, in the background eliminating this bottleneck, and doubling their manual voting speed once more (and showing them vote per minute stats).
Update – Just to be perfectly clear, anon didn’t hack reCAPTCHA. It did exactly what it was supposed to do. It shut down the auto voters instantly and effectively. The only option left after Time added reCAPTCHA to the poll was a brute force attack. Ben Maurer, (chief engineer on reCAPTCHA) comments on the hack: “reCAPTCHA put up a hard to break barrier that forced the attackers to spend hundreds of hours to obtain a relatively small number of votes. reCAPTCHA prevented numerous would-be attackers from engaging in an attack. In any high-profile system, it’s important to implement reCAPTCHA as part of a larger defense-in-depth strategy”. As Dr. von Ahn points out “had Time used reCAPTCHA from the beginning, this would have never happened — anon submitted *tens of millions* of votes before Time added reCAPTCHA, but they were only able to submit ~200k afterwards. And to do this, they had to resort to typing the CAPTCHAs by hand!” One thing that Time inc. did that made it much easier for the anonymous hack was to allow leave the door open for cross-site request forgeries which allowed anon to create a streamlined poll that never had to fetch data from Time.com.
With the streamlined manual voting process, a single, motivated voter could cast 30 votes per minute (perhaps only 20 VPM if they were watching porn). But some calculations showed that they needed about 200K votes to cast to get everyone in their proper position. If they were going to succeed they really had to organize their votes. They churned the numbers and came up with this plan:
TOTAL VOTES NEEDED 191,209
Alexander Levedev (up to 37.5) 6,541 votes
Rick Warren (more than 1,902,130) 7,255 votes
Kobe Bryant (up to 39.50) 109,174 votes
Sheikh Ahmed bin Zayed Al Nahyan (up to 35.50) 5,000 votes
Hu Jintao (up to 31.50) 19,836 votes
Elizabeth Warren (up to 27.50) 43,403 votes
With a sprinkling of help from folks on /b/, the core team of about a dozen got down to manual voting. (To get help from /b/ they put together info on how to streamline the captcha process, how to configure the browser to mask referrals, deal with proxies and provided some other (perhaps not-safe-for work incentives). Some of the most hardcore voters (I call them ‘devoters’) spent 40+ hours voting. At their peak, they were casting about 200 votes per minute (compared to the many, many thousands per minute that they could cast via autovoter before Time added the captcha).
With 200k votes to cast, they knew it would be close, and they didn’t know exactly when the polls were closing. In the final days the crew was getting demotivated. But one boost to their productivity and morale occurred when they sussed out how Time actually did the final ordering (they round the average rating to the nearest rating, and then use the total number of votes to break a tie). With this little nugget of information, they were able to redistribute how they voted, eliminating the need for about 30K of the 200K votes. They discovered a few more quirks in how Time.com ranked the candidates which allowed them to shave even more votes off the required total for a total savings of 46k votes. With these vote savings, the goal was close at hand, with their boosted morale they were able to push across the finish line.
The End Game
Finally, on Friday, Time closed the poll, but funny thing was they didn’t turn off the polling URLs, so even though you couldn’t vote through the official Time.com website, it was still possible to vote via the streamlined manual voter – and so the ballot stuffing continued. On Saturday afternoon, the message was restored, but the voting continued – as the team tried to gain a cushion of safety, should voters for other candidates mess things up at the last minute. Early morning on April 27th Time.com published the results. And there, for the whole world to see was the message, completely intact,”mARBLECAKE ALSO THE GAME”.
Celebrations were in order – there was cake
and a general sigh of relief from the group.
It is 12 hours after Time.com poll has been closed. The mood among Anonymous is high – the hack was completed, it is there for the world to see. Time.com behaved as expected – they refused to acknowledge the hack and the Message – but the word is out there. People are reading about the hack on 4chan, Reddit and Digg – people know that the poll was hacked and they know that Anonymous is responsible. They started with a goal and despite some rather severe setbacks were able to meet that goal
From where I sit, I really have to wonder about Time.com. They spent their time promoting and running this poll that they know (or should know) is a total farce. They give a wink and nudge to the questionable results by saying “This is an Internet poll. Doubting the results is kind of the point.” Which is just stupid. Perhaps the point should be “if you want to maintain any kind of journalistic integrity, don’t conduct online polls”.
So what’s next for Anonymous? One hacker (knowing the stereotype people have for an Anonymous hacker) says “we’re going to resume masturbating and being the total failures that we are “. When I asked Zombocom, the mastermind of the Message , if he had any message for moot – the man that they put on top of the world – Zombocom replied: ‘ “The Game” – but still, enjoy it.’
Update: A mini-interview with moot:
A friend put me in touch with moot so I could ask him about the hack. Since he’s so influential I kept my questions short and to the point. Here’s the mini-interview:
Time makes a joke a your expense (“To put the magnitude of the upset in perspective, it’s worth noting that everyone Moot beat out actually has a job. “). Any response to Time magazine about this:
I wasn’t offended by the blurb on TIME.com. To clarify, I never claimed to be unaware of the “concerted plan to influence the poll,” just that I hadn’t instructed anybody to vote for me. They did it all on their own (as you already know).
Time also indicates that they rebuffed the attempts to hack the poll. (“TIME.com’s technical team did detect and extinguish several attempts to hack the vote. “). This seems to me to be a lie. Likewise, they ignore the ‘marblecake, also the game’ message completely. Anything to say about this?
Honestly, I think Time had as much fun with the poll as we all did. It drove a lot of traffic to their site, and after the final results were released, generated a lot of buzz about the upcoming issue.
There’s a group of a dozen or so guys who’ve devoted a couple of months to this. Anything to say to them?
As for a response to the players: “Thanks.”
It looks like Time has taken some action to combat the hack of the Time 100 Poll. They are now using a captcha to verify that the voter is a human – the result being that the 4chan autovoters are now being banned.
With the new defenses in place, the delicate balance of the poll results order can no longer be maintained by the /b/tards. The Message is no more:
After just a couple of hours, the Message has decayed from “marblecake also the game” to “mablre caelakosteghamm”.
I don’t think this is the final shot in the war. I suspect that even as I type this the 4chan folks are poking and prodding, looking for another chink in Time’s armor. It will be interesting to see if and when they respond. Still, 4chan has awoken the sleeping giant. They’ve been noticed, and whatever they do, now that the giant is awake and paying attention, it will be much harder for them. But, I wouldn’t bet against the /b/tards yet. It’s like the final moments in Star Wars Episode V. Yes, Han Solo is currently frozen in carbonite, but honestly, you know he’s going to make it out in the next episode.
There’s a scene toward the end of the book Contact by Carl Sagan, where the protagonist Ellie Arroway finds a Message embedded deep in the digits of PI. The Message is perhaps an artifact of an extremely advanced intelligence that apparently manipulated one of the fundamental constants of the universe as a testament to their power as they wove space and time. I’m reminded of this scene by the Time.com 100 Poll where millions have voted on who are the world’s most influential people in government, science, technology and the arts. Just as Ellie found a Message embedded in PI, we find a Message embedded in the results of this poll. Looking at the first letters of each of the top 21 leading names in the poll we find the message “marblecake, also the game”. The poll announces (perhaps subtly) to the world, that the most influential are not the Obamas, Britneys or the Rick Warrens of the world, the most influential are an extremely advanced intelligence: the hackers.
At 4AM this morning I received an email inviting me to an IRC chatroom where someone would explain to me exactly how the Time.com 100 Poll was precision hacked. Naturally, I was a bit suspicious. Anyone could claim to be responsible for the hack – but I ventured onto the IRC channel (feeling a bit like a Woodward or Bernstein meeting Deep Throat in a parking garage). After talking to ‘Zombocom’ (not his real nick) for a few minutes, it was clear that Zombocom was a key player in the hack. He explained how it all works.
Zombocom told me that it all started out when the folks that hang out on the random board of 4chan (sometimes known as /b/) became aware that Time.com had enlisted moot (the founder of 4chan) as one of the candidates in the Time.com 100 poll. A little investigation showed that a poll vote could be submitted just by doing an HTTP get on the URL:
where ID is a number associated with the person being voted for (in this case 1883924 is Rain’s ID).
Soon afterward, several people crafted ‘autovoters’ that would use the simple voting URL protocol to vote for moot. These simple autovoters could be triggered by an easily embeddable ‘spam URL’. The autovoters were very flexible allowing the rating to be set for any poll candidate. For example, the URL
could be used to push 160 ratings of 1 (the worst rating) for the artist Rain to the Time.com poll.
In early stages of the poll, Time.com didn’t have any authentication or validation – the door was wide open to any client that wanted to stuff the ballot box. Soon these autovoting spam urls were sprinkled around the web voting up moot. If you were a fan of Rain, it is likely that when you visited a Rain forum, you were really voting for moot via one of these spam urls.
Soon afterward, it was discovered that the Time.com Poll didn’t even range check its parameters to ensure that the ratings fell within the 1 to 100 range. The autovoters were adapted to take advantage of this loophole, which resulted in the Time.com poll showing moot with a 300% rating, while all other candidates had ratings far below zero. Time.com apparently noticed this and intervened by eliminating millions of votes for moot and restoring the poll to a previous state (presumably) from a backup. Shortly afterward, Time.com changed the protocol to attempt to authenticate votes by requiring that a key be appended to the poll submission URL that consisted of an MD5 hash of the URL + a secret word (AKA ‘the salt’).
“Needless to say, we were enraged” says Zombocom. /b/ responded by getting organized – they created an IRC channel (#time_vote) devoted to the hack, and started to recruit. Shortly afterward, one of the members discovered that the ‘salt’, the key to authenticating requests, was poorly hidden in Time.com’s voting flash application and could be extracted. With the salt in hand – the autovoters were back online, rocking the vote.
Another challenge faced by the autovoters was that if you voted for the same person more often than once every 13 seconds, your IP would be banned from voting. However, it was noticed that you could cycle through votes for other candidates during those 13 seconds. The autovoters quickly adapted to take advantage of this loophole interleaving up-votes for moot with down-votes for the competition ensuring that no candidate received a vote more frequently than once every 13 seconds, while maximizing the voting leverage.
One of the first autovoters was MOOTHATTAN. This is a simple moot up-voter that will vote for moot about 100 times per minute. (Warning, just by visiting that site, you’ll invoke the autovoter – so if you don’t want to hack the vote, you should probably skip the visit).
Here’s a screenshot of another autovoter, a program called Mooter, developed by rdn:
Mooter is a Delphi app (windows only) that can submit about 300 votes per minute from a single IP address. It will also take advantage of any proxies and cycle through them so that the votes appear to be coming from multiple IP addresses. rdn, the author of Mooter, has used Mooter to submit 20 thousand votes in a single 15 minute period. In the last two weeks, (when rdn started keeping track) Mooter alone has submitted 10,000,000 votes (about 3.3% of the total number of poll votes).
From the screenshot you can see that Mooter is quite a sophisticated application. It allows fine grained control over who receives votes, what type of rating they get, voting frequency, the proxy cycle, along with charts and graphs showing all sorts of nifty data.
In addition to highly configurable autovoting apps, the loose collective of #time_vote maintains charts and graphs of the various candidate voting histories. Here’s a voting graph that shows the per-minute frequency of votes for boxer Manny Pacquiao.
More charts are available for browsing at (the very slow to load) http://fun.qinip.com/mvdc/mootvote.php
So with the charts, graphs, spam URLs and autovoters #time_vote had things well in hand. Moot would easily cruise to a victory. Although they still had some annoying competition, especially from fans of the boxer Manny Paquoia. Zombocom says that “it can take upwards of 4.5K votes a minute to keep Manny in his place”. Despite the Manny problem, the #time_vote collective had complete dominance of the poll.
The Ultimate Precision Hack
At this point Zombocom was starting to get bored and so he started fiddling with his voting scripts. Much to his surprise, he found that no matter what he did, he was never getting banned by Time.com. Zombocom suspects that his ban immunity may be because he’s running an ipv6 stack which may be confusing Time.com’s IP blocker. With no 13 second rate limit to worry about, he was able to crank out votes as fast as his computer would let him – about 5,000 votes a minute (and soon he’ll have a new server online that should give him up to 50,000 votes a minute.) With this new found power, Zombocom was able to take the hack to the next level.
Zombocom joked to one of his friends “it would be funny to troll Time.com and put us up as most influential, but since we are not explicitly on the list we’ll have to spell it out. ” His friend thought it was impossible. But two weeks later, “marblecake’ was indeed spelled out for all to see at the top of the Time.com poll.
So what is the significance of ‘marblecake’? Zombocom says: ” Marblecake was an irc channel where the “Message to Scientology” video originated. Many believe we are “dead” or only doing hugraids etc, so I thought it would also be a way of saying : we’re still around and we don’t just do only “moralfag” stuff .
To actually manipulate the poll, Zombocom wrote two perl scripts. The first one, auto.pl is pretty simple. It finds the highest rated person in the poll that is not in the desired top 21 (recall, there are 21 characters in the Message) and down-votes them (you can view this as eliminating the riff-raff). The second perl script, the_game.pl is responsible for maintaining the proper order of the top 21 by inspecting the rating of a particular person and comparing that rating to what it should be to maintain the proper order and then up-voting or down-voting as necessary to get the desired rating. With these two scripts, (less than 200 lines of perl) Zombocom can put the poll in any order he wants.
Ultimately, this hack involved lots of work and a little bit of luck. Someone figured out the voting URL protocol. A bunch of folks wrote various autovoters, which were then used by a thousand or more to stack the vote in moots favor. Others, sprinkled the spam urls throughout the forums tricking the ‘competition’ into voting for moot. When Time.com responded by trying to close the door on the hacks, the loose collective rallied and a member discovered the ‘salt’ that would re-open the poll to the autovoters. The lucky bit was when Zombocom discovered that no matter what he did, he wouldn’t get banned. This opened the door to the fine grained manipulation that led to the embedding of the Message.
At the core of the hack is the work of a dozen or so, backed by an army of a thousand who downloaded and ran the autovoters and also backed by an untold number of others that unwittingly fell prey to the spam url autovoters. So why do they do it? Why do they write code, build complex applications, publish graphs – why do they organize a team that is more effective than most startup companies? Says Zombocom: “For the lulz”.
A bad day for my friends at Spotify. First the news of a security breach that compromised the personal information of their one million users – followed by the outage of the Spotify.com website as a million people all tried to change their passwords at once. But despite all of this trouble, the Spotify player kept playing music.
It is interesting to see how Spotify is handling their first big crises. So far, they seem to be doing most things right – they are being open about what the problem was and they have already fixed the problem that has caused the breach. Looks like they may need to be a bigger web server though.