The Powers That Beat: Denial of Service: Deconstructed
Denial of Service, Deconstructed
Denial-of-service attacks are an old and crass way to disrupt a network, and yet still are immensely effective. DoS attacks overload the pipes that connect computers to the Internet with massive amounts of legitimate but useless data. DoS attacks create epic traffic jams. The cars in this analogy would be requests for service that hackers send to the target website. Each time the target site gets a request, it must deny it. But because the hacker sends massive numbers of requests from thousands of computers, the target must use nearly all of its time and resources just to deny these requests for service, effectively blocking access to anyone with a legitimate request.
Before that, though, the hacker must create a network of computers big enough to overwhelm the target. They don’t buy these computers, they commandeer them. They plant software scripts on systems distributed throughout the world (hence, distributed denial of service, or DDoS). These compromised computers are called zombies, or bots, because they generate attack traffic automatically, without the owners’ knowledge.
Hackers create zombies by scanning for exposed systems that they can manipulate remotely. Often these are home and office broadband users. (Lately, existing bot networks have been found scanning for more computers to turn into bots when they’re not launching attacks of their own—akin to an army recruiting its soldiers in peacetime. One security consultant said he connected an unsecured computer to the Internet to see what would happen, and it was recruited within three minutes.) Hackers can also insert their attack code through phishing, spyware, viruses and social engineering. Universities have long been popular spots for creating zombies because of the number of easily accessible, unsecured public computers.
With a zombie network in place, the only issue left is scale. The more zombies on a network, and the more aggregate upstream bandwidth they have, the swifter and more severe havoc they can wreak. Several hundred computers could generate 100MB of traffic, enough to knock a small network offline. A 10,000-computer bot network could deliver a 1Gb attack, enough to knock anyone offline who hasn’t installed some rudimentary anti-DDoS infrastructure.
Some experts believe that right now different sets of hackers are engaged in an arms race to see who can build the biggest zombie network. Not for bragging rights, but for renting out the networks to anyone who wants to launch an attack, the raw capitalist idea being that the biggest network will generate the best rental business.
Tuesday, Nov. 25, 2003: Running Out of Time
The extortionists’ e-mail that arrived on this morning demonstrated that they were losing whatever patience they had: [all typos sic] “I told you that if you try and f*** with us that your site will be down forever…. The excuse that you were in the hospital does not matter to me. So here are your choices: 1) You have until 4pm est today to send us our $40K. 2) You have until 4pm est Wednesday to send us $50K if you can not send the $40K today. 3) You do not pay and your site will be down for 4 days starting Thursday and it will cost you $75K to come back up Monday. 4) You do nothing and do not respond to this email within an hour and we will make sure you are down forever….”
Richardson was panicked. He can’t remember precisely when—the entire week has blurred in his memory—but by this time, he had reported the crime to the National Hi-Tech Crime Unit (NHTCU) in Scotland Yard. According to an NHTCU spokeswoman, the unit had already opened a similar investigation with a British gaming site called CanBet.
According to Richardson and Lyon, the NHTCU encouraged Richardson to wire two extortion payments of a few thousand dollars each to separate Western Union offices in Eastern Europe. The NHTCU wanted to nab anyone who showed up to take the cash. (NHTCU won’t confirm this; the spokeswoman said the unit does not discuss investigative tactics.) Richardson agreed, but for a different reason: He wanted his site back up. “I knew another person [in the industry] who was successful getting back online by sending three or four small payments like this,” Richardson says, “and those guys didn’t even have a solution to the problem when they paid. I knew Barrett was getting closer and closer to a solution. So I sent the payments, thinking maybe I can get a good week out of this.”
But no one took the bait. After about two weeks, Richardson pulled the money back.
Wednesday, Nov. 26, 2003: Barrett’s Big Bet
From Sacramento, Lyon instructed the PureGig engineers who would turn on his system 630 miles southeast, in Phoenix. Another 2,400 miles southeast from Phoenix, everyone at BetCris waited impatiently.
Lyon’s system intercepted traffic headed for BetCris’s servers in Costa Rica, diverted it to his creation in Phoenix, scrubbed off the attack traffic and delivered legitimate traffic back to Costa Rica. It was designed to bar DDoS traffic from touching BetCris. If the system failed, it couldn’t defend BetCris, and it wouldn’t be able to send legitimate traffic to Costa Rica. But BetCris itself wasn’t getting attacked. The system did a lot of other stuff too: monitoring, capacity planning, logging and analysis.
It wasn’t perfect. After it was installed, Lyon had to tweak routers on the network, install new versions of software and add capacity to his system. The extortionists kept changing attack vectors, and Lyon and his team kept tweaking. It was a constant battle, but Lyon was confident that the system would enable BetCris.com to stay online. Wilson at PureGig called Lyon’s system “ingenious” not because it was unique—it was monitoring and filtering at a proxy location—but because Lyon’s monitoring and filtering seemed to stop attacks better than any other effort he’d seen.
But when it was first turned on, the extortionists stuffed too much traffic down its throat. Wilson recalls the math: “We had 100MB links to the DNS servers. We went from handling under 2MB per link to, all of a sudden, 600MB.” That’s six times a full load. Imagine Fenway Park, which holds about 35,000 people. Now imagine 200,000 people trying to get inside Fenway Park at one time.
The DNS servers were overloaded, and Phoenix got tense.
Costa Rica had been tense for nearly a week (as much as half a million dollars in lost revenue), but now BetCris was bordering on despair. Mickey Richardson lacked sleep, and he struggled to make decisions and lead. His IT staff was fracturing, feeling impotent as they watched the attacks and waited for Lyon. BetCris’s small call center staff was getting abused around the clock by customers calling in to vent frustration and demand to know what the heck was going on. The simple task of creating a smart message about what was happening eluded Richardson. “You can’t just have your call center staff tell people you were hacked,” Richardson says, because it creates more questions than answers.
At the same time, his decision not to pay the extortionists was affecting other wagering sites that shared the same ISP and were experiencing network problems. “I’m getting calls from friendly competitors saying, ‘Look, Mickey, we paid. Just pay. We’re going down because of you.'”
He was running out of time and energy. Richardson remembers around this time having to update his staff—275 or so people who weren’t entirely sure they’d have a job soon—and he couldn’t even find words. He thought, “I wish they could read my mind because I’m too exhausted to explain it anymore. I don’t have any answers.”
In hindsight, Richardson says, he would have spent more time preparing for these human issues attached to the crisis—decision making under pressure, keeping the staff together—and less time worrying about technical defenses. Yes, create those technical defenses and make sure you have a crisis response plan. But also focus more on issues like exhaustion and emotional distress, and how they can be handled.
It was in this context that Richardson received an e-mail, at 11:12 a.m. It caused him to feel, for the first time, “blind fear.”
“I would like to thank you for not keeping your end of the deal and making this upcoming weekend an enjoyable one for me.” The extortionists demanded $75,000, but then seemed to disregard the money. “I do not care how long I have to destroy your business and I will. You will learn the hard way that you do not make a deal and then f*** around with us…. Let the games begin.”
Richardson would soon learn they were not bluffing. They could destroy his business, and they were going to try. For BetCris to survive, Lyon’s slapdash system in Phoenix, which was just starting to find its purchase, would have to stand up to the biggest DDoS attack any of them had ever seen.
The DNS servers that had overloaded in Phoenix were brought back online in a couple of hours, after Lyon and Wilson adapted some filtering scripts and increased the size of their network pipes.
Lyon then spent Thanksgiving and Friday eating leftover turkey his girlfriend delivered and tweaking his system to absorb bigger DDoS attacks. On Friday, he believed it could handle a 1Gb attack, and he felt good about that. He assured a frayed Richardson that he’d never see an attack that big. It would take tens of thousands of zombie computers.
Which is exactly what happened. It turns out the extortionists had more than 20,000 zombies. PureGig’s data center suffered badly, which affected several of its ISP customers. PureGig decided to take Lyon’s system offline to fix it.
“The attack went to 1.5Gb, with bursts up to 3Gb. It wasn’t targeted at one thing. It was going to routers, DNS servers, mail servers, websites. It was like a battlefield, where there’s an explosion over here, then over there, then it’s quiet, then another explosion somewhere else,” says Lyon. “They threw everything they had at us. I was just in shock.”
Richardson recalls the attack: “So I have Barrett on the line, who I think is the second coming, and he says, ‘Let me think about this. Give me some time.’ And I say, ‘OK, I don’t want to pressure you. I have faith. But if you don’t fix it, I’m out of business.'”
Why Online Extortion Works
It was never supposed to have gotten to this point; Richardson was supposed to have paid long ago. The extortionists expertly optimized the chances of it.
To ensure a quick, quiet transaction, the extortionists did what all extortionists (in the physical or online world) do: They exploited the problem of the commons. An ecological principle, the problem of the commons states that people will act in self-interest if it profits them in the short term, even if that act will hurt everyone, including themselves, in the long term. Every act, every threat, every negotiation tactic, every single move extortionists make is designed to make paying the protection fee not only appealing, but in fact, the smartest business decision you can make in the short term, even if you know in the long run that you haven’t stopped the problem at all.
Thus, extortionists attack when it hurts the target the most; they ask for $10,000 to $100,000 (generally considered the sweet spot of extortionist profitability versus victim willingness to pay, depending on the size of the victim company).
In BetCris’s case, the extortionists revealed they were Eastern European, which would make them hard to find, never mind prosecute. Online crime laws are weaker in Eastern Europe than in the United States and the desire to enforce them weaker still (and the FBI wouldn’t get involved with offshore gaming sites being extorted from overseas).
The online version of extortion provides unique advantages (relative anonymity, low probability of prosecution, lots of easy targets, diminished chance of physical violence) that have made it a highly lucrative business alternative for bad guys.
BetCris was just another easy target. What the extortionists didn’t count on was the unlikely confluence of Richardson’s resolve, Lyon’s ingenuity and an ISP that would provide them a place to fight back.
Friday, Dec. 12, 2003: BetCris Wins the War of Attrition
The extortionists must have screamed “Hooy na ny!” or some other Russian expletive after their blitzkrieg, when Lyon “got the chemistry down” and managed to absorb the massive amounts of attack traffic and get PureGig and BetCris back up and running. Lyon assumed the bad guys would come back with something bigger, as hard as that was to imagine, so he set out to scale up his system “for whatever was next, a 6Gb attack or something.”
Elyssa D. Durant, Ed.M.DailyDDoSe © 2007-2014