Saturday, Nov. 22, 2003, 7:57 a.m. Origins of an Onslaught
The e-mail that started the online extortion demands began, "Your site is under attack," and it gave Mickey Richardson two choices: "You can send us $40K by Western Union [and] your site will be protected not just this weekend but for the next 12 months," or, "If you choose not to pay...you will be under attack each weekend for the next 20 weeks, or until you close your doors."
Richardson runs BetCris.com, an online wagering site, one of hundreds of sites ensconced in Costa Rica that take bets from Americans (and others around the world) without concern for U.S. bookmaking laws. Richardson received the e-mail just as he and his competitors were preparing for the year's busiest wagering season. With pro and college football, pro and college basketball and other sports in full swing, and with Thanksgiving and Christmas about to create plenty of free time, BetCris and the others stood to rake in millions over the holidays. Richardson was even planning an advertising blitz for the season to drive new traffic to his site.
If BetCris went down, he knew his customers would find another online bookmaker, "which will cost you tens of thousands of dollars in lost wagers and customers," the extortionists reminded him.
Despite all that, the e-mail didn't have the fearsome effect on Richardson that the extortionists hoped it would. He just asked his network administrator, Glenn Lebumfacil, if they should be concerned. "I said—God, in hindsight, what an idiot—I said, 'We should be safe. I think our network is nice and tight,'" recalls Lebumfacil.
As a precaution, Richardson alerted his ISP, but essentially, he says, "We kind of fluffed it off." The veteran bookmaker didn't panic because, in fact, he had dealt with online extortionists before. Two years earlier, hackers crashed BetCris.com with a denial-of-service (DoS) attack, and then demanded by e-mail a $500 protection fee in eGold (an online form of trading bullion). Richardson paid without a second thought. Compared to downtime, $500 was trivial.
That first attack got his attention, though. Richardson consulted another industry veteran who confessed to having a similar problem, and who told Richardson to call a consultant named Barrett Lyon in Sacramento, Calif. Lyon didn't come to BetCris's offices—he had no interest in baby-sitting infrastructure in Costa Rica—but he did recommend some off-the-shelf products that had recently been developed specifically to fight DoS attacks. Lyon thought (actually he hoped) that he'd never hear from them again. Richardson and Lebumfacil were confident they had protected themselves.
When the attack finally came on that Saturday in November, sometime after that first e-mail but before 11:30 a.m., BetCris crashed hard. The off-the-shelf products Lyon had recommended survived less than 10 minutes. BetCris's ISP crashed, and then the ISP for BetCris's ISP crashed. Richardson ran to the IT department, where Lebumfacil was watching the biggest DoS attack he'd ever seen. He remembers feeling sick to his stomach.
At 1:03 p.m., another e-mail arrived. "I guess you have decided to fight instead of making a deal. We thought you were smart.... You have 1 hour to make a deal today or it will cost you $50K to make a deal on Sunday." Then they knocked BetCris.com offline again.
The Extortion Problem
We know this about online extortion: It happens. Evidence of its prevalence or damage is speculative and anecdotal but useful nonetheless in guiding CSOs to understand the nature of the crime. Anecdotally, experts from law enforcement and information security consultants believe that perhaps one in 10 companies has been threatened with online extortion; in one survey by Carnegie Mellon University researchers, 17 out of 100 small and midsize businesses reported being targeted. Interviews with security consultants and industry players suggest that as many as three out of four cases of online extortion are never reported. Maybe a third or more of targeted companies pay extortion fees, drawing the money from disaster funds, acceptable loss budgets or insurance. Consultants like to tell stories of being called for help after companies pay protection money twice.
For CSOs and CISOs, it would be easy to view online extortion as indigenous to gambling sites, the karmic price one pays for choosing that line of work. It would also be wrong. True, the Thanksgiving-week attack on BetCris fronted a wave of extortion against gaming sites, but that wave has since ebbed (in part, we'll see, due to BetCris) while the online extortion phenomenon has not.
In fact, that wave of attacks against gaming sites, starting in late 2003 and going through mid-2004, appears to have been a training ground for extortionists. Now they've moved on, applying what they learned, along with more sophisticated technical tools, to attack far less prepared and more mainstream targets—such as online payment services, foreign currency exchanges and financial services companies. Here is a good rule of thumb: Anyone who could lose money by being offline is a potential online extortion target. And the more one stands to lose, the bigger the bull's-eye.
Yet you probably haven't thought much about online extortion unless you've been targeted. As with fraud, a certain shame attaches itself to victims, especially those who choose to pay protection fees. Even antiextortion consultants participate in a code of silence. One such company contacted for this story declined to comment "because we feel it brings attention to the crime."
That's why we're telling this story—to bring attention to the crime. To enable readers to learn from a real-world case what worked in an extortion crisis and what didn't. To sort out the choices one has before the choices one has are dictated by an e-mail.
Saturday, Nov. 22, 2003: Pleas for Time—and Help
Richardson and Lebumfacil decided to reply to the extortionists' e-mail. They stalled. Lebumfacil, the network administrator, recalls the pleading tone of their missives. (They sent several.) They'd say that they would lose their jobs if they didn't get more time. Richardson reluctantly admits that he feigned a family emergency and begged the extortionists to give him time until he could return from that to set up a payment.
Meanwhile, Lebumfacil and the IT team tried in vain to stop the attacks and get BetCris back online. The equation was simple: Downtime equals lost revenue. Richardson says the company stood to lose $1.16 every second, as much as $100,000 per day.
He tracked down Barrett Lyon, who was in Phoenix helping another company fight off a DoS attack. Lyon told Richardson to call the off-the-shelf equipment vendor. (He did. No help.) Call the ISP. (It couldn't help, either.)
Lyon says he sensed desperation, and he was right. Lebumfacil, who had a 5-month-old daughter at home, says, "I thought about losing my job. I thought about the company going out of business. There was a lot of money on the line. It was a constant state of panic." That night he tried in vain to sleep and says he even entertained the fantasy that "everything could be OK in the morning."
But it wasn't OK in the morning. At 10:01 a.m. on Sunday, Richardson got another e-mail. This one sounded less like a threat and more like the start of negotiations. "Dear Mickey, The attacks have been stopped 2 hours prior to the last e-mail. Your site is back up for most and should be up for all shortly...P.S. We will e-mail you Monday."
Still, Richardson wasn't encouraged. The site wasn't up at all; it only came to life sporadically and for short periods of time. No one knows for sure, but the extortionists might have stopped their attack. At some point, the downtime was the result of BetCris's ISP deciding to null-route the site's traffic. Null-routing means the ISP collects all of the traffic going to a site and drives it into the ground. This frees up the ISP's pipes when a site it hosts is receiving massive amounts of DoS attack traffic; even if the extortionists stopped attacking, the site would stay down.
Confusion and stress reigned. Richardson called Lyon again. This time, Lyon agreed to help. "I was thinking this would be a big mess for me," he says. "But they had no one to turn to. I knew by Sunday I couldn't pass them off any longer." Lyon flew back to Sacramento and started working on the problem. He, too, had dealt with online extortionists before.
Sunday, July 21, 2002: Flashback: The Kid Who Saved Vegas Sports Books
From a low-slung building off of Flamingo Drive in Las Vegas, a company called Don Best delivers the ever-fluctuating odds on sporting events to most of the glitzy sports books on the Strip. All of this is done by computers, and late in the evening on July 21, files started moving around one of those computers by themselves. An employee working late called Don Best's general manager, Rick Allec, and asked him what to do. Allec told him to turn off the server. The employee couldn't, so he literally pulled the plug out of the wall.
Allec rushed to the office, and soon he was holding the printout of an extortion e-mail demanding $200,000. He replied—and stalled—just as Richardson would a year later.
The next day, a security consultant told Allec to call Barrett Lyon for help. "When Barrett showed up," Allec recalls, "I remember thinking, There's no way he can help us."
Lyon was 23 and looked at least that young. His blond hair offset a tan, handsome face. Allec says Lyon looked like he had given up a day of surfing to swing by and help out.
Lyon had never taken a computer science class. His degree from California State University, Sacramento, was in philosophy, applied law and ethics. And yet he was cocky about computers. Once, he bet some friends he could map the entire Internet in a day. They scoffed. He launched Opte.org and mapped the entire Internet in a day. (Sort of. The open-source project is ongoing.) "People have never worried about my background," Lyon says, "because when they ask questions, I can answer them."
He had to win over Allec quickly, since Allec's customers were irate. A sports book forced to turn away wagers is like a bank turning down deposits. "We were down for three hours at one point, which was absolutely unheard of in our business," says Allec. "But Barrett made me comfortable. He would say, 'They're going to do this next, and we'll fight it this way.' And every time, he was exactly right. It was almost eerie."
At the time, off-the-shelf anti-DoS hardware wasn't readily available. Lyon's solution for Don Best was not to turn back the attack, but to scale Don Best's infrastructure of Web servers, load balancers and other hardware so that it was bigger than the volume of attack traffic coming in. "We basically built a humongous Web farm in, like, four days," Lyon says.
It proved to be enough to fend off the extortionists, who were sloppy. They attacked during the slowest gambling season, when the mark had less impetus to capitulate under pressure. They also asked for so much money that Allec didn't immediately determine that paying would be his smartest option.
Within a week, it was over. Except—and this impressed Allec the most—"a couple of weeks later I get a call from Barrett, and he says, 'I know who attacked your site.'"
Lyon says, "I could have left it alone, but I had gotten attached, and I started investigating. I came up with some interesting techniques to trace back the attacks." He turned over his work to several law enforcement agencies, but he never heard about it again.
It was Allec who recommended Lyon to Richardson after the $500 eGold incident. "During that time when all those sites were getting extorted, you only stopped it one of two ways," Allec says. "You either paid them off, or you called Barrett."
Monday, Nov. 24, 2003: Building the Defense
Lyon's plan for BetCris was to build a system that would absorb huge DoS attacks, and he had an idea how, technically, he might do that. But he had little idea how he would convince a tier-one hosting facility (essentially an ISP's ISP), to host his system—to voluntarily accept massive DoS attacks to see if his little project could thwart them.
Through his Opte.org project, Lyon knew of an ISP called PureGig in Phoenix with a 10Gbps pipe, plenty of bandwidth to host his system without disturbing PureGig's other customers. Lyon called Matt Wilson at PureGig. He begged.
A heated internal debate took place at PureGig. The company was ready to say no, Wilson told Lyon. Lyon begged harder.
Lyon believes what tipped PureGig to support his cause was altruism. "They told me they don't like to back down from challenges," he recalls. But it probably had as much to do with generating business. For, if Lyon and PureGig did figure out how to stop DoS attacks, they would have something that their competitors didn't.
"There was a great deal of skepticism here; it was not a popular idea," recalls Wilson. "My thinking was that normally the ISP's solution for DoS attacks is to shut off the customer," he says. (In other words, null-route them like the ISP did in shutting down BetCris.) Wilson adds, "In our minds, that wasn't a good long-term solution. Revenue issues aside, we thought maybe we could learn how to fix the problem. But still, it was a huge risk."
With PureGig committed, Lyon worked for the next three days without sleep, designing, building, testing, rebuilding and retesting his system. "I used all the methodologies I knew, all the code I knew, plus some new ideas."
Lyon kept in constant contact with PureGig and with Lebumfacil in Costa Rica. Lebumfacil deferred to Lyon. "I was part of it, I stayed up all night with him on the line," Lebumfacil says. "I was never allowed to touch any of the boxes. I would make suggestions, and he'd take some of it and not take some of it.
"Barrett had his idea. There was so much uncertainty. Many times I thought, I hope he knows what he's doing. But Barrett had this calm confidence. You want to freak out, and he just works. He's so focused."
By Wednesday, Lyon had something. A patchwork of original code stitched together with commercial products, he described it as "a highly fortified data center with proxy and security software and some monitoring, and more bandwidth than the bad guys."
Denial of Service, Deconstructed
Denial-of-service attacks are an old and crass way to disrupt a network, and yet still are immensely effective. DoS attacks overload the pipes that connect computers to the Internet with massive amounts of legitimate but useless data. DoS attacks create epic traffic jams. The cars in this analogy would be requests for service that hackers send to the target website. Each time the target site gets a request, it must deny it. But because the hacker sends massive numbers of requests from thousands of computers, the target must use nearly all of its time and resources just to deny these requests for service, effectively blocking access to anyone with a legitimate request.
Before that, though, the hacker must create a network of computers big enough to overwhelm the target. They don't buy these computers, they commandeer them. They plant software scripts on systems distributed throughout the world (hence, distributed denial of service, or DDoS). These compromised computers are called zombies, or bots, because they generate attack traffic automatically, without the owners' knowledge.
You spent the last two hours handicapping a football game that is now approaching kick-off. Your efforts tell you that the Green Bay Packers should win the contest by at least three points. At the time you started picking apart this Monday night game, stat by stat and injury by injury, your sportsbooks were offering a tasty -3 +105, BUT the line moved against you while you were checking your almanac. Your bookmaker is now asking you to give up -3.5 points at -110 for a play on the cheese heads. If you don’t have access to a bookie who is still hanging the -3, you are either forced to pass on the event or go for it, risking more than you know you should. Many of us are action junkies and would lay the extra half point, although betting in this manner will eventually eat away at our bank roll.
