FRAMINGHAM (06/26/2000) - Tim Lloyd thought he had committed the perfect electronic crime.
Lloyd, a former network administrator at Omega Engineering Corp., thought he had not only destroyed his former employer's manufacturing capabilities, but also had destroyed the evidence that would link him to the crime as well.
The one thing Lloyd didn't count on was that there were investigators with enough computer savvy to put the pieces of what prosecutors described as his "elaborate and Machiavellian plan" back together and trace the evidence to Lloyd's own doorstep.
In a purely legal sense, Lloyd's conviction in a Newark, New Jersey, federal court last month was a precedent-setting victory that proves the government is capable of tracking down and successfully prosecuting corporate computer crime.
In human terms, it's a case of a trusted, 11-year employee gone bad. Lloyd built the Novell NetWare computer network at Omega South and then blew it up with a software time bomb after he fell from corporate grace and was ultimately fired for performance and behavioral problems. Today, he faces a sentence of up to five years in prison.
In a business sense, the loss of its key manufacturing programs cost Omega, which builds measurement and instrumentation devices for customers like NASA and the U.S. Navy, more than US$10 million, dislodged its footing in the industry and eventually led to 80 layoffs.
The 1996 incident set off an intense investigation that brought together the U.S. Secret Service and one of the world's top data recovery and forensics experts to piece together the evidence that would ultimately lead to Lloyd's arrest and conviction.
"It's a unique case even to this day," says Leo Jackson, assistant to the special agent in charge with the U.S. Secret Service. "This was probably the first type of investigation [of this kind] that the service ever did. . . . It was about computers, but this case was good old hard detective work."
July 31, 1996
Omega South manufacturing plant, Bridgeport, N.J.
Between 8 and 8:30 in the morning, a worker at Omega Engineering's manufacturing plant started the day by logging on to the central file server.
Instead of booting up, a message came on the screen saying an area of the operating system was being fixed. Then the server crashed, and in an instant, all of the plant's 1,000 tooling and manufacturing programs were gone.
"I didn't know why the server was down,'' said Omega South Plant Manager Jim Ferguson, who was immediately alerted to the crash. "I just knew we had to get it back up. We needed those programs.''But the server wasn't coming back up. So Ferguson ordered that the manufacturing machines be kept running with programs that had been loaded the day before. It didn't matter if the orders already had been filled. He had to keep the machines running.
Then Ferguson went to get his salvation -- the backup tape. If the file server didn't come back up, he still would have all the programs safely stored on a backup tape kept in a filing cabinet in the human resources department.
But the tapes were gone. It was later discovered Lloyd had gone to Omega South's human resources department on July 1, taken out the backup tapes and never returned them, according to testimony.
Ferguson then turned to the workstations connected to the file server. The programs -- at least a good chunk of them -- should have been stored locally on the individual workstations. But the programs weren't there.
Ferguson learned that Lloyd had removed the programs from the workstations just days before he was fired and had centralized everything on the one file server.
"It was an awful feeling,'' Ferguson recalled. He quickly telephoned Lloyd.
"Tim, Tim do you have the backup tapes?'' said Assistant U.S. Attorney V. Grady O'Malley describing to the jury Ferguson's desperate call to Lloyd that day.
"Tim, we need those tapes. Are you sure you don't have the tapes?''Ferguson said Lloyd told him he didn't have the backup tapes. Lloyd, according to testimony, said he left them in the upper left corner drawer of his desk at Omega. But Ferguson himself had helped clean out Lloyd's desk. There was no backup tape.
In the days that followed the crash, Omega called in three different people to attempt data recovery, and Ferguson called Lloyd again and again.
Ferguson even went to Lloyd's house to plead in person. Lloyd handed Ferguson a few pieces of Omega property during the visit, but no tape.
"I had trusted Tim Lloyd completely," Ferguson told the jury. "We relied on Tim Lloyd. He was responsible for the security of the system."
Lloyd, the only Omega employee responsible for maintaining, securing and backing up the file server, wasn't replaced after he was fired. That meant Ferguson and other Omega executives had to turn to outside experts for help.
Five days after the crash, Ferguson was told by yet another data-recovery technician that the programs were gone and there didn't appear to be any way to get them back. Ferguson started shifting workers around the department and shutting down machines that were running out of raw materials or creating excess inventory. He took steps to hire a fleet of programmers to start rebuilding some of the 1,000 lost programs.
And he called in Ontrack Data International, a data-recovery firm out of Eden Prairie, Minn. Technicians from Ontrack, which handled 25,000 data recoveries in 1999 alone, made a mirror-image copy of Omega's damaged hard drives at the local office of the Secret Service and begin what would be a months-long search for the missing programs.
"We were doing everything we could. The other step would have been to shut down and lay off everybody," Ferguson told the jury. "We were just starting to get an idea of all the impact and what this was going to mean and how it was going to affect us."
The crash still affects Omega.
Ralph Michel, Omega's chief financial officer, testified that the software bomb destroyed all the programs and code generators that allowed the company to manufacture 25,000 different products and to customize those basic products into as many as 500,000 different designs.
"That department gave us flexibility to modify our products and gave us the ability to lower our costs," said Michel, who noted that Omega had shown 34 years of growth but started slipping after the computers crashed. "We lost both of those advantages in July 1996. . . . I believe the server crash was one of the principal reasons for the drop in sales, if not the reason."
"We will never recover," Ferguson told the jury.
The criminal investigation
August 12, 1996
Office of the Secret Service, Washington, D.C.
Omega executives put in a call to the U.S. Secret Service and told them they suspected the file server crash was the result of a criminal act.
Two days later, Special Agent William Hoffman arrived at Omega South. At the time, a relatively new statute made computer sabotage a federal offense if it affected a computer used in interstate commerce and caused more than $5,000 worth of damage to the company over a 12-month span.
"This wasn't just a simple investigation of a guy's PC at his home," said Hoffman in an exclusive interview after the trial. "We were looking at the network of a major corporation. . . . The sheer magnitude of it was beyond our experience at the time."
Hoffman, who has been with the Secret Service for four years, splitting his time between criminal investigations and protective service, started his probe by interviewing about 50 people at Omega, everyone from the company owners to people working on the shop floor.
"It was apparent to me very early on that this was not an accident," Hoffman said. "The files that had been deleted were surgically removed from the database. They specifically were the files the company needed to survive."
And from the beginning, all roads led to Lloyd. Hoffman noted that Lloyd was the only person who was tied to several key facets of the incident -- he had complete access to the network, he had Novell training and he was the last one known to have had the backup tape.
The search warrant
August 21, 1996
Timothy Lloyd's house in Delaware
At this point, Hoffman had enough to get a search warrant and arrived at Lloyd's home early in the morning.
Hoffman, working with several other agents from the Secret Service, went through Lloyd's home and garage, seizing about 700 pieces of potential evidence. That haul included computers, motherboards, keyboards, more than 500 disks, CD-ROMs, 12 hard drives and tapes. "It was enormous," Hoffman said.
What immediately stuck out were two backup tapes, which had both been erased.
One was labeled "Backup" with the dates "5/14/96" and "7/1/96" and Tim Lloyd's name. (July 1, 1996 was the date that Lloyd had asked for and been given Omega's backup tape.) Both tapes had been reformatted, a process which erases the data, the day before Ferguson visited Lloyd's house seeking the tapes.
"The moment I found out the backup tapes had been reformatted, my level of suspicion was elevated dramatically," said Hoffman, who acted as guardian of the evidence.
With the tapes, the 12 hard drives seized from Lloyd's house, and mirror images of the damaged hard drives from Omega's shop floor, Hoffman called in Ontrack.
Cracking the code
Ontrack offices in Minnesota
When Ontrack's data-recovery specialists realized that the programs had been blown up and scattered in random chunks through the million different storage spaces on the NetWare 3.12 operating system, they contacted Greg Olson, their director of Worldwide Data Recovery Services.
"We do data recoveries when companies are losing millions of dollars a day," said Olson, who has written data-recovery tools for Novell's NetWare operating system and even was brought in by the U.S. government to recover files from some of Kuwait's computers damaged during the Gulf War. "It's not uncommon for me to be working with people in panic mode, but. I've never seen this massive of a deletion in my 10 years of experience."
Olson said a few oddities about the way the file server was set up immediately raised red flags for him. "It was odd that the user accounts, most of them, had supervisory rights," he explained. "It's odd that Account 12345 had supervisory rights and no password."
Olson began by doing searches for common commands or phrases used in deletions, such as DEL /S; \*.*, DEL F:, DELTREE F: and PURGE F:\.
"I was just thinking of common things to search for and these were taking hits," Olson said. "Immediately, I knew this was hot when I saw PURGE take a hit."
Olson continued to systematically pull programming strings sitting in their raw form out of the code wreckage until he had pieced together six lines that looked like they could do some real damage.
"What's unusual are these six strings together," he said. "First of all, the date was meaningful because the data loss was the next day. The second thing was this logon account 12345, which had supervisory rights and no password. The next thing unusual is the fifth line that refers to all the data on the server, and /Y is a common command-line switch to make the program default to yes.
"This is the type of stuff you'd find in a utility to do mass something," Olson added. "The last thing is the PURGE. Having the PURGE there with the F:\ refers to the server and everything on it. And combined with that date, it was very unusual. You're not going to go into another company's file server and find that combination of strings. That was definitely a red flag situation."
Next, Olson set out to determine what part FIX.EXE -- which is not a NetWare executable and would not normally be found on a NetWare system -- played in the string. The way the strings were set up, he said he knew FIX.EXE must have deletion powers, but now it was a matter of proving it.
So Olson went out on the drive and pulled off 670 raw executables. He tested each and found one that appeared to be DELTREE.EXE, a DOS-based command that enables administrators to delete files from Windows operating systems.
"I pulled DELTREE and executed it with these command lines to see what would happen," Olson said. "I was shocked when the normal DELTREE function, saying 'deleting this, deleting this', was replaced with 'fixing this, fixing this.' I knew I was on to something there."
What he knew was that the DELTREE executable had been modified to disguise its deleting message by dropping in a 'fixing' message in its place. That was FIX.EXE. That one step camouflaged the deletion process so the user logging on to the system would never know what was actually happening.
Those six lines of code, which made up the time bomb, were written so it would detonate on boot up, no matter which user logged on first. Olson explained that the program deleted everything except NetWare-specific utilities, which are designed to be undeletable. And any deleted files normally go into a specific folder, where they could still be retrieved if someone knew how to look for them. The purge command, though, wipes away any trace of the 'addresses' for those deleted files, so even though the data is still sitting on the server, there's no longer any way to find them.
"Purge erases all evidence of where the data is," said Robert Hackett, remote data-recovery operations supervisor for Ontrack. "If you do a delete, it's like somebody putting paper in the trash bin. Purge is like shredding the paper into pieces . . . and taking the hundreds of thousands of pieces and tossing them up in the air."
Putting the code to the test
To test the code, Olson took an exact copy of the Omega file server and set up a test environment with an attached workstation. He then set out configuring the system for various dates prior to the July 30, 1996 date at the beginning of the code string.
Olson configured the system for Jan. 1, 1996 and logged on. Nothing unusual happened.
Then he tried April 30, 1996 and logged on. Nothing unusual happened. Then July 30, 1996. Nothing.
Then he configured the system for July 31, 1996, the exact date of the crash at Omega. "I logged on and everything on the system was deleted," he told the jury. "On the screen, it was saying it was fixing an area of the system, but actually it was deleting everything.
"The puzzle had been put together," he added. "There's absolutely no doubt in my mind that this is what caused the data loss."
Along with the six lines of code that did the damage, Olson also found three similar test strings.
Those three programs, each similar to the six lines of code in the damaging program, were dated Feb. 21, April 21 and May 30, 1996. One substituted a simple test folder, which could have held as little as one word, for the line in the damaging code that called for everything on the server to be deleted.
The third test program dated for May 30 was set up exactly as the code that brought down the system.
"If I wanted to test [my code] and didn't want to affect the use of the server, I would test it using a test folder," Olson said. "If it was May 30, 1996, and I knew it was going to trigger from the next day on, I would manually go in and move the date up."
"When Ontrack found the data string, I knew this was it," said Hoffman, who flew out to Minnesota to be at Ontrack for three days in February 1997. "I needed to know if it could have been hardware malfunction. User error. Human error . . . When all of these things were disproved, we knew we had a crime here."
With the code in hand, Olson went looking through the rest of the hard drives that Hoffman had given him to examine. And in that pile, he found those exact same six lines of code on one of Lloyd's personal hard drives that also stored his public relations photos, his checkbook software and personal letters.
"That's when I knew we had our guy," Hoffman said. "Then Grady O'Malley steps into the picture, and it was about getting the indictment.''The trialApril 17, 2000U.S. District Court, Newark, N.J.
O'Malley got the indictment on Jan. 28, 1998, and after several postponements, the trial began on April 17. It lasted four weeks.
Lloyd's defense was that Omega executives were blaming him for their own failings. "Computers crash. Networks crash. Sometimes you can't get them back up. That's what happened at Omega," argued attorney Edward Crisinino of Westmont, N.J.
"These are the guys who didn't have a network administrator. These are the guys whose heads are on the chopping block," he added during his closing arguments.
"It's about going to your boss and explaining why you didn't have a network administrator. It's about explaining why you didn't have a backup protocol."
Lloyd, who did not testify in court, said during an exclusive interview with Network World after the trial that he did not commit the crime.
"There's no way in the world I did this," Lloyd said. "I had complete access to the mainframe system from home. . . . If I was a vindictive person, do you think I'd go after a teeny, tiny little network?"
Lloyd even denied that he was the network administrator at Omega. "I never had an IT position anywhere," said Lloyd, who added that he's now working as a machinist at a Delaware company. "I've always been a machinist. The file server was so simplistic it didn't need to be maintained. The most ridiculous statements made during the trial were that I was the network administrator."
O'Malley, however, pointed out that Lloyd was the one who built the network for Omega. Lloyd was in charge of doling out passwords and access rights on the server. Lloyd installed the virus software and maintained the entire LAN.
Hoffman added, "It was clear to every single person I interviewed that Tim Lloyd was in charge of and maintained the system."
And O'Malley told the jury that it could not have been anyone other than Lloyd who could have taken that file server down in such a strategic and calculated fashion.
"Was the real guy sitting next to Tim Lloyd and fiddling with the system and changing dates?" O'Malley asked the jury. "I suggest not. Who could do all this and not be questioned by the administrator? No one. It was the administrator.
He was setting this up months in advance."
And what was Lloyd's motive? About a year before the crash, Lloyd had found himself losing status and clout as the company grew into a global corporation, acquiring businesses and adding plants and offices around the world. The technology star was being reduced to just another member of the team, according to witnesses for the prosecution.
Witnesses added that Lloyd's damaged ego and jealousy eventually took the form of physical intimidation of his coworkers, knowingly running faulty designs to make coworkers look bad, and bottlenecking a project because he wasn't in charge.
The prosecution contended that Lloyd was planning to leave Omega months before he was fired, that he had been going on job interviews and that he had tested the malicious code months prior to the actual crash.
"This was [Lloyd's] parting shot to a company he was leaving, a going-away gift. . . . And it was almost a perfect crime," O'Malley said. "There are two flaws here. One was the Secret Service's investigation and search of his house.
The other was that someone would untangle the web of his scheme that he thought had been purged from the file server."
The jury convicted Lloyd on the computer sabotage charge after three days of deliberation, but it acquitted him on the second charge of interstate transportation of stolen goods.
Lloyd, who maintains his innocence and says he will appeal, is remanded to his home state of Delaware until his scheduled July 31 sentencing - four years to the day after the server crashed at Omega Engineering.