|
A month after Katrina, Oreck's IT systems were mostly restored, and the company began to plan for moving back to New Orleans.
It was far from easy. Evanson looked into syncing up the New Orleans data center with Boulder over the network, but the volume of data and the system architecture ruled out that possibility. Turning off the system, running backup tapes and restarting again was the only practical option. That, too, was complicated. Getting the tapes to New Orleans proved difficult: Flights to the city were still limited, and cancellations common. A full backup of the AS/400 also takes eight hours, and twice that time for a second set of tapes. It would take another day to get everything back up and running. All of which meant that Oreck would have to go offline for three days.
"Three days," Evanson notes, "was a real problem for the business."
The factory was starting to catch up with production, but distribution was still lagging. Oreck had invested heavily in advertising, and its call center partner needed the systems to handle orders.
Must-Haves in a DR Plan |
Redundancy. Back up critical IT systems, but also have a plan for running all systems necessary to keep the business alive.
Communications. Devise a way to get in touch with employees if local cell phones don't work.
Operations. Have a plan for running the business from a new location for an extended period of time.
Families. Recognize that employees need housing and food before they can return to work and that their children need schools to attend
Infrastructure. Account for a worst-case scenario in which local government and municipal services shut down.
--M.Y.
|
Evanson told his IT department to find a way to do it faster. "I gave my team the challenge to do it in one day," Evanson says. "We had one shot."
On Thursday, Sept. 29, the company committed to moving back over the weekend of Oct. 8. The company ordered a superfast $15,000 tape drive for next-day delivery. It would perform a backup in three hours and would be used to transfer data from backup tapes made in Boulder to the machines that returned to New Orleans.
On Friday, however, the package didn't show up.
"Emery [a delivery company] was more or less unresponsive, didn't know where the drive was and said they couldn't deliver it before next Friday," Evanson recalls. "That was obviously too late."
Luckily for Oreck, the company had a strong relationship with UPS, which had recently acquired Emery. UPS found the package in Emery's system and sent a team to pick it up in Shreveport, La. The drive wasn't there. UPS sent teams to walk Emery terminals around the country looking for the package. The drive was finally located -- on a train headed to Dallas. UPS yanked the box from the train and drove it to New Orleans, where it finally arrived at around 11 a.m. on Thursday, Oct. 6, just in time for the weekend's planned switchover.
The next morning, server administrator Pat Eiermann flew from Dallas to Boulder. That day the network servers in Dallas were turned off and trucked to New Orleans. At 6 p.m. on Saturday, Oreck shut down the AS/400 in Boulder and ran off two sets of backup tapes. At 6 a.m. on Sunday morning, Eiermann hopped aboard a $10,000 chartered plane with the tapes. He was in New Orleans by 11 a.m.
On Monday morning, Oct. 10, Oreck was back home.
|
