Disaster recovery plan: Finding the best solution for the money |
 |
By Michael Ybarra, Contributor
16 Sep 2008 | SearchCIO-Midmarket.com |
|
|
When Hurricane Frances slammed into West Palm Beach in 2004, MedVance Institute went into its disaster recovery (DR) mode: IT Director Dan Weiss packed the school's servers into his car and drove to his house, where he set up a makeshift data center for several days.
MedVance, which trains medical technicians at campuses across four Southeastern states, was hard hit by the hurricane. It took out three schools, including blowing the roof off one, and it destroyed servers housing several weeks' worth of user data that wasn't backed up.
"We didn't have a cohesive disaster recovery plan in place," Weiss said. "We took our core services for customer relationship management and email and physically relocated them. The roof was blown off a couple of campuses; servers wound up underwater. That was a big wake-up call."
MedVance is typical of many companies that have just entered the midmarket space: Growth and dependence on technology is exploding, but DR and business continuity (DR/BC) planning is lagging. The company started with a single campus seven years ago in Tennessee but has grown to 11 locations (with headquarters in West Palm Beach). There are plans to open two more schools this year and four the next.
Weiss has been at the company for five years, during which time the IT department has changed dramatically. Previously, MedVance had a director of IT, a senior network engineer and a director of information systems. The roles were combined into a single director-level position and the company outsourced its technology support to vendor partners, mainly CDW Government Inc. (CDW-G), a consulting firm based in Vernon Hills, Il.
Growing the business had been IT's mandate in the early years, but the hurricane experience impressed upon the company the importance of protecting the business as well. The old data center was a server closet, which the company had intended to augment with a colocation facility as a backup center -- something that became urgent after the hurricane.
Getting management to sign off on investing in DR/BC was a no-brainer after the Frances episode, but Weiss says it was difficult to sort through all the competing methodologies and stay on budget.
"The biggest challenge was figuring out the best solution for the cost," he said. "There are so many different options. Good or best practices but usable and affordable for a midrange company with aggressive growth targets is what we were looking for. We wanted the most cost-efficient way to get the greatest redundancy."
Weiss wound up working with CDW-G and other technology partners to research options and ensure that all components would properly configure. His recommendation was to move the data center to a remote location, which management approved. In a week, the company shipped the servers to Atlanta and had the new data center up and running. A second failover facility is being added in Dallas. Both facilities offered the ability to scale to MedVance's growth projections.
MedVance also deployed a turnkey storage area network solution from LeftHand Networks Inc. using four network cards, which eliminated the problem of running out of storage space while also continuously replicating all data. The solution also improved network performance by running the data on four networks, instead of bundling traffic through one band.
|
|
|
|
|
The roof was blown off a couple of campuses; servers wound up underwater. That was a big wake-up call.
Dan Weiss
IT director, MedVance Institute
|
|
|
|
|
|
|
|
|
|
If the primary server fails, the network can be up and running in two minutes, Weiss said. And when facing a catastrophic failure, the network can be back up in six hours. Additionally, all key data is replicated in real time and redundant copy is saved on the storage area network.
Protecting IT was one aspect of DR, but Weiss also needed a way to keep employees and students aware of what was going on in the event of a disaster. During Frances, it was difficult to get the word out about which campuses were open or closed.
"The biggest thing for us was communication," Weiss said. "We had a hard time tracking everyone down."
MedVance is deploying Microsoft's SharePoint collaboration solution so staff can check in with the company during a disaster, as well as establishing 800 numbers with information for employees and students.
It's a big change from 2004, when DR failover was counted in days, not minutes -- mission-critical systems are fully protected and data recovery is robust. Now, Weiss said, the company considers itself well prepared for whatever Mother Nature can throw at it.
"The DR plan scales well," Weiss said. "That was one of the key components. We learned through bitter experience."
Michael Ybarra is a monthly columnist for SearchCIO-Midmarket.com and a former senior writer at CIO Decisions magazine. He is also the author of Washington Gone Crazy. Write to him at editor@searchcio-midmarket.com.
|
|
|
|
