Many businesses that go to the trouble and expense of putting together a disaster recovery plan fail to factor in the one critical element to success in an actual calamity: disaster recovery testing.
"Most people select the strategy they are going to use for disaster recovery and they don't pay any attention to how they are going to test it. We have to plan not only to succeed but to test," said Jon Toigo, CEO of Toigo Partners International, a technology advisory based in Dunedin, Fla.
Somewhere between 50% and 70% of organizations develop disaster recovery plans, industry surveys show, Toigo said. "Of those, fewer than half actually test those plans, which is tantamount to having no plan at all," he said.
"Testing is the long-tail cost of disaster recovery. Even companies that succeed in going through the process of developing the plan usually fail when it comes to testing the plan," said Toigo this week at a Storage Decisions Seminar on disaster recovery planning in Newton, Mass.
Rock-solid plans? Really?
Asked about their disaster recovery plans, 39% of IT shops say theirs are "rock solid," 45% say theirs "might work" and 16% are "keeping their fingers crossed," Toigo said, citing an industry survey. He said he doesn't believe anyone who says his plan is rock-solid.
"Those people scare the hell out of me. They obviously have never experienced a disaster," Toigo said.
Additionally, the vast majority of organizations that do disaster recovery testing do so only once a year, Toigo said, or not nearly enough to take into account changes to business processes, products, staff or the technology systems. (Bizarrely, 44% of disaster recovery planners polled haven't told anybody that a DR plan exists, Toigo said, perhaps with the thought that if it doesn't work they can't be...
To continue reading for free, register below or login
To read more you must become a member of SearchCIO-Midmarket.com
blamed.)
DR becomes a capability only after testing. The exercises provide valuable gap analyses and rehearsals. Rehearsals prove valuable in the "smoke of battle," not only for executing the predetermined plan but also for handling the unpredictable. "Maybe you'll be able to think on your feet, if you're not learning everything for the first time," Toigo said.
Therefore, it is important to train a team. Toigo said it is less important which particular job titles are on the team -- an obsession of a lot of DR consultants -- than whether the person has the requisite knowledge for his job. The network gal needs to know how to get the network up and running. The business process guy needs to know the ins and outs of the business process he is responsible for, and so on. "Get somebody who knows what they are doing," Toigo said.
Second, look for people who have demonstrated an ability to work under pressure and are goal-oriented. Frame of mind is important, Toigo said. "You want people who have a certain level of enthusiasm when faced with a challenge, and you want people who like to complete what they start."
Toigo is not alone in advocating testing. DR providers interviewed about how companies can trim DR costs in these budget-constrained times all said that companies should not forego testing.
"People always want to talk about delaying testing in the economic downturn. Never, never! It is worth so much to test. Otherwise what you have is an idea. You have a potential but not a possibility or probability," said Bill Hughes, director of consulting services at SunGard Availability Services LP.
Bob Boyd, CEO of Charlotte, N.C.-based Agility Recovery Solutions Inc., which will bring a trailer on-site in the event of a disaster, said he's seen companies develop disaster recovery plans to satisfy regulatory requirements and then let them gather dust.
"They haven't actually done an exercise in three years. That is going to be less and less acceptable, given the nature of the regulatory pressure right now," he said. But he empathizes. Testing is not only expensive but also hard to get time for at a large hot site, because the vendors are booked up for 18 months.
Disaster recovery testing in the real world
Based on a smattering of interviews with attendees of Toigo's three-hour seminar, a DR plan -- let alone testing -- remains a hard sell at many small organizations.
Disaster recovery funding "goes in cycles," said Paul Fontaine, manager of network administration at Providence College in Providence, R.I. He said he's hoping to capitalize on the current interest in DR to move ahead on choosing a location for a DR recovery site.
As a member of the Rhode Island higher education consortium OSHEAN Inc. (The Ocean State Higher Education / Economic Development and Administrative Network), Providence College can use a facility in Springfield, Mass., for disaster recovery, Fontaine said, but the cost of sending staff members there is a deterrent. If the college does set up a DR site, it will likely make do with another location on campus and in the meantime rely on existing redundancy built into its systems, he said.
David Skrabal, a junior system administrator at Bedford, Mass.-based Insulet Corp., a maker of diabetes management products, said money and time constraints have put DR on the back burner. "We have a plan on paper. We have never really tested it," he said.
Insulet is considering three nearby locations for a disaster recovery site. But the company also has a far-flung sales team that does much of its work using Web-based application providers such as Salesforce.com Inc., giving the company some assurance of business continuity.
Steps to disaster recovery testing
Here are some of Toigo's suggestions for best test results:
1. Create an annualized testing plan. This is what you show management. State what you are you going to test, what kind of budget you need, which people are involved, how many times a year you're going to test (four), and so on.
2. Don't test all systems -- the reservations systems, the maintenance systems, etc. -- in one test event.
"Anytime you align one test event that has interdependent tasks, if any one test fails, it screws up the test," Toigo said. Instead, modularize procedures so you can test them independently of one another, thus making good use of test time.
3. Write a specific test plan. This is a detailed game plan for how you will perform a series of tests on a specific date. "This is the 'keep it simple, stupid' model that you learn in the military," he said. Specify task, mission and standards.
4. Don't conduct unannounced tests. Execution is everything, so that means no unannounced tests. "I don't believe in surprise tests," he said. "What do you learn from them?"
Let us know what you think about the story; email: Linda Tucci, Senior News Writer
