Disaster recovery plans solve bare-metal recovery problem via VMware

By Linda Tucci, Senior News Writer 23 Jun 2009 | SearchCIO-Midmarket.com

Technology news and tips for midmarket CIOs Digg This!     StumbleUpon     Del.icio.us

DR revamp at a glance Company: North Carolina Farm Bureau Mutual Insurance Employees: 650 at home office Amount of business-critical data: 12 terabytes Data center before: Included mainframes, 70 Windows servers backed up to tape. Data center after: Mainframes, five Windows servers using VMware, replication to disk arrays. RTO/RPO (mainframe): From 24 hours to 15 minutes for each. RTO/RPO (Windows farm): From essentially unrecoverable to 36 hours/15 minutes. What's next: The company doesn't yet use virtualization on production servers, but it is working hard to do so using VMware for Windows and z/VM and Linux for WebSphere and distributed DB2.

It is also the story of an IT executive in an industry so accustomed to managing risk (insurance) that he was able to get approval for an expenditure that was "a bit shocking" (approximately $900,000) by laying out what IT could and could not do and leaving it to the business to calculate the cost.

North Carolina Farm Bureau Mutual Insurance Company Inc. is a midsized business with an oversized need for speedy and reliable disaster recovery processes. Based in Raleigh, the property and casualty insurer operates in all 100 counties of the Tar Heel state. It has 650 employees at its home office, another 2,100 people spread out over the counties and direct written premiums of just less than $900 million a year.

"Our goal is to cover business-critical functions within 48 hours of a disaster declaration," said Steve Zeidman, information systems division manager.

Cash flow is king at insurance companies, Zeidman said. The ability to mail bills, to verify coverage for claims processors, to take loss notices from customers and to pay claims relies on digital processes. The IT infrastructure is multi-tiered: a mainframe and a large Windows farm with data centralized at the home office. For years, the disaster recovery plans involved backing up and recovering data from tapes, using Tivoli Storage Manager. Then about two years ago, the insurer began to reassess those capabilities because recovery times were too slow.

Bare metal recovery problems, tape too time-consuming

With its time-tested backup procedures, the mainframe was a known quantity. IT could confidently assure the business that it would be back up within 24 hours of an outage, with 24 hours' worth of data lost. The 70 Windows servers were another story, Zeidman recalls.

"We found out the server farm was essentially unrecoverable. It would have taken weeks to recover it, if we had continued along with the methods we were using," he said.

Bare-metal recovery was a problem, Zeidman said, referring to the process of rebuilding a computer after a catastrophic failure. Some configurations require that the hardware configuration used for restoring the data be identical to the hardware configuration used for the backup -- a steep climb even when the servers are the same make.

"We could never get the hardware to match close enough without running into driver issues, which would cause the server to 'blue screen,'" he said.

In addition, recovering from tape proved far too time-consuming. "Due to tape mounts and unmounts, we found restores were consuming more than five hours per server. This was the point where we determined we absolutely needed data to be resident at the recovery center," Zeidman said.

Enter server virtualization, replication

IT decided to use virtualization and data replication to make that happen.

Thanks to physical-to-virtual conversion, the insurer's disaster recovery plans now involve five cold IBM servers running VMware, located at the firm's disaster recovery (DR) center 500 miles away in Sterling, N.Y. (The company initially tried to turn its IBM blade servers into virtual servers but found the blade technology "was not suited to that," Zeidman said, so it replaced them with the large IBM servers.) An OC-3 circuit connects headquarters to disk arrays that store data replicated via IBM Global Mirror and NetApp SnapMirror at the Sterling site, replacing backup tapes for the 12 terabytes of business-critical data.

On the mainframe, both the recovery point objective (RPO) and recovery time objective (RTO) have dropped from 24 hours to 15 minutes.

As for the recalcitrant server farm, since the VMware implementation in 2007, Zeidman's team has gotten recovery time down from 70 hours to 36 hours and is confident it can shave more hours off when it tests again next month. The RPO is 15 minutes.

Expert advice on virtualization and DR

Companies looking to server virtualization for their disaster recovery plans need to realize that quicker RTOs require more than just virtualization, one analyst cautioned. "Server virtualization by itself won't buy a user much if the production data and virtual system image are still being backed up to tape," said Gartner Inc. analyst John Morency, via email. "The real-time savings comes from a combination of using disk-to-disk replication for most, if not all, of the production data, along with a standardized approach -- e.g., use of VMware Site Recovery Manager -- for virtual machine backup and failover (this approach also using disk-to-disk)."

Richard Jones, service director for Midvale, Utah-based Burton Group Inc.'s data center strategies group, said companies like one recent client find time savings from virtualization in other areas besides recovery. "To their surprise, their DR testing time -- they test every six months -- dropped by more than 50%, and the number of IT staff required to perform the test also decreased by nearly half," he said.

Gap analysis lays foundation for business decision

Speedy and reliable DR, no surprise, does not come cheap. Zeidman said the cost of his company's solution came to 0.1% of its direct written premiums, leaving the reader to do the math ($900,000). Zeidman did not do a cost analysis or a business impact analysis for the solution.

More DR strategy resources Virtualization as a disaster recovery strategy? New offerings that may cut the cost of your IT disaster recovery plan

"My input was purely from an operational level. I presented management with my opinion of what we could do and could not do in a disaster recovery and what parts of the business would be missing. I told them the cost of getting to the point of where we needed to be and they accepted," Zeidman said.

"The initial cost was a bit shocking, but they understood the benefit," he added.

Business is not always so accommodating. A recent Harris poll of 220 IT managers and 277 business line leaders for SunGard Availability Services shows a significant disconnect between IT and business executives when it comes to disaster recovery preparedness. While both groups overwhelmingly agree that information availability is important to a business' success, 74% of IT managers believe DR and business continuity are important to business success, vs. 49% of business executives who think so.

"We have very smart management in insurance. We buy reinsurance, for example, to make sure we can pay any catastrophic claim," Zeidman said. "They saw this as another layer of reinsurance."

Tags: Disaster recovery planning for the midmarket,  Virtualization for the midmarket,  Data storage for the midmarket,  Tools and technology: Keep your information secure,  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us

<< PREVIOUS | NEXT >>

VIEW ALL IN THIS CATEGORY

RELATED CONTENT Disaster recovery planning for the midmarket 2009 IT Geek Halloween costume ideas Virtualization management strategies ezine for CIOs Free risk management tools and resources for the enterprise At your peril, disaster recovery testing gets short shrift Covering the people side of a disaster recovery program Disaster recovery and business continuity planning templates Enterprise content management a player in disaster recovery program Managed IT services for disaster recovery and business continuity These disaster recovery services for SMBs include an on-site trailer Disaster recovery services options for smaller businesses on a budget Virtualization for the midmarket Virtualization management strategies ezine for CIOs Data storage technology: Know your FAQs and options Midmarket data center management guides: Tips and best practices 10 tips for renegotiating your virtualization licensing contracts Management tools for virtualized servers: A look at the options Virtual server management vs. physical servers: What's the difference? Virtualization technologies: A quiz on license best practices and more Virtualization technology use spreading into desktops and storage Virtualization technology 2.0: Current and future trends for IT Are SaaS and server virtualization helping you turn the budget corner? Data storage for the midmarket Data storage technology: Know your FAQs and options Midmarket data center management guides: Tips and best practices What do you know about data center outsourcing? Virtualized storage and SANs drive disaster recovery plan The price of data center outsourcing: Security, costs and more explored Data center virtualization: User best practices Firm moves from tape backup to managed backup and recovery service IRobot CIO dishes on virtualization, disaster recovery and compliance Leading iRobot's IT: Virtualization, disaster recovery and compliance Midmarket IT budgets hit by economic downturn

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary