The CIO Watchdog
Along with contracts come even bigger challenges: control issues that range from security to support. For any CIO, loss of control over key parts of an environment has a nightmare-come-true quality to it. Failure to keep core competencies inside the IT organization can allow the outsourcer to hold your company hostage.
Meet the Terminator
Barry Brunetto will tell you straight up: He's not the easiest customer for vendors to handle. IBM learned that the hard way.
"I'm a Sicilian from New Jersey. My job is to basically watch out for the company assets," says Brunetto, vice president of information systems for Blount Inc., a manufacturer of outdoor products and industrial cutting tools in Portland, Ore.
Brunetto recently terminated his contract with IBM and its subsidiary Corio for service and support failures in an outsourcing deal involving the management of Blount's SAP system.
"We live in a touchy-feely world where you get a lot of people singing 'Kumbaya' about being partners with your vendors. So a lot of CIOs don't say anything when there's a problem," Brunetto says. "Well, I have a simple rule: I pay you for a service, and I expect one to be delivered."
"IBM doesn't comment on the specifics of customer engagements," IBM spokesman Jim Larkin responds. "But we can say that we're working closely with this client to address the concerns."
Brunetto's resolute philosophy sits well at a company like Blount, founded 60 years ago by a World War II pilot and his brother, who got their start building fish ponds and bridges in Alabama. Today, this $750-million concern employs more than 3,000 people at operations worldwide.
Three years ago, Blount executives decided to outsource management of a new global enterprise SAP system (along with its data center hosting) to a small applications service provider (Nexus Technology), which was later acquired by Corio. The production environment is hosted at an AT&T facility in Mesa, Ariz. "In our minds, hosting and managed services went together -- that's how the vendors want to sell it," Brunetto explains. "We wanted to pick a place that we felt would be a safe place, protecting it from a disaster. If we kept it in Portland, we could still be hit by earthquakes."
Early last year, IBM acquired the financially troubled Corio and folded its 300 employees into IBM's on-demand business unit. Services had already started to falter for Blount by that point, Brunetto recalls. When IBM took over Blount's contract, Brunetto was characteristically blunt with the new managers. "I warned IBM: 'We're going to open the gates of hell on you if you mess this up.'"
But mess it up they did. The real trouble hit last summer with the apparent mismanagement of the SAP application on a storage area network (SAN) at the AT&T facility. The worst of it hit during an 18-hour period one weekend last July during a planned shutdown to switch Blount's SAP application over to a new SAN. "They didn't have the SAN laid out correctly, and they didn't follow SAP's recommendations," Brunetto says.
"They changed our disk to low-speed disk to save money." Once the system move was completed on Sunday, Blount's IT group started letting people sign on from four manufacturing and distribution locations.
"We did not see the problem until Monday morning, when the typical load was placed on the environment," Brunetto recalls. Users were asked to limit use of the system to customer-related processes and critical business needs. In the meantime, his in-house technical team -- which fortunately included experts in SAP Basis and server technologies -- evaluated what was happening. "It seems that we were not getting all the facts from IBM, and it was evident that they did not spread the files properly across [the] spindles based on SAP specifications," Brunetto says. "I authorized them to bring down the system and install more disk spindles. Later that evening, they released the system back to us."
Now it was Tuesday morning, and system performance was worse than the day before. "I was getting nasty e-mail from around the globe," the VP recalls. "It was horrendous."
By now he was on the phone with account executives from SAP as well as his own contacts within IBM, who were trying to help behind the scenes. "Already we had started hearing excuses from IBM that sounded like the problem was not related to the SAN and that we needed to increase the number of servers, which of course meant more money," he says.
With yet another conference call pending later that Tuesday morning -- involving a widening circle of irate company officials -- Brunetto's team figured out that a particular switch in the SAN was probably set incorrectly. But they didn't have access to the production environment in Arizona to verify their suspicions.
"Before that last call, my team contacted the technical team at IBM to inquire about this setting. We never got a verification on what this setting was. But by the time we had the call, the system performance had improved drastically," he says. "So we really believed they had this set wrong. ... And from that point in time, we had a workable system. It wasn't optimal, but at least acceptable."
In the aftermath, Blount decided to stay with the AT&T facility but invested $850,000 for its own Dell and EMC hardware to install at the hosting site and replace the boxes owned by IBM. The company also hired one more SAP expert to join the IT staff in Oregon.
The data center hosting charge will be $88,000 a year now, versus the $505,000 annual cost of the hosting plus managed services piece. "If we need a new application server, it won't affect our monthly hosting fee," Brunetto says. "We can add one for about $3,000 instead of paying IBM $2,700 per month to add a new server to the contract. That's how they charge their management fee, so as you grow, that becomes a big ticket."
What was destroyed in the whole experience was the company's trust in IBM as a managed services provider. "The one key thing we all realized was that if we hadn't raised all holy hell, it just would have gone on," Brunetto says grimly. "And that could have really hurt our business."
So how much expertise do you need in-house to keep a proper eye on your vendor partners?
"If we had outsourced all the SAP environments like some of the vendors wanted us to do, there would have been no reason to have SAP Basis and technical support people in-house," says Brunetto. "If we did not have these people with the knowledge base looking out for the company, keeping IBM honest and focusing on the real problem ... this whole episode would have gone on for a much longer period of time and had a real impact on our business." Brunetto was able to prevent major business disruption when he grabbed the reins of his production application back.
"I don't think a lot of people, when they are looking to outsource all of their technical expertise, realize this," he adds. "I don't believe the outsourcing companies purposely try to harm their customers. But The Wall Street Journal and Fortune magazine are full of stories of stupid things that companies do today. My story shows that having technical expertise on staff can prevent major problems for the operations."
Keeping in-house expertise has been critical at Kelley Blue Book as well. The publisher (which is privately held and doesn't disclose revenues) chose to outsource only the "ping, power and pipe," says Tim Abbott, director of infrastructure. (Ping for sending a packet to the server to see if it responds, power for the electricity and pipe for the high-reliability Internet connection.) The hosting provider takes care of Kelley Blue Book's 90 servers at three data centers in Santa Clara and Irvine, Calif., and St. Paul, Minn. All applications remain within the company's control.
"We're in a period of rapid growth, and this is a time when we'd rather have our own hands on managing our applications and hardware for availability reasons," CIO Yaros explains. "I've gone through the process of implementing managed services. It's not something to take lightly."
When the first GM/Kelley Blue Book commercial aired, "Tim and his people were actively monitoring to see if the commercial had bogged down our servers," the CIO recalls. "We could see firsthand if it had an effect on the server loads. Tim was there to make sure there were no surprises. This makes us more comfortable."
Analysts and other experts also stress the importance of technical oversight of the outsourcing contract -- no matter how extensive the third-party arrangement may be.
"There are some other functions that you must retain. When you outsource, you also have to have a good contract manager," notes Gartner's Matlus. "Your data center manager knows what went right in the data center and what was happening, so someone should be [managing] the overall contract ... , looking at your [service-level agreement] monthly and validating that [the outsourcer] is delivering what is in the contract."
One Gartner client who had outsourced his wide-area network ended up paying much more than he did before the deal due to the lack of a technical person involved in oversight. "Accounts payable people were looking at the bill and just paying for it," Matlus recalls. "What the finance people didn't know is there was a charge on the bill for a $100,000 service that they were not getting."
This was first published in April 2006