Case Study: Five more ways to screw up virtualisation
Don't try this at home...
By Robert Mitchell, Computerworld | Computerworld UK | Published: 01:00, 16 April 2007
The last "gotchas" instalment talked about correctly configuring virtual machines and having the right supporting cast of hardware resources associated with them. But other problems can crop up, too, especially as virtual server adoption builds.
Too much, too quickly
Once virtualisation takes hold, it can grow rapidly in many directions and become difficult to manage -- a phenomenon that Matt Dattilo, vice president and CIO at PerkinElmer, calls "VM creep."
"You have servers here, servers there, but there's no overall management or function in place. No one is keeping track of what is going on," says Bruno Janssens, senior architect of infrastructure architectural services at The Hartford.
As the number of virtual machines climbs from dozens to hundreds, or thousands, tracking what you have becomes challenging. "It's easy to lose track of what's out there in the virtual world," because there are no physical machines to see or serial numbers to record, says David Rossi, managing partner at Sapien LLC, an application service provider, that runs its operations on virtual servers. "The cataloguing and administration of that is one of the ‘gotchas,'" he says.
To help resolve that problem, Sapien developed specific process controls and tools to manage its virtual server population.
It's tough enough to track servers that are virtual. Live migration and dynamic workload-balancing tools that continuously move virtual machines between hosts can add to the confusion – and run afoul of change management policies. "If you have strict change management procedures in-house, you have to fine-tune" the dynamic resource scheduler, says George Scangas, lead IT infrastructure analyst at Welch's.
Virtual machines are essentially image files, and those need to be tracked and managed just like any other server image. As virtual servers become more prevalent, IT organisations need to have a change management database for quickly reproducing virtual server images, says IDC analyst John Humphreys.
A consistent library of virtual machine images can also improve efficiency.
"If you're repeating things or constantly developing new images and replacing them as opposed to fixing them in the virtual world, then you're not quite adopting the concept behind virtualisation," says Richard Cardona, principal engineer at Surgient, a vendor of virtualised application development and test lab services based in Austin. "Customers with less frequent updates are working smarter," he says.
Other issues can arise as the number of virtual servers climbs into the thousands. The Hartford's Janssens relies on VMware's VirtualCentre suite of tools to manage some 4,000 to 5,000 virtual machines but says he has already scaled beyond what that tool can effectively support today.
"If you go over 1,000 virtual instances, you will hit a limit with VMware products," says Janssens. However, a VMware representative responded that the limit in the latest version supports 1,500 virtual instances.
With nearly 5,000 virtual machines to manage, Janssens had to install multiple copies of VMware's VirtualCentre and manage them in 1,000 separate server silos. That limits his ability to manage effectively across all virtual machines because tools such as VMotion and Dynamic Resource Scheduler (DRS) won't work across VirtualCentre boundaries. Traditional enterprise system management tools such as HP's OpenView and IBM's Tivoli offer agents for virtual machines and can provide a unified view for monitoring all virtual machines. However, moving virtual machines between VirtualCentre silos remains a manual task, Janssens says.
Stumbling on security
In the rush to virtualise servers, it's easy to mix servers with different security requirements. "Do you want to consolidate a server in the DMZ with one behind it?" Today the answer is you don't, says IDC's Humphreys.
The DMZ is the no man’s land between the corporate firewall and the public Internet where Web servers and other public-facing services reside. So you wouldn’t want a Web server or database server in the DMZ residing on the same physical hardware as an internal server app such as a human resources database. Larry Rencken, vice president and CIO at Welch's, calls such configurations "reckless deployments." "We treat each virtual environment as though it was a physical box from a security perspective," he says.
Security for virtual machines is still evolving. "How do you apply virtualised security on a per-virtual-machine basis? How do you tie the security policies for a virtual machine so that when it is migrated to another physical server they are preserved?" Those issues still need to be worked through, Humphreys says.
Unexpected security glitches can also come up, says Rossi at Sapien. "When you virtualise one box and clone it and move it to another, sometimes the security identifiers might not come across successfully. We were using local accounts rather than Active Directory accounts, and those were getting hosed when we cloned boxes" he says. After Sapien migrated to Active Directory accounts, the problem went away.
But the biggest security management issue may be monitoring and keeping patches up to date for thousands of virtual machines.
One way to reduce the problem is to use virtual application environments such as Solaris Containers or SWSoft's Virtuozzo, rather than lower-level virtual machines. Those products sit on top of the operating system (Solaris and Windows, respectively) and don't include a copy of the operating system in each virtual session. That reduces the amount of patching required, but the approach does require that all applications be compatible with the operating system version running on the host.
Failure to sell the benefits
While virtualisation may sound like a good idea to IT organisations, the idea doesn't always sit well with users. The problem is, many organisations don't know how to sell the benefits to constituents. "They focus on 'I'm giving you part of a server,' and from a business owner's perspective, that's all they're getting," says Raghu Raghuram, vice president of product and solutions marketing at VMware "Educating as purely a resource utilisation benefit, that's not attractive enough for a lot of companies."
The major government transit agency in Montreal certainly initially encountered resistance from its business users, who were reluctant to try an unproven technology and needed much convincing. Users aren't the only ones who may voice concerns. Early on, "There was a lot of resistance both within the organisation and even with our own server operations staff," says Philip Borneman, assistant director of IT for the city of Charlotte, NC. Now, he says, the IT staff is 100 per cent behind the technology.
"If there's anything we've learned it's that you have to be very careful that the social environment is able to accept and adopt these tools in the way in which you've designed them," says Tom Goguen, vice president of Sun Solaris at Sun Microsystems
But users' wariness is understandable, says Janssens. "People are afraid of losing control of their physical environments, and secondly, they still don't trust [virtualisation] because they don't understand how it works," he says.
One way around the issue is to simply not mention the fact that servers are being virtualised. "For the most part, what we virtualised in recent years, the business just didn't know," says Welch's Rencken, adding that that's OK so long as the migration is transparent to the end user.
Bryan Peterson, principal systems engineer at The University of Utah Health Care, has debated how much his users need to know about underlying infrastructure. "We're not sure if we want to advertise to our customers that they're running on a virtual machine, [but] we don't deliberately hide it from people," he says.
When things go wrong, however, that strategy can backfire.
VMware's Raghuram thinks it's best to be up front with clients and answer the question of "What's in it for me?"
"In cases where we see the infrastructure people talking about the benefits – better availability, responding to application demands faster, we see them having more success," he says. That tack has worked for Peterson. "We've demonstrated that we can provide higher availability on a VM because they're on an ESX host," he says.
But some servers may not be candidates for virtualisation until it's time to upgrade the infrastructure, says Sun's Goguen. "If I have a mission-critical application I've been running successfully for years and it's all working, why would I take the risk of changing that environment?" IT needs to weigh the risks and rewards before moving the user's application, and the answer isn't always to virtualise those existing applications. "To think that you could turn the corner in virtualising everything in your environment may not be a realistic scenario," he says.
Falling into the departmental divide
Many departments have evolved to support their own applications over time. The economies of centralised virtualisation and server consolidation tend to cut horizontally across departmental applications fiefdoms. "When you virtualise you blur the lines between those roles and responsibilities," says Goguen, and that can result in turf battles. "There's a whole new set of policies, procedures and maybe even organisational structures that need to be worked out," he says.
Procedures may also be very different from department to department. "If one department has one set of standards and another department has another set of standards and they're going to start sharing infrastructure, are they going to fight together as opposed to work together?" asks Nick van der Zweep, director of virtualisation and integrity server software at HP. That's why many IT organisations are pushing for enterprise standards based on specifications such as the IT Infrastructure Library, he says.
Borneman faced those issues in Charlotte. "We've had to deal with the traditional silos between our departments, and getting people to share was a challenge," he says. What ultimately sold them was the cost savings, he says -- and the fact that in many cases the virtual machines for each department are clustered together on the same physical hardware.
The licensing trap
While virtualisation can substantially reduce the numbers of servers through consolidation, if you don't plan right, it can actually increase software licensing costs, users warn. For the city of Charlotte, "Licensing was probably the biggest ['gotcha'] in terms of hidden cost," Borneman says.
For the city of Charlotte, "Licensing was probably the biggest ['gotcha'] in terms of hidden cost," Borneman says. It's easy to see why. The virtual machines created by products like VMware's ESX Server and Microsoft's Virtual Server require a dedicated copy of the operating system for each virtual server created.
In contrast, virtualised application environments such as Sun's Solaris Containers or SWSoft's Virtuozzo sit on top of a single instance of the operating system and create virtual application partitions that don't require a separate operating system for each application. But that doesn't necessarily mean you'll pay for fewer operating system licences.
For example, Microsoft requires a licence for every Windows virtual server application created, whether or not the virtual environment includes full copy of Windows. But while Windows Server Standard Edition users pay a licence fee for every virtual machine or virtualised application environment created, the Enterprise Edition license includes up to four VMs per physical machine. Data Centre Edition users get unlimited VMs.
From a licensing standpoint, Borneman says the ability to provision a new VMware virtual server in less than 30 minutes was both a virtue and a curse. Once users discovered how fast virtual servers could be deployed, demand went up and the number of virtual machines skyrocketed. "It was too easy. A phone call and 30 minutes later they were up and running," he says. But each virtual machine also increased the number of Windows Server licences that the city needed. "Do that enough times and when you do your true-up you have a big bill," he says.
That's a common problem, says van der Zweep. "You were self-regulated before because you had to buy a server for every copy of the OS you installed. Now you just willy-nilly press a button virtually and create another OS instance. All of a sudden your licensing costs go up."
At the city of Charlotte, creating servers is still fast, but acquiring them is no longer free. Borneman tallies the software costs of issuing new virtual machines and charges the licensing fees back to the user.
In both cases application licensing can also run afoul of multi-processor hosts. "People think that with many virtualised instances you don't have to pay any more. That's not the case. It can even have the opposite effect," says Janssens. For example, application vendors such as Oracle licence software on a per-CPU basis. Depending on the terms of the software license, 10 applications running on 10 dual-processor servers might end up costing quite a bit less than deploying the same applications on 10 virtual servers running on a single, quad-processor machine, he says.
What you pay also depends on how you configure your virtual machines, says Goguen. For example, Sun has worked out an agreement with Oracle so that a Solaris Container that's isolated to two CPUs on an eight-processor system won't have to pay for an eight-CPU licence. Oracle wants to know that the machine can't dynamically reallocate more resources to the application on demand but must be taken down and reconfigured to use more processors, says van der Zweep.
"Oracle has a well-thought-out policy as to how to work in this type of environment. I can't say that every [software] vendor has done that work yet," says Goguen.