VM Bulletin 2007

[Mar 2, 2007] Five tips for getting started with server virtualization

You need to calculate saving to avoid spending more money then with an old solution. Also large servers are really expensive so hardware saving might be illusionary. You should always consider blades as an alternative.

The big day arrives -- after months of planning and preparation, the time comes to actually move physical to virtual. Consider some migration tools to help you. Microsoft is releasing a tool in the near future that will allow you to move a fully installed server running a supported version of Windows to a virtual hard disk format, which is fully supported by its Virtual Server product. VMware has a similar tool in the works. These migration utilities can save you hours, if not days, of performing the actual move.

Other things to consider:

• Take advantage of clustering capabilities. Using high-performance clusters gives your virtual machines higher availability while also improving their performance.
  
• Think about management. How will your staff manage the virtual machine collection you will have? What scripting languages and APIs do your virtual server software support? Are you able to access certain controls via the command line for simple remote access-based administration?
  
• Don’t forget about storage. You’ll need a very fast disk subsystem to get maximum performance for your virtualized servers. Typically, you’ll find the biggest bang for the buck is in iSCSI-based disk offerings. They are reasonably priced but have great configurability, and such products are fast, too.

As with any ongoing process, it’s important to keep tabs on the virtualization project as you begin transitioning users and services to the new platform. Establish some performance and usage guidelines and thresholds, and evaluate what these metrics will mean for future tweaks and enhancements. Consider tweaking hardware configurations, network setups or increasing bandwidth as needed. Your job isn’t over once the final boot into the virtualized operating system is finished.

[Mar 2, 2007] The Virtualization Procrastinators What's The Holdup

Fear Factor

Shahri Moin, IT director at Oscient Pharmaceuticals Corp. in Waltham, Mass., is testing Microsoft Virtual Server 2005. So far, so good, he says, but Moin has reservations about upsetting the status quo. “Putting it in production scares the daylights out of me,” he acknowledges. And since he has just a few dozen servers to manage, the most common motivation for adopting virtualization — consolidation — isn’t a big concern. “It’s not going to allow me in a meaningful way to reduce staff or operating costs,” Moin says.

PerkinElmer first started using virtual machines in 2005 in a project designed to address space, power and cooling problems in its Boston data center by consolidating physical servers. Jeff Brittain, IT director for the city of Hickory, N.C., found another way to achieve a similar goal. He tested Microsoft Virtual Server 2005 but decided to migrate 40 rack-mounted servers to IBM blade servers instead. “That is accomplishing the consolidation we were looking at,” he says. With no pressing reason to go ahead with Virtual Server, he says he’ll revisit the technology “down the road.”

John Nordin, CIO at Insurance Auto Auctions Inc. in Westchester, Ill., has been testing VMware, but he says he doesn’t trust the technology enough to use it on the 130 servers that run the company’s auction business.

During an auction, a car is sold every 40 seconds, and many bids come in electronically. With 750,000 auctions a year, Nordin says he can’t afford problems. “This is super-mission-critical stuff. When it isn’t there, we can see it on the bottom line,” he says.

In early testing of VMware, a virtual server inexplicably reverted back to a prior configuration. That corrupted the system, which had to be restored from tape. Nordin says explanations from his vendors, VMware and Microsoft, haven’t been forthcoming. “I got a lot of the classic multivendor finger-pointing,” he says. “Nobody has been able to give me a root cause, and this thing is never seeing production until I know why this happened.” Even if those answers come, Nordin says, he’ll start out slowly by using the technology only on his print servers.

Bob Holstein, CIO at National Public Radio Inc., isn’t worried about the reliability of VMware ESX Server, which he calls “rock-solid and production-worthy.” He’s testing the product now and plans to do a small rollout on production servers later this year. If all goes according to plan, application servers with low utilization rates will be consolidated and then moved to a collocation facility to make room for more servers that handle live digital audio feeds for NPR’s radio programs. Those servers will stay on physical hardware, however. “Those vendors are not going to support a virtual environment, and they’re too mission-critical to take that risk,” Holstein says.

“I have a ‘show me’ attitude about [VMware] right now,” Holstein says, adding that staffers need to get plenty of hands-on experience with the technology before moving to a production environment.

Nordin agrees. “People who haven’t worked in any partitioned environment shouldn’t underestimate what their system engineers need to learn,” he says. “Make sure you put the training dollars in.”

“Virtualization is kind of a leap of faith,” says Mike French, senior network engineer at Perkin�Elmer. He has spent time explaining the technology to his peers. “It’s a tough thing to break the barrier, but if you build the environment rock-solid with redundancy and safeguards, nobody should ever have a problem.”

Although the technology has been in use for several years, it’s still common to find applications that aren’t supported on virtual machines. “A lot of vendors won’t certify applications as VMware-compatible,” Dattilo says. “In most cases, we assumed the risk, unless it was a mission-critical application.”

A VMware spokesman says that the problem has diminished. Indeed, a few of Dattilo’s vendors, such as Hyperion Solutions Corp. and Business Objects SA, have begun supporting virtual machines since he started working with the technology. As for the others, Dattilo says that most software vendors’ support organizations will still work with his staff on problems, but he hasn’t had any so far.

Nordin says the fact that some software vendors still don’t support applications on virtual servers is evidence that the market still isn’t fully mature.

“Those types of issues have been long resolved in the MVS, VM and Unix space,” he says, adding that server virtualization products “need to get going.” With Red Hat, SUSE and Microsoft embedding hypervisors into the Linux and Windows operating systems, however, application vendors will have little choice but to support it, analysts say.

Software vendors aren’t the only ones who’ve been slow to support their products running in virtual servers. Jon Elsasser, CIO at The Timken Co. in Canton, Ohio, says IT staff resistance to deploying homegrown applications on virtual machines has stopped some projects. “Some internal application-support personnel are a bit leery of it,” Elsasser says, but he expects attitudes to change over time.

Slow Uptake

Even companies that have embraced virtual server technology have limited its penetration into the data center. Those that have adopted virtualization have, on average, only about 20% of their environment virtualized, according to IDC analyst John Humphreys.

After a pilot last year, Timken went on to virtualize 35% of its servers. It now has 125 virtual servers running on six quad-processor physical machines, but Elsasser has no plans to expand beyond that. The 100 Windows and SQL servers supporting a new SAP ERP software implementation are off-limits to virtualization, he says. “At this point, we’re just glad it’s running,” Elsasser says.

At PerkinElmer, Dattilo’s goal is to have 52% of servers virtualized by the time the current project there is completed. That includes application servers with relatively low utilization levels, but others, such as Exchange Server mailbox nodes, are staying put.

Payton plans to roll out virtualization at Case Design/Remodeling this fall. “We’ll stay away from Exchange and SQL Server” and focus on low-utilization applications like domain controllers and file- and print-sharing servers, he says. Most users aren’t ready to consider virtualization for important applications, says IDC analyst Steven Elliot. “The really mission-critical stuff is further down the line,” he says.

Tools for managing virtual machines are still evolving, and their availability is “still a little light,” says Dattilo. However, he adds that tools included with the recently introduced VMware Infrastructure 3 Enterprise Edition have solved some of his problems. Currently, every virtual machine on a physical server needs its own instance of Backup Exec. VMware Consolidated Backup eliminates that problem.

Distributing loads across virtual machines — a time-consuming, manual process today — can be automated using Distributed Resource Scheduler. Payton has been testing with the previous version of VMware and says, “We’re running into CPU utilization problems because the limit is set statically.” He and Dattilo both plan to migrate to the new version.

With VMware so far ahead of the competition, there’s little pressure on pricing today. “Software licensing costs are a little high, and their maintenance is out of this world,” says Dattilo. Application software licensing on virtual machines is also in flux. “There’s a lot of confusion among application vendors as to how those will be licensed,” he says, noting he doesn’t want to pay a per-proc�essor premium for running on a quad-processor machine when an application is running in a virtual machine and using just a fraction of those resources.

Analysts expect the adoption curve to accelerate this year as users become more comfortable with virtualization technology. Says Elliot, “2006 is the year of production for large enterprises.”

But that doesn’t mean everyone should rush ahead. “If you don’t have a clear benefit from virtualization today, you can wait,” says Reynolds. That said, most companies will find at least some immediate benefits, whether from consolidation or reduced server configuration and deployment costs. Elsasser says server procurement time savings made his project worthwhile. “It used to take two weeks to deploy a new server,” he says. “Now we can do it in two days, and in an emergency, we can do it in an hour.”

A compromise strategy is to focus on “high-value production deployments” but put off broader implementations, says Reynolds. The next version of Windows Server will offer technology to compete with VMware in 18 months or so, and Linux distributions with virtualization technology will be here even sooner. With those competitive pressures, “VMware could become significantly less expensive,” Reynolds says.

[Jan 09, 2007] Linux.com Virtualization begins to materialize in the Linux kernel by Michael Stutz

Virtualization, the ability to run multiple "virtual" systems on a single host, is nearly as old as solid state computing. But it's been all the rage again in recent years, with more than a dozen large-scale Linux virtualization projects -- and while they require special custom kernels, patches, and software, the first steps toward OS-level virtualization have been implemented in the Linux kernel.

"To me, what we see now on the virtualization front is a natural evolution, rather than a revolution," says Kir Kolyshkin, project manager for OpenVZ, an open source virtualization effort supported by SWsoft. "We went from single-user to multi-user, from single task to multitasking. Now we go from single OS to multiple OSes, from single instance to multiple isolated VMs or VEs."

Virtualization has many applications, including testbeds and virtual server hosting. "The core motivation for most users is that a single machine now has more power than they can use for a single application," says Eric W. Biederman, a kernel hacker who works on system software and clusters for the high-performance supercomputing market. "Therefore it makes sense to consolidate servers on a single machine."

"Newer machines are 95% idle when used for 'typical' setups," agrees Herbert Poetzl, project leader of the Linux-VServer project. According to Poetzl, virtualization is also important for security. "Isolating services without adding unnecessary overhead and/or blocking out wanted communications is a big advantage," he says. "Lightweight isolation allows you to save a lot of resources by sharing, and increases security for service isolation."

Biederman says that virtualization also opens up the possibility for migration of running programs. "[I]f you can take everything with you that is global [such as a system's IP addresses and PIDs], and use those same global identifiers after you migrate, the problem becomes tractable without application modifications."

Two types of virtualization

Biederman explains that there are actually two separate and non-conflicting approaches to virtualization that are being addressed for the Linux kernel. One is to use a hypervisor, which handles calls to hardware allowing a system to run multiple (and different) operating systems at the same time in virtual servers; with paravirtualization (as with Xen), the kernel or OS is modified so that it knows that it's running on a virtual machine.

The other approach is to give the kernel the ability to run multiple instances of itself in separate, secure "containers" on the host system by virtualizing the namespace of the kernel.

"The core idea is that several objects that the kernel manages have global identifiers, and by making those identifiers non-global and doing a context-sensitive lookup, you can have the feel of multiple UNIX instances without the overhead," says Biederman. "Essentially this is building a super chroot facility."

A team of many players

OS-level virtualization is a major feature; implementing it in the Linux kernel is a large undertaking and isn't going to happen casually. In fact, several developers in the Linux virtualization arena, including Biederman, Linux-VServer, OpenVZ, and IBM are contributing to the effort. IBM's contribution should be particularly useful, since the company was a pioneer in the development of virtualization, and has more than 40 years of experience with the technology. IBM says that its recent innovations being integrated into the Linux kernel "represent some of the most significant development initiatives underway at IBM's Linux Technology Center."

Biederman says that the first thing everybody had to do was learn how to work together. What helped, he says, was working on first implementing "generally non-controversial" features. "Plus, this cemented the incremental approach," he adds, noting that both Linux-VServer and OpenVZ have already announced that they're using these new features in their software.

To help things get going, some of the key developers met at the Linux Kernel Developers Summit in Ottawa, Canada this past summer.

"Face-to-face meetings are always helpful, and I think we need more," says Kolyshkin, who suggests a "virtualization track" at next year's Kernel Summit. He says that what they've accomplished so far is only the beginning.

"I think it's too early to say we have achieved something big," he says. "In fact, there are just a few building blocks, a few bricks that came in."

The first few bricks

All of these first virtualization building blocks were proposed, discussed and worked on throughout 2006, and were in Linux kernel developer Andrew Morton's merge tree for a while; now, having made it to a stable 2.6 kernel (2.6.19, released November 29), they're officially a part of mainstream Linux.

"Strictly speaking, [these patchsets] do not change the current behavior of the Linux kernel," says Poetzl, "but they pave the road to OS-level virtualization."

IPC virtualization

The IPC virtualization patch virtualizes Linux Inter-Process Communication (IPC), which is the ability of processes to communicate and share data with each other. With this patch, processes can only see and communicate with those processes in the same virtual container.

"Traditionally, there is a single set of IPC objects (shared memory segments, message queues, and semaphores) per a running kernel," explains Kolyshkin. "Since we want to create multiple isolated environments -- containers -- on top of a single kernel, we do not want those containers to see each other's IPC objects. Thus the need to virtualize IPC -- or, in other words, create IPC namespaces."

PID virtualization

PID (Process ID) virtualization allows PID space to be unique to each container so that inside a container no processes outside of it can be seen.

On a UNIX system, the init process always has a PID of 1, explains Kolyshkin. "Since with multiple containers there are multiple inits, all those inits also should have PID=1. Thus the need for PID namespaces, or PID virtualization."

Kolyshkin adds that this isn't the only time PID virtualization can be handy. "Consider the live migration scenario when you migrate a set of processes from one machine to another," he says. "Since there is no way to change the PID of the task, you have to have all the same PIDs on a destination machine. Without separate PID namespaces, you cannot guarantee that."

UTS virtualization

The UTS namespace patch virtualizes the utsname structure, which gives basic information about the operating system and hardware, as well as the hostname; with this patch, utsname is local to each container -- a necessity for virtualization.

"We cannot live with the hostname being the same for all containers," says Kolyshkin.

According to IBM, the purpose of this patchset isn't merely the functionality itself, but that it helps lay the groundwork for virtualizing other system resources in the same way. Towards that end, the patchset introduced the necessary structures and code and provided an example of how to use them.

"IBM contributed it to 'get the ball rolling' on development for application containers, which are needed both for virtual server and application migration functionality," the company said in a statement. "In developing the patch, IBM also helped to get all the parties who were working to develop such functionality privately, together. This both accelerates development of the features and increases the code quality by ensuring that all parties get the right to object to bad design or bad code."

But on the paravirtualization side, things have been advancing, too. "The truth is that the paravirtualization support has been trickling in for a long time," says Biederman.

Patches to begin implementing paravirtualization have been merged into the development kernels; another patch merged into the development tree is the Kernel-based Virtual Machine (KVM). It creates a /dev/kvm device that allows the system to run virtual machines.

Morton says that he expects both KVM and the base paravirtualization to be included in Linux 2.6.20. "OS-level virtualization continues to trickle in," says Morton. "I don't know when [or] if it will be complete."

What's next

Meanwhile, all of the developers are steadily working on making it happen. "My personal plan is to finish up the PID namespace before I move forward with network namespace," says Biederman. "In some senses that is the critical namespace, because once you start separating out processes it really begins to feel like a separate system."

IBM is continuing to develop full-featured Linux virtualization at three levels -- full OS, virtual servers, and lightweight application sets. Kolyshkin predicts that the next two big advances will be more resource management work, and a checkpointing and live migration capability. While it will be some time before all these pieces are in mainstream Linux, these careful steps show that it's materializing.

"As for the general idea of how to achieve it -- bit by bit, patch by patch, with peer review, suggestions and improvements from everybody," says Kolyshkin. "And this is how we do that. Slowly, but moving forward."