ER Publications



A Performance Comparisons of Hypervisors: Review Paper Priyanka Mishra, Sheetal Dhir Department of Computer Science Birla Institute of Technology, Mesra Ranchi Jharkhand - 83521 INDIAAbstractVirtualization has rapidly attained mainstream status in enterprise IT by delivering transformative cost savings as well as increased operational efficiency, flexibility and IT service levels. Intel and AMD have independently developed virtualization extensions to the x86 architecture called hardware virtualization. This and other recent hardware advances such as multicore processors are further fueling the adoption of virtualization. While a full virtual service-oriented infrastructure is composed of a wide array of technologies that provide resource aggregation, management, availability and mobility, the foundational core of virtual infrastructure is the hypervisor. This paper provides a quantitative and qualitative comparison of virtualization hypervisors.IntroductionIT infrastructure virtualization, and in particular server virtualization, has become a primary consideration for all IT managers and CIOs because of the tremendous financial benefits that it delivers through reduced server count, power savings, automated management of data center resources, and because it simplifies the delivery of high availability, disaster recovery and resource optimization in the data center. Virtualization is the buzz of the enterprise IT market, and the upcoming IPO of VMware, the industry’s most successful virtualization solution vendor, is of great interest. Customers and analysts alike are wondering whether any competitive products can compete with the runaway success of VMware’s ESX Server hypervisor and Virtual Infrastructure 3 product. At the heart of the question lies a debate about the scalability and performance of XenSource’s open source Xen hypervisor, which is delivered to market by Xen Source in its Xen Server product family and also integrated in the operating systems (OS) products from Novell, Red Hat, Sun Microsystems and others. Since the future Microsoft Windows Hypervisor is also modeled closely on the product family using industry standard benchmarks for performance and Xen hypervisor architecture, one could reasonably assert that any flaw in the Xen architecture could negatively impact the virtualization products not only of XenSource, but also those of all major OS vendors. Indeed, identifying any such weakness would shore up VMware’s credibility with its customers. This paper studies the enterprise readiness of hypervisor-based virtualization products from VMware and XenSource. Using the award-winning VMware ESX Server hypervisor as a de-facto industry benchmark for performance and enterprise readiness, we present comparative results from an assessment of XenSource’s XenServer virtualization scalability.The Virtualization LandscapeVMware is the most successful vendor in the server virtualization market with its VMware Server and ESX Server products, challenged until recently only by Microsoft’s Virtual Server, which competes only with VMware Server [1]. These two products use the same “hosted virtualization” architecture, in which the virtualization software is run on top of an existing operating system such as Windows or Linux. Neither VMware Server nor Microsoft Virtual Server is considered useful for virtualization of production or performance sensitive workloads because the hosted model imposes severe performance overheads due to the need to emulate the virtualized hardware layer. As a result, neither of these products is considered in this study. Recently XenSource has become a focus of industry attention with its leadership of the open source Xen hypervisor project and its delivery of low cost commercial products for server virtualization that directly compete with VMware’s ESX Server hypervisor. The Xen hypervisor is widely regarded as an open source industry standard for virtualization, because many of the major enterprise systems and software vendors, such as AMD, Intel, IBM, HP, Sun, Fujitsu, NEC, Dell, Unisys, Red Hat, Novell, and VA Linux amongst others have contributed to the open source Xen code base, and because the open source Xen code base is packaged and is delivered to market in commercial products by several vendors, including Novell’s SLES 10 and in the forthcoming Red Hat RHEL 5 and Sun Solaris 10 updates. Xen also shares key architectural features with Microsoft’s promised Windows Hypervisor, which will be brought to market as a “plug in” to the forthcoming Windows Server “Longhorn”. XenSource itself packages the Xen hypervisor, with proprietary management tools and enhanced features to virtualize Windows virtual machines, in its Xen Server product family.Xen is unique in its ability to fully exploit the latest processor features for hardware assisted virtualization, namely Intel VT and AMD-V. Xen relies on these processor extensions to virtualize Windows and other legacy virtualized guest virtual machines, whereas ESX Server uses a technique called binary translation, which involves transparently executing a virtualization-safe instruction sequence instead of the instructions the guest operating system issues. Moreover, Xen pioneered a novel technique, called paravirtualization (called enlightenment by Microsoft) to enhance guest performance through co-operation between guest and hypervisor. Linux is open-sourced and therefore can be fully enlightened, whereas legacy Windows guests are closed source, restricting the enlightenment opportunities to the virtualized I/O path. Future Longhorn Server and Linux operating systems will all be enlightened, and consequently VMware has promised to support this new technology at some point in the future. Since many production workloads are performance sensitive, and because server consolidation is an exercise in efficient multiplexing of server hardware between many virtualized server instances, performance and scalability are of great importance to customers. So much so that VMware’s End User License Agreement (EULA) prohibits licensees from publishing performance benchmarks without VMware’s permission, arguing that poorly constructed benchmarks could damage its credibility with customers. The Xen project takes the opposite view, encouraging open performance testing as the best way to identify performance optimizations. With this background, and given VMware’s reluctance to enter into comparisons, it was with great surprise that XenSource found a comparison of performance between VMware’s ESX Server 3.0.1 and open source version 3.0.3 of the Xen hypervisor, openly published by VMware. Perhaps less surprising were the results – which painted the Xen hypervisor in a very poor light. In virtually every test that VMware ran, their implementation of Xen performed worse than ESX - indeed so badly that VMware concluded that Xen 3.0.3 was in no way ready for enterprise use. An inescapable implication of the VMware benchmark results is the further conclusion that the hardware accelerated virtualization features of Intel VT and AMD-V are significantly to blame for the poor performance of Xen 3.0.3, since Xen relies heavily on these features to virtualize Windows. This paper presents corrected results for the same performance benchmarks as published by VMware, comparing XenSource’s XenEnterprise 3.2 commercial product which is based on Xen 3.0.4 and is packaged with XenSource’s enhancements for virtualized Windows guests, with the commercially licensed ESX 3.0.1. The results differ remarkably from those published by VMware. Suffice it to say that XenEnterprise performs just as well as ESX 3.0.1, and in many cases better. In a few tests it performs less well than ESX, and we have highlighted these as key points for improvement in later releases.Hypervisor Virtualization ApproachesOS Virtualization is achieved by inserting a layer of system software – often called the hypervisor - between the guest OS and the underlying hardware. This layer is responsible for allowing multiple OS guest VMs to share the resources of a single server. Each guest believes that it has the resources of an entire machine under its control, but beneath its feet, the hypervisor transparently ensures that physical resources are securely partitioned between different guests. The hypervisor manages all hardware structures, such as the MMU, I/O devices, DMA controllers and the like, and presents a virtualized abstraction of those resources to each guest. Two main approaches have emerged: emulation and binary translation, used by VMware, and paravirtualization, pioneered on the x86 architecture by Xen, and modeled after techniques used on IBM Mainframes.Emulation and Binary TranslationThere are several ways to achieve the appearance of real hardware for the guest. The most direct method is to completely emulate the underlying hardware platform in software, including the processor's instruction set architecture. For the x86, the privileged instructions – used exclusively by the OS (for the likes of interrupt handling, reading and writing to devices and virtual memory), form the class of instructions requiring emulation. By definition, these instructions cannot be executed by a user program. One technique to force emulation of these instructions is to execute all of the code within a virtual machine, including the OS being virtualized, as user code. The resident VMM then handles the exception produced by the attempt to execute a privileged instruction, and performs the desired action on behalf of the OS. This technique is known on the x86 architecture as “ring de-privileging”.While some CPUs were carefully architected with virtualization in mind, the x86 architecture evolved from earlier designs, which did not anticipate this. Providing full virtualization of a processor in such cases is challenging, often resulting in so called virtualization holes. For example, certain instructions execute in both user mode and supervisor mode, but produce different results, depending on the execution mode. A common approach to overcome these problems is to scan the OS code and invisibly replace the offending instruction sequences, either to produce the intended behavior or to force a trap into the hypervisor. This technique, called “binary translation” is used extensively by VMware ESX Server. It suffers the obvious disadvantage of the overhead of scanning, translating and managing traps from the virtualized guest.ParavirtualizationAn alternative way to achieve virtualization is to present a virtual machine abstraction that is similar, but not identical to the underlying hardware. This approach is called paravirtualization, and was introduced to the x86 architecture in the Xen hypervisor. In lieu of a direct software emulation of the underlying hardware architecture, the virtualized guests collaborate with the hypervisor to achieve optimal performance. Paravirtualization offers significantly improved performance, but it does require modifications to a guest operating system. Unlike binary translation which modifies the instructions actually executed on behalf of the guest “on the fly”, paravirtualization assumes OS Vendor participation – the guest is modified at load time to include paravirtualization extensions. The OS vendor must include the appropriate virtualization “hooks” in their product but derives from tremendous performance and security benefits as a result. For this reason all x86 OSVs are adopting paravirtualization for their next major OS release. In the case of Linux and Solaris this is achieved through the inclusion of the Xen hypervisor, and in the case of Microsoft through the use of so-called “enlightenments” for the forthcoming Windows Hypervisor.Paravirtualization, together with the complementary innovations of processor extensions for virtualization eliminate the need for binary translation, and offer the opportunity for near-native performance of virtualized guests that scales with Moore’s law. It is for this reason (presumably) that Microsoft has seized on the Xen architecture as its role model for the forthcoming Windows Hypervisor.In the subsections that follow, we discuss the Xen architecture in more detail.CPU & Memory VirtualizationIn Xen, virtualization of CPU and memory, and low level hardware interrupts is provided by a tiny, low level efficient hypervisor layer that is implemented in about 50,000 lines of code. When the OS updates hardware data structures, such as the page table, or initiates a DMA operation, it collaborates with the hypervisor by making calls into an API that is offered by the hypervisor. This, in turn, allows the hypervisor to keep track of all changes made by the OS, and to optimally decide how to manage the state of hardware data structures on context switches. This API is called the hypercall API, and is becoming a standard feature in future releases of Linux, Windows and Solaris. The Xen hypervisor therefore has an optimized way to virtualize these guests. Of course, these OS versions will be able to run both native (non-virtualized) and on the hypervisor, since the modifications being made to Linux, Windows and Solaris allow the hypercall API to be dynamically bound to the guest when it is loaded for execution.I/O VirtualizationThe Xen hypervisor exposes a set of clean and simple device abstractions, and a set of drivers for all hardware on the physical platform are implemented in a special guest (virtual machine) outside the core hypervisor that offers virtualized I/O to other guests. This is called “Domain 0” in Xen, and the “Root Partition” in the Windows Hypervisor. This architecture allows existing device drivers for the Domain 0 OS to simply be re-used for virtualization – an extremely powerful feature. VMware ESX Server is more like a microkernel than a true hypervisor, and contains special device drivers ported to its kernel. It therefore has more restricted hardware certification than typical implementations of Xen.In each Xen guest simple “front-end” device drivers replace hardware-specific drivers for the physical platform. These drivers are independent of all physical hardware, and represent each type of device (e.g. block I/O, Ethernet, frame buffer). This abstraction enables high-performance, virtualization-safe I/O to be accomplished by transferring control of the I/O to Domain 0, with no additional complexity in the guest OS. Importantly, the drivers in Domain 0 run outside the base hypervisor, at a lower level of protection than the core of the hypervisor itself. The hypervisor is thus protected from bugs and crashes in device drivers.Xen uses a generalized, shared-memory ring-based I/O communication primitive between guest “front end” and platform “back end” drivers that aim to achieve high throughput by batching requests. It also affords an innovative means to add features into the I/O path, by plumbing in additional modules between the guest virtual device and the real device driver, such as packet filtering, and intrusion detection.Hardware Support for VirtualizationRecent innovations in hardware, particularly in CPU, MMU and memory components (notably the Intel VT-x and AMD-V architectures), provide some direct platform-level architectural support for OS virtualization. Xen provides a common HVM (Hardware Virtual Machine) abstraction to hide the minor differences between the Intel and AMD technologies and their implementations.ESX Server does not currently leverage these technologies for virtualization of 32-bit guests. HVM offers two key features: First, for unmodified guests, it avoids the need to trap and emulate privileged instructions by enabling guests to run at their native privilege levels, while providing a hardware vector (called a VM EXIT) into the hypervisor whenever the guest executes a privileged instruction that would unsafely modify the machine state. The hypervisor begins execution with the full state of the guest available to it and can rapidly decide how best to deal with the cause of the VM EXIT. Hardware performance of VM EXIT operations has increased by a factor of 3 in the past year alone.A second feature of the HVM processors is that they offer paravirtualized guests the VM CALL instruction that calls directly into the hypervisor. This can be used by certain drivers to ensure that guest I/O takes the fastest path into the hypervisor’s I/O stack.Assessing Hypervisor Performance and ScalabilityVMware ESX Server 3.0.1 is widely used in both in test/development labs as well as production environments. The performance of ESX 3.0.1 has improved considerably by comparison with ESX 2.5 (we cannot publish results to confirm this), and VMware can rightly claim that ESX deserves consideration (despite its price) for any enterprise virtualization project. But can the same be said of the Xen hypervisor? This is a legitimate question, and one that it has been difficult to address without a head-to-head comparison of ESX 3.0.1 and XenSource’s Xen Server 3.2 using standardized benchmarks.Recently VMware published a report that used standard benchmarks to show that a VMware implementation of Xen 3.0.3 performed poorly compared to ESX 3.0.1. The tests, using a Windows Server 2003 as the guest OS, exercise the newest feature set in Xen that uses Intel VT and AMD-V hardware support for Windows virtualization – a feature set that has not had the benefit of 8 years of optimization, as the VMware code base has. Moreover, the version of Xen used in the tests (3.0.3) is understood in the Xen development community to be targeted primarily at Linux virtualization, and though partial HVM feature set support is present, it is not considered enterprise class. Additionally, VMware’s implementation and configuration of the Xen hypervisor could not be checked, and it is likely that inadvertent incorrect configuration by the testers influenced their results. It should be stressed that Xen 3.0.3 is a code base, and not a commercial product, and so it is difficult to understand what VMware was trying to achieve. Finally, the description of the test setup for the VMware benchmarks is incomplete, making it difficult to understand how some results are generated (for example their network performance appears to exceed the bandwidth capacity of a GigE NIC).As a result, the VMware results should be viewed as being of little value. They relate to a VMware specific implementation of a developing open source code base – and not to any vendor’s product. It is surprising that VMware did not use XenSource’s commercial product for its comparisons, since XenSource invests heavily in the optimization, testing, and certification that customers expect of an enterprise product.Even though we regard the VMware study rather meaningless, the question VMware posed, namely “is the Xen hypervisor ready for production use?” is valid in the specific context of a vendor’s commercial product. We therefore re-phrase the question thus: Does XenEnterprise perform poorly in comparison to ESX? Can it scale to serve enterprise users? The remainder of this paper attempts to answer these questions, fairly and openly.Test Methodology and System ConfigurationThis paper was designed to replicate the testing done by VMware and as a result the methodology and system configuration matches as close as possible to the setup used by VMware. In a number of cases the VMware paper did not provide all the details of their test setup which made it more difficult to replicate their setup. We have provided more details about the test setup so that others may reproduce this same testing in their own environments.Guest Operating SystemWe used an off-the-shelf copy of Microsoft Windows Server 2003 for these tests to match the configuration used by VMware. Since high performance virtualization is important to all guests, we have also provided data from relevant benchmarks for (para) virtualized Red Hat Enterprise Linux 4. The ability to run a mix of Windows and Linux operating systems is a core requirement for any enterprise virtualization platform.Test WorkloadsThe test workloads for this paper are the same as those used in VMware’s performance assessment. Although they do not represent typical workloads offered to a hypervisor, they are nonetheless well known benchmarks used for evaluating system performance, and include the following:The integer tests from the SPECcpu2000 benchmark suite, were used to represent CPU intensive applications,The Passmark set of benchmarks tests the major subsystems of an operating system. It is used to represent typical system workloads,Netperf is a network specific benchmark that was used to simulate network transmit and receive performance.The java application server benchmark SPECjbb2005 was used to represent an application server and its workload, andA compile-build test from the SPECcpu2000 INT package was used to assess performance for development workloads.Hardware ConfigurationServer SystemsThe system used to run the benchmarks was an IBM x3550 server with two VT enabled dual-core 2GHz Intel Xeon 5130 CPUs for a total of four CPU cores. The x3550 server is a rack mount version of the x3500 server that VMware used in their comparison. The only difference between the two systems is the speed of the Xeon processor and the speed of the SAS drives. VMware used a system with 2 3GHz processors while our test system has two 2GHz processors. Additionally our SAS drives were 10K RPM drives while they used 15K RPM drives. As a result we have rerun all tests on this platform so an accurate comparison can be made.Memory ConfigurationWhile the system had 7GB of RAM installed the system was booted for native tests with access to only 1GB of RAM to match the memory limits in the virtual work ConfigurationThe IBM x3550 server had an onboard 1GigE network interface which was used for testing. During the network specific testing two clients were connected to this server using a Linksys SR2016 Gigabit Ethernet switch. The two clients used during the network testing were running Windows 2003 R2. The two client systems were Dell 1950s each with onboard 1GigE network interfaces.Storage ConfigurationThe IBM test system was configured with two 73GB SAS disk drives running at 10K rpm. No RAID setup was used during the testing process.Software ConfigurationWe used XenEnterprise 3.2 (public beta) and ESX Server 3.0.1 GA releases.Virtual Machine ConfigurationEach VM was configured identically, with a single virtual CPU and 1GB of memory. For SPECjbb2005, we increased the memory limit to 1.6GB to match the VMware configuration. In addition, for SPECjbb2005 we used more than one virtual CPU. We used Windows Server 2003 Enterprise Edition R2 (32 bit) as the guest OS, to match the VMware configuration.As with the VMware paper no attempt was made to optimize the benchmark results in any way. The virtualization platform and the guests were installed with default settings. Windows guests had the VMware VM Tools or XenSource’s Xen Tools, as appropriate, for maximum I/O performance [1].For each benchmark pass, only a single virtual machine was powered on. Though this does not measure the ability of the hypervisor to provide resource guarantees to multiple guests, it is precisely the same as the test configuration used by VMware.SPECjbb2005 Java Virtual MachineSPECjbb2005 requires the use of a Java Virtual Machine. Like VMware, we used the BEA Systems JRocket 5.0 Java Virtual machine, configured in exactly the same way. The configuration options are listed below:-Xms960m –Xmx960m –Xgc: parallel –XXaggressive:opt–XXcompactratio:8 XXminblocksize:16kResultsThis section provides the detailed results for each of the benchmarks run. All of the results have been normalized to native performance measures, to match the result presentation in the VMware paper. Native performance is normalized at 1.0 and the various benchmark results are shown relative to that number. Thus, a benchmark result of 95% of the native performance would be shown as 0.95 on the normalized scale. Unless otherwise noted, the higher the number the better the performance of the particular virtualization platform.SPECcpu2000 IntegerThe SPECcpu2000 benchmark focuses on compute intensive performance [6]. The tests consist principally of user mode computational tasks, and include such workloads as file compression and object-oriented database transactions.The results show nearly identical performance for both systems, with ESX 3.0.1 having an average overhead of 2%relative to the same benchmark run native, whereas XenEnterprise shows an average overhead of about 3%. ESX has a slight lead in the gcc compiler sub-benchmark which accounts for the largest difference in performance. This is a known area for improvement for XenEnterprise. We believe that page caching in the hypervisor may be one reason why ESX performs well here – of course Xen does not do this because it is a class 1 hypervisor, which brings additional benefits of enhanced security, smaller code footprint and more robust partitioning of guests.PassmarkFigure 4 shows the benchmark results for Passmark CPU tests. The following CPU mark tests were run: Integer Math, Floating Point Math, SSE/3DNow, Compression, Encryption, Image Rotate, and String Sorting [4]. As user-mode tasks, these CPU performance benchmarks typically don’t incur high levels of overhead when virtualized.As expected, XenEnterprise and ESX perform equally well – in all but one of the benchmarks the two systems are within 0.5% of each other. For the more complex benchmarks (Compression, Encryption, Image Rotation, and String Sorts) the two products also produce essentially the same results.The only sub-benchmark where ESX has a very slight lead is the SSE/3DNow benchmark, where ESX measured 3.5% faster. This benchmark tests multimedia performance, which is not a workload that is ever run in a virtualized context, even in a hosted desktop infrastructure. As a result it has little relevance to actual use ESX or Xen Enterprise.From the SpecCPU2000 and Passmark results it is clear that both virtualization platforms are adept at handling CPU intensive workloads. This includes a wide variety of system configurations including database and application servers, as well as infrastructure and e-mail servers. The differences between the two platforms are minimal.Figure 5 shows the Passmark benchmark memory results. The Memory Mark tests include the Allocate Small Block, Read Cached, Read Uncached, and Write sub-benchmarks.Like VMware, we find that both virtualization platforms offer near native performance. However, whereas VMware claims that ESX leads in these tests, we find that XenEnterprise 3.2 actually delivers higher levels of memory performance than ESX 3.0.1. On average XenEnterprise shows 2% overhead vs. Native while ESX shows 4% overhead vs. pile WorkloadsFigure 6 shows results for the SPECcpu2000 INT build benchmark test. To match the VMware setup the same Microsoft Visual C++ 2005 Express Edition compiler with the Microsoft PSDK for Windows Server 2003 R2 was used. VMware claimed a considerable difference between ESX and Xen performance, and concluded (incorrectly) that (Intel VT) hardware-assisted virtualization support in the CPU was to blame. While ESX does outperform XenEnterprise slightly in this benchmark, the primary cause of the slowdown is the considerable amount of disk I/O in the benchmark, for reading source code from disk and writing compiled executable code back to the disk. VMware tested an instance of Xen 3.0.3 without XenSource’s Xen Tools for Windows that are part of the XenEnterprise product, that paravirtualize the Windows I/O path and make optimal use of Intel VT. With these installed, ESX is only about 6% faster on the build benchmark than XenEnterprise – which has never been optimized for this workload. We are confident that we can easily eliminate this difference in performance between the two products – a difference in execution run time of only 17 perfThe following Netperf benchmark involved using one or two client systems to generate network load on a target machine. The tests were done using the Netperf TCP_STREAM test [3]. As with the VMware testing the Message Size was set to 8192 bytes and the Socket Size was set to 65,536 bytes.Figure 7 shows the Netperf results for both one and two clients. The send and receive performance is presented separately. Both ESX and XenEnterprise exhibit very similar performance on the network receive tests in both the single client and dual client cases. There is a slight difference in performance on the network transmission tests between XenEnterprise and ESX. Xen Enterprise lags ESX by only 4% to 8% on the send benchmark.VMware reported that the Xen 3.0.3 performance was lower than 10% of native speeds for both send & receive. While this may have been the case for the VMware implementation of Xen 3.0.3, it is certainly not the case for XenEnterprise, which also has the benefit of XenSource Tools for Windows. XenEnterprise network performance is actually on par with ESX in all but one case, where it lags very slightly [2].Additional Notes on Netperf Test ConfigurationBased on the configuration data provided in the VMware paper, the assignment of network interfaces to guests was unclear to us. As a result, we made assumptions about the guest NIC configuration. We ran our tests with a single Ethernet adapter and port. Our results show a slight throughput increase when using two clients vs. one client in all three cases. XenEnterprise is perfectly capable of saturating a Gigabit Ethernet NIC, and we were reassured to find that our throughput is bounded above by the capacity of the single NIC card! (We have subsequently been advised by VMware that the two-client case was run with one NIC port assigned to each client. We will publish results for this configuration in the near future.)SPECjbb2005The SPECjbb2005 benchmark is used to test Java Application Server performance. It simulates a server side Java Application Server running on a Java Virtual Machine. This benchmark does not include any network I/O and disk I/O is limited to writing minimal log files during the benchmark runs. Java application servers are generally highly threaded workloads that can take advantage of having access to multiple CPUs on a system.VMware omitted results for the Xen because they could not launch a Windows guest VM with multiple virtual CPUs – a documented and well-known limitation of the open source Xen 3.0.3 HVM feature set. Fortunately, XenEnterprise 3.2 has full support for multiple virtual CPUs for HVM guests, including Windows. Below we provide data for runs with a single vCPU, two vCPUs, and four vCPUs. XenEnterprise’s support for and performance scalability for multiple vCPUs means that VMware’s claims that Xen lacks scalability should be disregarded.As expected the SPECjbb2005 performance for a single vCPU VM on both ESX and XenEnterprise are both excellent and within only 0.5% of the native score. Figure 9 shows that XenEnterprise outperforms ESX for the two-vCPU test with close to a 1.5% lead, but ESX is slightly ahead on the four-vCPU test with a lead of just under 1%. We conclude that both platforms perform these tests very well, with basically equivalent performance.Additional Notes on the SPECjbb2005 Test RunDuring our initial round of SPECjbb2005 testing we found that the results for the test when run native using 2 processors were worse than the results for both ESX and XenEnterprise. We found that when booting Windows natively with a two-way SMP configuration on the two-socket dual-core systems it would use both cores on the first socket rather than using both sockets. In the virtualized tests a VM would use one core from each socket, performing far better than when both vCPUs are constrained to run on the same socket. Once we re-ran the two-vCPU tests with this configuration, the results were as expected.SPECjbb2005 on LinuxThe SPECjbb2005 benchmark is used to test the Java Application Server performance on a RHEL 4.4 using the Sun’s Java Runtime Environment Version 5.0 Update 11. As mentioned earlier, Java application servers are generally highly threaded workloads that can take advantage of having access to multiple CPUs on a system.Figure 11 shows that XenEnterprise outperforms ESX for the 2 vCPU test by a margin of 13%. In fact, XenEnterprise is only 2% slower than native RHEL 4.4, whereas ESX is 15% slower than native. On the four-vCPU test, XenEnterprise performs even better, being 24% better than ESX. Compared to native RHEL 4.4, the overhead for XenEnterprise is 4% whereas the overhead for ESX is 23%.Additional Notes on the SPECjbb2005 on Linux Test RunAs with the Windows Server 2003 SPECjbb2005 runs, the RHEL 4.4 guests on both XenEnterprise and ESX were constrained to run on the same CPU parative Scalability of ESX and XenEnterpriseVMware repeatedly states that they were unable to run multiple virtual machines on the Xen platform. We have no idea why they struggled in this way, but customers routinely run large numbers of virtual machines on our commercial product XenEnterprise. To address the assertion that Xen is unable to support an enterprise workload, we have included some initial benchmark data with multiple virtual machines running on both ESX and Xen Enterprise.Our test uses for the Passmark CPU benchmark in a configuration where multiple VMs are loaded on the machine. Three identical virtual machines were launched on each platform. These virtual machines did not run any extra software. Each was left to settle for 5 minutes after booting to ensure that the three additional VMs were only using minimal system resources during the testing on both platforms. This gives us a way to identify how well each platform partitions server resources between different VMs. We note that the generalized case of multiple VMs competing for resources is complex, and warrants its own study – something we intend to produce in due course. We note also the VMware effort to have their methodology for multiple VMs on a single platform enshrined in the VMMark benchmark by the SPEC organization. Our purpose here is limited to rejecting VMware’s claims.Figure 10 below shows that the performance for both ESX and XenEnterprise is almost identical when loaded with multiple virtual machines. In fact both platforms show only 0.5% average differences in scores from the benchmarks run with a single virtual machine.Discussion of ResultsThe objective of this evaluation was to evaluate the performance and scalability of VMware ESX Server versus XenSource’s commercial XenEnterprise product. Our added aim was to identify whether XenEnterprise is suitable for deployment in the enterprise. Our results show convincingly that:1. XenEnterprise performs as well as ESX in most benchmark tests, a remarkable achievement for a first product.2. XenEnterprise scales as well as ESX, if not better.The exercise has been useful – the few benchmarks on which XenEnterprise lags ESX have given us valuable pointers for how to tune the next release of the product.Single Virtual CPU TestsFor both hypervisors we found that the majority of cases performance for both platforms was very similar. In some cases such as the Passmark memory tests the XenEnterprise platform provided higher levels of performance than ESX. In other cases as with compilation tasks the ESX platform provided a higher level of performance.The conclusions in this paper, based on our commercial product, are radically different from those published by VMware for their own implementation of open source Xen 3.0.3.Virtual SMP TestsThe SPECjbb2005 scalability testing showed that XenEnterprise and ESX have similar virtual CPU scaling results. In fact on the 2 VCPU case XenEnterprise outperforms ESX. These tests show that the XenEnterprise platform is able to provide the same ability to handle production loads from highly threaded applications that take advantage of multiple CPUs.Scalability of the PlatformOur initial scalability testing with running multiple virtual machines on the hypervisor platforms showed that as with the single virtual machine test the two platforms show very similar levels of performance. These tests show that XenEnterprise and ESX are able to scale well as additional virtual machines are added to the system.VMware claimed that they were unable to run multiple virtual machines. XenEnterprise is routinely used to execute workloads with large numbers of virtual machines. XenSource focuses specifically on conferring the enterprise requirements for virtualization on its packaged XenEnterprise product, of which scalability is a key attribute.ConclusionsVMware appears to have failed to appreciate the difference between our Xen open source code base and our commercial XenServer products. For example, had they read the release notes for Xen 3.0.3, they would quickly have established that Intel VT was only partially supported in that release. Moreover, XenSource’s Xen Tools for Windows, which optimize the I/O path, were not installed. The VMware benchmarks should thus be disregarded in their entirety.Our performance results show convincingly that XenEnterprise 3.2 performs equally well or better than VMware ESX Server 3.0.1 in all but a couple of tests. Given that we have not spent much time on optimizing our product for traditional benchmarks we are pleased to see that there is essentially no difference between the two products. Our tests highlighted a couple of areas in which ESX marginally outperforms XenEnterprise, namely on compile time, and for Netperf TX. XenEnterprise outperforms ESX on the Passmark memory operations. XenEnterprise scales well to utilize multiple CPUs to allow a wide variety of workloads to be run. Additionally the scalability testing found that XenEnterprise provides similar scalability to ESX when additional virtual machines are added to the platform.VMware also appears to have under-appreciated the performance advantages of the latest generation of Intel and AMD processors with built-in support for virtualization. Although VMware’s legacy technology cannot exploit these features, XenEnterprise’s high performance is proof that these technologies deliver tremendous price/performance advantages to end users. What’s more, the performance of these CPU features will increase further over time, with Moore’s Law.In conclusion, XenSource’s commercial products are reliable, affordable and offer the Windows or Linux IT Professional a high performance platform for server consolidation for production workloads. They are very easy to use, and offer open APIs that enable our ISV ecosystem partners to complement our products to deliver a range of solutions that in every way eclipses the expensive, single-vendor, closed “solutions” marketed by VMware.References[1] Adams K. and Agesen O. A Comparison of Software and Hardware Techniques for x86 Virtualization. ASPLOS October 2006. [2] Barnum P., Dragovic B., Fraser K., Hand S., Harris T., Ho A., Nuegebauer R., Pratt I., and Warfield A. Xen and the Art of Virtualization. Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, January 2003 [3] Netperf. [4] Passmark. [5] Popek, G. J., and Goldberg, R. P. Formal requirements for virtualizable third generation architectures. Commun. ACM 17, 7 (1974), 412–421. [6] Standard Performance Evaluation Corporation (SPEC). [7] Standard Performance Evaluation Corporation (SPEC). [8] University of Cambridge Computer Laboratory [9] Virtual Iron Virtualization Blog. [10] A.L. Ankudinov et al., “Parallel Calculation of Electron Multiple Scattering Using Lanczos Algorithms,” Physical Review B, vol.65, no. 10,2002,pp. 104107-104118.[11] E. Walker, “Benchmarking Amazon EC2 for High-Performance Scientific computing,” login; vol. 33, no. 5, 2008, pp.18-23.[12] F.Sullivan,”Cloud Computing for the Sciences,” special issue , Computing in Science & Eng., vol. 11 , no.4, 2009,pp. 10-11.[13] Cherkasova L, Gupta D and Vahdat A 2007. Comparison of the three CPU schedulers in Xen. SIGMETRICS Perform. Eval. Rev. 35 pp 42-51 doi:10.1145/1330555.1330556 [14] The Kernel Virtual Machine [15] Henning J L 2000 SPEC CPU2000: measuring CPU performance in the New Millennium Computer, 33, 7, pp 28-35 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download