Introducing TPC-VMS: a new benchmark suite for virtualized databases
Server virtualization has become mainstream and has changed the way resources are provisioned and accessed within the data center. (Did you know the number of virtual machine shipments now exceeds the number of physical servers being shipped?). Effective measurement and characterization of complex applications in virtual environments is critical to both vendors and customers.
The Transaction Processing Performance Council (TPC) today announced a new industry standard benchmark suite, TPC-VMS (Virtual Measurement of Single-system), that enables comparison of performance, price-performance and energy efficiency of database applications in a virtualized environment.
The benchmark suite is built upon existing TPC standards: TPC-C, TPC-E, TPC-H and TPC-DS. The benchmark test sponsor chooses one of these workloads, and runs three equally sized instances of the same workload on three virtual machines on the system under test. The primary performance metric is the slowest of the three instances and is reported as VMStpmC (for TPC-C), VMStpsE (for TPC-E), VMSQphH@Size (for TPC-H) or VMSQphDS@Size (for TPC-DS).
(1) TPC-VMS Press Release
(2) TPC-VMS Specification
(3) W. Smith, Characterizing Cloud Performance with TPC Benchmarks, LNCS vol. 7755, Springer 2013
(4) P. Sethuraman, R. Taheri, TPC-V: A Benchmark for Evaluating the Performance of Database Applications in Virtual Environments, LNCS vol 6417, Springer 2010