Ecole des Mines de Nantes. Journée Thématique Emergente "aspects énergétiques du calcul" - PDF

Description
Ecole des Mines de Nantes Entropy Journée Thématique Emergente aspects énergétiques du calcul Fabien Hermenier, Adrien Lèbre, Jean Marc Menaud Outline Motivation Entropy project

Please download to get full document.

View again

of 26
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information
Category:

Short Stories

Publish on:

Views: 8 | Pages: 26

Extension: PDF | Download: 0

Share
Transcript
Ecole des Mines de Nantes Entropy Journée Thématique Emergente aspects énergétiques du calcul Fabien Hermenier, Adrien Lèbre, Jean Marc Menaud Outline Motivation Entropy project Dynamic consolidation principle Reconfiguration problem Some results Extension to HPC Cluster Context Switch Conclusion 2 Motivation DataCenter/Cluster environment Static allocation of the resources to the jobs Resources are underused static allocation of resources vs. dynamic utilization - Data center are oversized For a PUE = 2 Air C. 50 % 50 % Servers AC/DC CPU 55 % Fan 45 % Memory Idle 80 % Run 20 % Disk Data center Servers CPU 3 Motivation Dynamic Consolidation Each task of a job is embedded into a Virtual Machine (VM) The resources are allocated depending on the needs VMs are mixed to be hosted on a reduced number of nodes VM must be always online Servers unused can be turned off VMs are remixed when it is necessary, without downtime But remixed VMs take time! Packing the VMs implies several migrations Some migrations has to be delayed to succeed. Temporary hosting is necessary... - Performance degradation Reactivity is a key factor 4 Outline Motivation Entropy project Dynamic consolidation principle Reconfiguration problem Some results Extension to HPC Cluster Context Switch Conclusion 5 Dynamic consolidation Entropy observes the current CPU, memory and network requirements of each VM and computes a globally optimized placement of them that satisfy all their requirements while using a minimum number of hosts. Entropy can be cataloged as an IaaS system 6 Global Design A Configuration : Each VM is assigned on a node, Each VM requires a fix amount of memory. Each VM requires a variable amount of CPU. (Simplification : VMs executing a computation are active and require their own CPU.) - May be viable 7 Using Live Migrations at Cluster Scale The Virtual Machines Packing Problem (VMPP) Compute the minimum number of nodes to use to have a viable configuration 8 In Action ) 4 servers 4 Tasks ( % CPU time Without Entropy Server n 3 stopped With Entropy 4 Tasks, 3 or 4 Servers Consumption is reduced by 25% 9 Outline Motivation Entropy project Dynamic consolidation principle Reconfiguration problem Some results Extension to HPC Cluster Context Switch Conclusion 10 Order VM Operations (1/2) Current Status Correct Status Non-viable manipulations 11 Order VM Operations (2/2) 2 steps 12 Migration Interdependences One additional node is required (critical energy consumption) 3 steps 13 Optimizing the reconfiguration process Determine an efficient reconfiguration plan (thanks to a cost function) Cost model : the necessary steps before migrating a VM the amount of memory to migrate the parallelism inside a single step Cost = 4 Cost = 9 2 steps 3 steps 14 Architecture overview Entropy is a virtual machine (VM) manager for clusters and acts as an infinite control loop, which performs a globally optimized dynamic VM placement without downtime according to cluster resource usage and scheduler objectives Compute a viable configuration using a minimum number of nodes Plan and reduce the migration process if migrations are necessary Extract the current configuration : The position of each VMs and their states (active or inactive) Migrations orders are sent to the concerned hypervisors 15 Outline Motivation Entropy project Dynamic consolidation principle Reconfiguration problem Some results Extension to HPC Cluster Context Switch Conclusion 16 The interest of the dynamic consolidation is limited by the duration of the reconfiguration process. Entropy computes equivalent configurations with cheap reconfiguration plans until the minimum What are benefits? Better reactivity A stable packing A reduced overhead From 14 to 6 minutes Always better Reduced by 9% 17 Outline Motivation Entropy project Dynamic consolidation principle Reconfiguration problem Some results Extension to HPC Cluster Context Switch Conclusion 18 Dynamic consolidation Servers unused by online applications (web, HA etc.) can be : Turned off OR Can be used by preemptive applications (simulation HPC etc...) The main problem How can i improve my cluster by running a maximum of preemptive applications 19 Entropy Advanced Processors! Running! 1st! job in the! queue! 2nd! 2nd! job 2nd! 3rd job! 4th job! Jobs arrive in the queue and have to be scheduled. Time! Processors! 2nd! job! Running! 3rd job! 1st! job in the! queue! 4th job! FCFS + Easy backfilling Jobs 2 and 3 have been backfilled. Some resources are unused (dark areas) Time! Processors! 2nd! job! Running! 3rd job! 1st! 4th! job in the! queue! job! Time! Easy backfilling with preemption The 4th job can be started without impacting the first one. A small piece of resources is still unused. consolidation and preemption to finely exploit distributed resources 20 General idea: manipulate vjobs instead of jobs In a similar way of usual processes, each vjob is in a particular state: A cluster-wide context switch (a set of VM context switches) enables to efficiently rebalance the cluster according to the: scheduler objectives / available resources / waiting vjobs queue (elasticity) [VTDC 2010] 21 Reconfiguration plan 22 Experiment on a cluster Benefits improve resources usage suspend/resume transparent for the developer Resources usage 23 Outline Motivation Entropy project Dynamic consolidation principle Reconfiguration problem Some results Extension to HPC Cluster Context Switch Conclusion 24 Conclusion Manipulate VMs is tedious and may be non cost-effective Entropy manage VMs instead of process Provides a efficient and reactive dynamic consolidation policy and a generic cluster-wide context switch based on mechanisms provided by VMM http ://entropy.gforge.inria.fr LGPL ANR Arpège SelfXL ( ) ANR Arpège MyCloud ( ) FUI Cool-IT ( ) ANR Emergence Entropy ( ) Uses an Abstract VMM (Jasmine-VMM) ESX, Hyper-V, Xen, KVM... 25 J.M. Menaud - Juin
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks