Memory and CPU hotplug on Ubuntu Guests and VMWare ESXi hosts (including vCloud)

Introduction

With some cloud projects in mind and some virtual hardware estimations I was curious to see if it's possible to hot plug virtual memory and CPU to Ubuntu guests. Here is what I discovered.

• Infrastructure: vCloud Director on Acens.com (VMWare ESXi)
• Guest: Ubuntu server 10.04 (kernel 2.6.32-45-server #100-Ubuntu SMP Wed Nov 14 11:02:27 UTC 2012 x86_64 GNU/Linux)
• virtual HW version: 7
• vCPU: 1
• vMemory: 1 GB

Hot add virtual CPUs

Check current CPUs:

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping : 1
cpu MHz  : 2266.747
cache size : 12288 KB
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat
bogomips : 4533.49
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Now, add the virtual CPUs directly from the vCloud control panel (from the hardware tab of the VM properties). Initially, these new CPUs are offline. To active each new CPU we have to:

echo 1 > /sys/devices/system/cpu/cpuX/online

where X is the number of the new CPU.

Check that new CPUs are online:

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping : 1
cpu MHz  : 2266.747
cache size : 12288 KB
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat
bogomips : 4533.49
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping : 1
cpu MHz  : 2266.747
cache size : 12288 KB
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat
bogomips : 4533.49
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

You can use the script provided in this article:

http://communities.vmware.com/docs/DOC-10493

This enables automatically all new vCPUs.

Hot add virtual memory

This operations is analogue to one described above, but the kernel module that detects the new memory is not automatically loaded (at least in my Ubuntu install), so we have to do it manually:

# modprobe acpi_memhotplug

Let's check current memory first:

# free -m
             total       used       free     shared    buffers     cached
Mem:           997        112        885          0         10         42
-/+ buffers/cache:         59        937
Swap:         2015          0       2015

Add the memory through vCloud (or VMWare vSphere Infrastructure Cliente - VCLI). Although the kernel will have detected the new memory, it's still not available. We have to enable it in the same way we did it before with virtual CPUs:

echo 1 > /sys/devices/system/memory/memoryX/online

where X is the number of the new memory block.

After the memory blocks have been enabled, we can check the available memory:

# free -m
             total       used       free     shared    buffers     cached
Mem:          1893        141       1752          0         10         42
-/+ buffers/cache:         88       1805
Swap:         2015          0       2015

As with CPU hot plug, you can use the script provided in this article:

http://communities.vmware.com/docs/DOC-10492

This enables automatically all new memory blocks.

Epilogue

It's easy to add new resources on the fly but it may not be that easy to remove them.