Enable vEGM on a guest

To provide a virtual machine (VM) with contiguous system memory backed by host Extended GPU Memory (EGM) character devices, enable Virtual EGM (vEGM) for the VM. This improves performance for workloads with large GPU memory footprints.

Before you begin

Procedure

  1. Open the XML configuration of your VM for editing: 
    # virsh edit <vm_name>
  2. Add or modify the <memoryBacking> section to use shared file allocation: 
    <memoryBacking>
  <source type='file'/>
  <access mode='shared'/>
  <allocation mode='immediate'/>
</memoryBacking>
  3. Locate the <hostdev> section for your assigned GPU, and add an alias name so it can be referenced by the memory device, for example: 
    <hostdev mode='subsystem' type='pci' managed='no'>
  …
  <alias name='ua-hostdev0'/>
</hostdev>
  4. Add a <memory> device with the egm model. Replace the following values with your environment-specific settings:
    • <path> — the host EGM character device, for example, /dev/egm4
    • <size> — the total EGM size as reported in the host sysfs
    • <pciDev> — the alias name of the GPU host device

      For example:

      <memory model='egm' access='shared'>
  <source>
    <path>/dev/egm4</path>
  </source>
  <target>
    <size unit='MiB'>56896</size>
    <node>0</node>
    <pciDev>ua-hostdev0</pciDev>
  </target>
</memory>
  5. Save the configuration, and exit the editor.
  6. Start the VM: 
    # virsh start <vm_name>

Results

  • In the guest operating system, verify that EGM is enabled: 
    $ nvidia-smi -q | grep -i enabled | grep EGM

    If the output returns multiple lines confirming EGM : enabled, the configuration is successful.

  • If the VM starts but QEMU attempts to create an excessively large file in the host file system, the specified EGM character device is missing. To detect this condition, monitor host disk usage while the VM is starting: 
    # df -h / && ls -lh /var/lib/libvirt/qemu/memory/

    If a multi-gigabyte file is growing in the memory backing directory, shut down the VM immediately and verify that EGM is properly enabled in the host firmware and that the nvgrace_egm module is loaded.

What to do next

  • Configure the guest operating system