Sunday, January 8, 2023

https://www.nakivo.com/blog/how-to-use-usb-devices-with-a-vmware-vsphere-virtual-machine/

 https://www.nakivo.com/blog/how-to-use-usb-devices-with-a-vmware-vsphere-virtual-machine/


How to Use USB Devices with a VMware vSphere Virtual Machine

When using virtual machines, the principle of hardware virtualization is used and hypervisor software emulates devices used by VMs on the software level. Virtual controllers, disks, DVD-ROM, video card, network cards, input devices, and other devices are usually created in a virtual machine. Sometimes you may need to connect a physical device to a virtual machine directly. This device can be a printer, scanner, USB flash drive, USB hard disk drive, USB smart card reader, security dongles, USB license keys, etc. If your external physical device has the USB interface, you can attach this device to a virtual machine running on VMware Workstation and ESXi server. VMware allows you to connect USB devices to virtual machines by using the USB passthrough feature (passing the device through). This blog post explains how to use this feature on ESXi hosts in VMware vSphere.

What Is USB Passthrough?

VMware USB device passthrough is a feature that allows you to connect a physical USB device attached to a USB port of a physical ESXi server to a virtual machine as if the USB device was connected to a virtual machine directly. VMware passthrough is a feature available starting with ESXi 4.1.

Reasons vary for using passthrough for a USB device from the ESXi server to a guest operating system (OS) of a virtual machine. For example, you have an old scanner or printer that doesn’t have drivers for the latest Windows OS versions. You can install the old Windows version supported by the USB device and connect this device to a virtual machine. Another use case of this feature is when you need to copy files from/to a USB hard disk drive to/from a virtual machine directly with high speed. It may happen that users need to use software that is protected by hardware USB license keys on virtual machines.

How USB Passthrough Works

There are three main components required for USB passthrough functioning: the arbitrator, USB controller, and physical USB device.

The arbitrator is a dedicated service on an ESXi server. The arbitrator scans physical USB devices attached to the physical server and is responsible for USB device traffic routing between a physical host and a virtual machine. Only one VM can access the USB device at a time. Access to the USB device connected to one VM is blocked for other VMs. The USB arbitrator supports up to 15 USB controllers (vSphere 7.0). The Arbitrator service is enabled on ESXi by default.

A USB controller used on a physical ESXi server is the controller to which physical USB ports are connected. The physical USB controller works with physical USB devices inserted into the physical USB ports of the server. USB controllers can be onboard (they are also called integrated and are built-in on a chip in a motherboard) or external (implemented as PCI or PCIe cards that are inserted into the appropriate slots on a motherboard of a server). The onboard USB controller acts as a bridge between a USB bus and a PCIe (or PCI) bus. The onboard USB controller is physically connected by the appropriate number of PCIe lanes to a chipset on a motherboard.

A virtual USB controller is an emulated device created especially for a virtual machine running on a hypervisor. A virtual USB controller interacts with a physical USB controller when you passthrough a physical USB device from a physical ESXi server to a guest OS on the VM. Presence of a virtual USB controller in VM configuration is required to use the USB passthrough feature on the VM. A maximum of eight virtual USB controllers can be connected to a VM on an ESXi host. If you need to remove a virtual USB controller from a VM configuration, you have to remove all USB devices connected to that controller first. VMware USB driver is installed in a guest OS after adding a virtual USB controller to VM configuration.

A physical USB device is any USB device that has a USB interface and USB port for being connected to a computer. Maximum 20 USB devices can be connected to a single VM or ESXi host.

USB standards and controller modes are:

  • USB 2.0 and USB 1.1 (EHCI+UHCI)
  • USB 3.0 (xHCI) VM hardware version 8 or higher

After connecting a physical USB device to a virtual machine by using the VMware USB passthrough feature, you may need to install drivers for the connected USB device in a guest operating system.

Requirements for USB 3.0

USB 3.0 is available for ESXi servers starting from VMware vSphere 5.5 patch 3. The xHCI USB controller is required in VM configuration for this purpose. OS requirements to use USB 3.0 passthrough are: Windows 8.1 or higher, Windows Server 2012 R2 or higher, or Linux with Linux kernel 2.6.35 or higher. USB 3.1 devices are supported on their maximum speed since VMware vSphere 7.0

Limitations for USB Passthrough

There are some limitations you should know about before starting to use the USB passthrough feature:

  • A USB drive cannot be used as a boot drive for a VM.
  • If a USB drive is used to boot ESXi, this USB drive cannot be attached to a VM.
  • Virtual hardware version 7 or higher is required for a VM.

Some USB devices can be not supported by VMware:

  • USB hubs
  • Audio devices and video cameras that use asynchronous data transfer
  • Some USB keyboards and mice (you can find details in the VMware 1021345 article)
  • Some smart cards using CCID may need additional tuning for connection to a VM

vMotion is possible for VMs with connected USB passthrough devices. After proper VM configuration and enabling vMotion, USB devices remain connected to the VM after migration to another ESXi host.

There are vMotion limitations for VMs that use USB passthrough disks:

  • Distributed Power Management (DPM) is not supported because an ESXi host cannot be shut down for power saving without disconnecting a USB device from a VM (a USB device is disconnected if the host is powered off by DPM). DPM should be disabled on this host.
  • Fault Tolerance is not supported.
  • A VM cannot be suspended or powered off (otherwise USB devices are disconnected from a VM and they can be reconnected only after migration of the VM to the initial ESXi server to which USB devices are physically connected). The VM must be manually migrated to the ESXi host with the attached USB device when you need to power on this VM. Linux guests that were resumed can mount USB devices to different mount points after reconnection.
  • ESXi hosts that have VMs with USB passthrough devices must be accessible via the VM kernel interface vmk0 (for the management network) and TCP port 902.

Disconnect USB devices from a VM before using the hot-add feature and adding CPU, memory, or PCI devices. Otherwise USB devices are disconnected automatically, and when you use the hot-add feature, some data can be lost. If you suspend a VM and then resume the VM, USB devices are disconnected and then reconnected to the VM again.

As for creating a VMware virtual USB drive or virtual flash drive for attaching to a virtual machine running on ESXi or VMware Workstation, just like you would mount a virtual floppy disk or CD/DVD disc, currently this option is not possible.

How to Connect a USB Device to a VM

Now that we have familiarized with the theory, we can go to the practical part and look at configuration in VMware vSphere.

Insert a USB device into a USB port of your ESXi server. In my case, I’m going to insert a USB flash drive.

You can check whether the USB device is connected to ESXi after that in the ESXi command line interface

lsusb

As you can see on the screenshot below, my flash drive is now connected to the ESXi server.

VMware USB devices configuration

The USB arbitrator service must be running. You can check the service status with the command:

chkconfig usbarbitrator --list

If the USB arbitrator is stopped, you can start the USB arbitrator with the command:

/etc/init.d/usbarbitrator start

Read more about ESXiCLI in the blog post.

The VM is residing on the ESXi host that is managed by vCenter Server in my example. Windows is installed as a guest OS on my VM. Open VMware vSphere Client, go to Hosts and Clusters. In the Navigator pane select a virtual machine to which you want to connect the USB device by using the passthrough feature. The VM should be in the powered-off state. You should install VMware Tools in the guest OS.

Once the VM is selected, click Actions > Edit settings.

Editing VM settings to connect USB drive to VMware virtual machine

In the Edit Settings window, check whether a USB controller is present in the VM configuration. If the USB controller is missing, click Add New Device > USB Controller, and select the USB controller type (USB 2.0 or USB 3.0).

If the USB controller is present, you should add the USB device that is attached to the ESXi server to your VM. Click Add New Device > Host USB Device.

Adding a VMware USB device to a virtual machine

A new string is added to the VM configuration. In the New USB Host device string, select one USB device that you want to connect to the VM. Now I have two USB flash drives attached to my physical ESXi server, and I’m connecting the Verbatim Flash drive to my VM (Verbatim product 0x0302). If you need to enable vMotion support, expand New Host USB device, and select the “Support vMotion while device is connected” checkbox. Keep in mind vMotion requirements and limitations. Hit OK to save the configuration and close this window.

VM USB configuration – selecting a USB device that should be connected to a VM

Start a virtual machine. Once the guest OS is loaded, wait until the USB flash drive is initialized and installed. Then open Windows Device Manager. Right click My Computer and, in the context menu, click Manage. In the Computer Management Window, click Device Manager and expand the Disk drives section (as we connect a USB flash drive, this device should be displayed in this category of devices). We can see Verbatim USB device, which is the name of my USB flash drive connected to the VM with the VMware USB Passthrough feature.

Open Disk Management in the Computer Management window, and make sure that a disk drive is assigned to the USB flash drive. In my case, everything is correct, and I can now copy data from the USB flash drive to the virtual disk of a VM in guest Windows and vice versa.

ESXi USB passthrough is working and a USB drive is connected to a VM

VMware PCI Passthrough

If the USB device that you want to connect to a VM by using the VMware USB passthrough feature is not supported by VMware, you can try using the PCI passthrough feature and connecting the entire physical USB controller that has the PCI interface to a VM. If your ESXi server has a USB controller inserted into a PCI slot, you can passthrough this PCI device to a virtual machine. VMware PCI passthrough is also called VMDirectPass. This method is less convenient than using USB passthrough but can help you get the task done.

Requirements

Intel Directed I/O or AMD I/O Virtualization Technology (AMD IOMMU) must be supported by server hardware and enabled in UEFI/BIOS. If you have only one USB controller in your server, you should attach an additional USB controller to avoid losing connections to USB devices used by your physical ESXi server.

Restrictions and limitations:

  • A VM cannot be suspended.
  • A VM cannot be migrated to other ESXi hosts with vMotion.
  • VM snapshots are not supported.

Let’s look at the workflow of connecting a USB device to a VM by using PCI passthrough and connecting the entire USB controller to which a USB device is connected to a VM.

In order to identify the USB device and USB controller to which your USB device is connected (if there are multiple USB controllers installed on your server) use this command:

lsusb -v | grep -e Bus -e iSerial

Find the string with the name of your USB device connected to the USB controller and identify the id of the controller (for example, bus002).

You can check the VMkernel log when connecting a USB device to an ESXi server.

tail -f /var/log/vmkernel.log | grep -i USB

Open VMware vSphere Client, and go to Hosts and Clusters.

Select the ESXi host on which your VM (to which you need to connect a USB device) is residing.

Go to the Configure tab, then select PCI Devices in the Hardware section.

Find and select the needed USB controller in the list of PCI devices by using the ID you have detected before.

The status of the selected controller should now change to Available.

Hit OK to save settings.

Reboot the ESXi host for changes take effect.

Make sure that your VM is powered off. Go to VM settings, click Add New Device > PCI device > Add. Select your USB controller in the dropdown menu by using the correct ID.

Power On the VM. Check devices in the guest OS. If Windows is installed on the VM, open Device Manager, then check controllers and storage devices.

Connecting USB Smart Cards

Some USB devices that have the appropriate class of CCID may not be supported by default for connection to VMs. The error message "Cannot connect ‘path:0/1/6/1’ to this virtual machine. The device was not found" is displayed. In logs, you can see the message:

(vmx-vcpu-0) did not claim interface 0 before use

PCSCD is the ESXi daemon that controls smart card readers. Access to USB smart cards is disabled for virtual machines to avoid conflicts because a smart card needed by an ESXi server. As a result, you cannot use USB passthrough with default configuration for USB smart cards. You need to edit the configuration of the ESXi host to enable USB passthrough for USB smart cards.

Stop the virtual machine. Open virtual machine settings, select the VM Options tab, and click Edit Configuration.

Editing VM options for VM USB smart card connection

In the Configuration Parameters window, click Add Configuration Params.

Editing VM USB configuration for a USB smart card reader

As an alternative, you can edit the VMX configuration file of the virtual machine in the command line interface by connecting to ESXi console with an SSH client.

Edit or add this line to the VM configuration if the parameter is not present:

usb.generic.allowCCID = "TRUE"

Save VM configuration.

Stop the PCSCD service by using this command in the ESXi command line:

sudo /etc/init.d/pcscd stop

Make sure that the PCSCD service is not running:

ps | grep pcscd

Power on the VM, and check devices to verify that the USB smart card reader is now connected to the VM.

Conclusion

This blog post has covered ESXi USB passthrough, which is a useful feature supported by VMware vSphere. You can connect diverse devices attached to USB ports of physical ESXi servers to virtual machines directly and use them in guest operating systems of VMs just like you use these USB devices on regular physical computers. You can even migrate virtual machines using external USB devices connected to the parent ESXi host with vMotion to another host. Despite some limitations, the VMware USB passthrough feature adds more capabilities and flexibility in different situations.

VMware VM backup is important for those who use VMware vSphere. Download the Free Edition of NAKIVO Backup & Replication and try this universal data protection solution to protect your VMware vSphere VMs and other data in your environment.

How to mount VMFS 6 in Debian https://www.nakivo.com/blog/mount-vmfs-datastore-in-linux-windows-esxi/

 

How to mount VMFS 6 in Ubuntu

  1. Create a directory to be used as a mount point to mount VMFS 6 in Linux:

mkdir /mnt/vmfs6

Note: If you try to mount the VMFS 6 file system in Linux with vmfs-fuse, you get an error because vmfs-fuse supports VMFS 3 and VMFS 5 but doesn’t support VMFS 6. In this case, the following message appears:

VMFS: Unsupported version 6
Unable to open filesystem

VMFS tools don’t support VMFS 6

You need to install VMFS6-tools which contains vmfs6-fuse that is used to mount VMFS 6 in Linux. You can find VMFS6-tools on a website with deb packages: https://packages.debian.org/sid/vmfs6-tools
  1. Download the current version of VMFS6-tools:

wget http://http.us.debian.org/debian/pool/main/v/vmfs6-tools/vmfs6-tools_0.1.0-3_amd64.deb

  1. Install the downloaded deb package:

dpkg -i vmfs6-tools_0.1.0-3_amd64.deb

Installing VMFS 6 tools to mount VMFS in Linux

Note: libc6 >= 2.28 is required to install VMFS6-tools. If you use Ubuntu 18, you may encounter errors during installation for this reason.

  1. Now you can mount VMFS in Ubuntu 20 to /mnt/vmfs6 with the command:

vmfs6-fuse /dev/sdc1 /mnt/vmfs6

Viewing contents of a VM datastore with VMFS in Linux

The VMFS 6 file system has been successfully mounted in Ubuntu 20 in read-only mode. Now you can copy VM files to the needed location. You can temporarily run copied VMs on Linux with VMware Workstation installed or on a Windows machine with VMware Workstation or Hyper-V until your ESXi server hardware is repaired or a new server is delivered (if you don’t have another ESXi host to run the VMs).

Remember that when you copy thin provisioned disks from a VMFS file system to ext4, NTFS, or other conventional file systems, the virtual disk files take up as much space as if they were thick provisioned. That’s why be ready to prepare enough disk space.

How to mount VMFS with multiple extents in Linux

Let’s look at a more complex example of mounting VMFS in Linux when a VMFS volume consists of two extents. We have two disks of the same size, which are combined into a single VMFS volume (datastore12).

USB Devices as VMFS Datastore in vSphere ESXi 6.5 https://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/

 In ESXi 6.5, there are some changes concerning devices connected with USB. The legacy drivers, including xhci, ehci-hcd, usb-uhci, and usb-storage have been replaced with a single USB driver named vmkusb. The new driver has some implications if you are trying to use USB devices like USB sticks or external hard disks as VMFS formatted datastore.

Some people have reported that they have issues with USB Datastores since ESXi 6.5. I've tried to reproduce and fix those problems. This post explains the changes in the new version and how to create VMFS 5 or VMFS6 formatted USB devices as datastore on your ESXi host.

Please remember that using USB Datastores is not supported by VMware so if you have problems, do not open service requests for that.

Device Naming Changes in ESXi 6.5

Which I noticed first was that the devices naming in 6.5 have been changed. In ESXi 6.0, all USB devices are recognized as mpx.vmhbaXX while in 6.5 some are listed as naa.XXX. Here is an example of a Samsung M.2 flash drive connected with a USB3 adapter:

# ESXi 6.0u2
-rw-------    1 root     root     128035676160 Nov 27 15:57 naa.5000000000000001

# ESXi 6.5
-rw-------    1 root     root     128035676160 Nov 27 18:16 mpx.vmhba36:C0:T0:L0

However, not all devices are affected. My 16GB flash drives shows up with the old mpx.vmhba naming on both versions:

# ESXi 6.0u2
-rw-------    1 root     root     16022241280 Nov 27 16:02 mpx.vmhba34:C0:T0:L0

# ESXi 6.5
-rw-------    1 root     root     16022241280 Nov 27 16:18 mpx.vmhba34:C0:T0:L0

The new naming does not have an impact on USB datastores, despite it might happen that you overlook the USB device when checking the /dev/disks/ directory. You can still partition and format USB devices with VMFS like in previous versions, just use the new name.

Existing Datastores are not mounted automatically

A common problem which I've heard multiple times, but only seen since ESXi 6.5, is that when you connect a flash drive that already contains a VMFS datastore, the datastore does not show up. You can see the device in /dev/disks but the datastore remains offline. The vmkernel.log hints the problem:

2016-11-27T10:25:05.885Z cpu0:65593)VMWARE SCSI Id: Id for vmhba34:C0:T0:L0
0x50 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x41 0x53 0x32 0x31 0x31 0x35
2016-11-27T10:25:05.885Z cpu0:65593)ScsiDeviceIO: 9120: Get VPD 80 Inquiry for device "naa.5000000000000001" from Plugin "NMP" failed. Not supported
2016-11-27T10:25:05.885Z cpu0:65593)ScsiDeviceIO: 9167: Get VPD 86 Inquiry for device "naa.5000000000000001" from Plugin "NMP" failed. Not supported
2016-11-27T10:25:05.887Z cpu0:65593)ScsiDeviceIO: 7851: Could not detect setting of QErr for device naa.5000000000000001. Error Not supported.
2016-11-27T10:25:05.891Z cpu0:65593)ScsiDeviceIO: 8365: Could not detect setting of sitpua for device naa.5000000000000001. Error Not supported.
2016-11-27T10:25:05.898Z cpu1:65562)ScsiDeviceIO: 2948: Cmd(0x439500b60000) 0x1a, CmdSN 0x238 from world 0 to dev "naa.5000000000000001" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-11-27T10:25:05.905Z cpu0:65593)LVM: 11136: Device naa.5000000000000001:1 detected to be a snapshot:
2016-11-27T10:25:05.905Z cpu0:65593)LVM: 11143: queried disk ID: <type 2, len 14, lun 0, devType 0, scsi 0, h(id) 4847995147196820166>
2016-11-27T10:25:05.905Z cpu0:65593)LVM: 11150: on-disk disk ID: <type 0, len 11, lun 0, devType 0, scsi 0, h(id) 8386732072863042940>
2016-11-27T10:25:05.914Z cpu0:65593)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 40, Parameter: 0x43029e372700, Registered!
2016-11-27T10:25:05.914Z cpu0:65593)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 200, Parameter: 0x43029e372700, Registered!
2016-11-27T10:25:05.914Z cpu0:65593)ScsiDevice: 4265: Successfully registered device "naa.5000000000000001" from plugin "NMP" of type 0

The datastore obviously gets detected as snapshot. I don't know why and I've only seen this in ESXi 6.5 but it's very easy to fix. Get a list of VMFS snapshots with ESXCLI:

# esxcli storage vmfs snapshot list
583b1a72-ade01532-55f6-f44d30649051
 Volume Name: usbflash
 VMFS UUID: 583b1a72-ade01532-55f6-f44d30649051
 Can mount: true
 Reason for un-mountability:
 Can resignature: true
 Reason for non-resignaturability:
 Unresolved Extent Count: 1

Use the VMFS UUID to mount the datastore:

# esxcli storage vmfs snapshot mount -u 583b1a72-ade01532-55f6-f44d30649051

The datastore should now show up within /vmfs/volumes and the Embedded Host Client.

Create VMFS5/6 Datastore on USB drives

Creating VMFS5 or VMFS6 datastores works just as in previous versions:

  1. Connect to the ESXi host with SSH
  2. Stop the USB arbitrator service. This service is used to passthrough USB device from an ESX/ESXi host to a virtual machine. (When disabling it, you can no longer passthrough USB devices to VMs)
    ~ # /etc/init.d/usbarbitrator stop
  3. (optional) Use this command to permanently disable the USB arbitrator service after reboot.
    ~ # chkconfig usbarbitrator off
  4. Plug in the USB device to your ESXi host. While connecting the USB device you can either watch /var/log/vmkernel.log to identify the device name or identify it within /dev/disks.
    Devices are either named mpx.vmhbaXX or with an naa.X number.

    ~ # ls /dev/disks/
  5. Write a GPT label to the device (Assuming that the Device ID is naa.5000000000000001)
    ~ # partedUtil mklabel /dev/disks/naa.5000000000000001 gpt
  6. To create a partition you need to know the start sector, end sector, which depends on the device size and the GUID.
    The start sector is always 2048
    The GUID for VMFS is AA31E02A400F11DB9590000C2911D1B8
    The end sector can be calculated with the following formula (Use the numbers from getptbl):

    ~ # partedUtil getptbl /dev/disks/naa.5000000000000001
    gpt
    15566 255 63 250069680

    15566 * 255 * 63 - 1 = 250067789

    You can also calculate the end sector with the following command:

    ~ # eval expr $(partedUtil getptbl /dev/disks/naa.5000000000000001 | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1
    250067789
  7. Create the VMFS partition (Replace with your end sector)
    ~ # partedUtil setptbl /dev/disks/naa.5000000000000001 gpt "1 2048 250067789 AA31E02A400F11DB9590000C2911D1B8 0"
  8. Format the partition with VMFS5 or VMFS6
    # VMFS v5
    ~ # vmkfstools -C vmfs5 -S USB-Datastore /dev/disks/naa.5000000000000001:1
    
    # VMFS v6
    ~ # vmkfstools -C vmfs6 -S USB-Datastore /dev/disks/naa.5000000000000001:1

The USB-Datastore should now appear in your datastores view.
vmfs6-usb-datastore-esxi65

This is how your command output should look like:

[root@esx4:~] partedUtil mklabel /dev/disks/naa.5000000000000001 gpt
[root@esx4:~] eval expr $(partedUtil getptbl /dev/disks/naa.5000000000000001 | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1
250067789
[root@esx4:~] partedUtil setptbl /dev/disks/naa.5000000000000001 gpt "1 2048 250067789 AA31E02A400F11DB9590000C2911D1B8 0"