Cannot create a new datastore from vCenter Server or directly from the vSphere Client Creating a new datastore fails The Add Storage Wizard reports the error: An error occurred during host configuration. Call “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object “ha-datastoresystem” on ESXi “xxx.xxx.xxx.xx” failed. An internal error occurred in the vsphere client.
Cause
This issue may occur if:
The size of the LUN is more than 2TB – 512 bytes. The maximum LUN size in vSphere 4 is 2TB – 512 bytes. For more information on this limitation, see Troubleshooting a LUN that is extended in size past the 2TB/2047GB limit (1004230).
The LUN being presented was used as an RDM earlier and does not have a valid partition table.
The LUN being presented was used as a disk device on an other operating system and does not have a valid partition table.
The LUN contains a GPT partition that cannot be removed.
Resolution
To prevent this issue:
Ensure that the LUN size is less than 2TB – 512 bytes or 2047GB. If the LUN is larger then 2TB – 512 bytes, delete it from your Storage Array and recreate it with a size less than the 2TB – 512 bytes limit.
If the disk was used by another operating system in the past (I.E.: it was an RDM, or Linux, FreeBSD, or other filesystem) or contains a GPT partition, you must delete the partition information.For more information

Authentication and authorization govern access. vCenter Single Sign-On supports authentication, which means it determines whether a user can access vSphere components at all. Each user must also be authorized to view or manipulate vSphere objects.
vSphere supports several different authorization mechanisms, discussed in Understanding Authorization in vSphere. The focus of the information in this section is how the vCenter Server permission model works and how to perform user management tasks.
vCenter Server allows fine-grained control over authorization with permissions and roles. When you assign a permission to an object in the vCenter Server object hierarchy, you specify which user or group has which privileges on that object. To specify the privileges, you use roles, which are sets of privileges.
Initially, only the administrator user for the vCenter Single Sign-On domain, [email protected] by default, is authorized to log in to the vCenter Server system. That user can then proceed as follows:
Add an identity source in which users and groups are defined to vCenter Single Sign-On. See the Platform Services Controller Administration documentation.
Give privileges to a user or group by selecting an object such as a virtual machine or a vCenter Server system and assigning a role on that object for the user or group.

Storage I/O Control v2
Storage I/O Control (SIOC) was initially introduced in vSphere 4.1 to provide I/O prioritization of virtual machines running on a cluster of ESXi hosts that had access to shared storage. It extended the familiar constructs of shares and limits, which existed for CPU and memory, to address storage utilization through a dynamic allocation of I/O queue slots across a cluster of ESXi servers. The purpose of SIOC is to address the ‘noisy neighbor’ problem, i.e. a low priority virtual machine impacting other higher priority virtual machines due to the nature of the application and its I/O running in that low priority VM.
vSphere 5.0 extended SIOC to provide cluster-wide I/O shares and limits for NFS datastores. This means that no single virtual machine should be able to create a bottleneck in any environment regardless of the type of shared storage used. SIOC automatically throttles a virtual machine which is consuming a disparate amount of I/O bandwidth when the configured latency threshold has been exceeded. To allow other virtual machines receive their fair share of I/O bandwidth on the same datastore, a share based fairness mechanism has been created which now is supported on both NFS and VMFS.
vSphere 5.1 introduced a new SIOC feature called Stats Only Mode. When enabled, it doesn’t enforce throttling but gathers statistics to assist Storage DRS. Storage DRS now has statistics in advance for new datastores being added to the datastore cluster & can get up to speed on the datastores profile/capabilities much quicker than before.
Another 5.1 feature was Automatic Threshold Computation. The default latency threshold for SIOC is 30ms. Not all storage devices are created equal so this default was chosen as a sort of “catch-all”. There are certain devices which will hit their natural contention point much earlier than others, for example All Flash Arrays, in which case the threshold should be lowered by the user. However, manually determining the correct latency can be difficult for users. This gave rise to the need for the latency threshold to get automatically determined at a correct level for each device. Using the I/O injector modeling of SIOC, peak throughput and corresponding latency of a datastore is measured. The latency threshold value at which Storage I/O Control will kick in is then set to 90% of this peak value (by default). vSphere administrators can change this 90% to another percentage value or they can still input a millisecond value if they so wish.
The default latency threshold for SIOC can be reduced to as low as 5ms.
SIOC V1 Overview
SIOC V1 is disabled by default. It needs to be enabled on a per datastore level, and it is only utilized when a specific level of latency has been reached. By default, the latency threshold for a datastore is set to 30ms, as mentioned earlier. If SIOC is triggered, disk shares (aggregated from all VMDKs using the datastore) are used to assign I/O queue slots on a per host basis to that datastore. In other words, SIOC limits the number of IOs that a host can issue. The more VMs/VMDKs that run on a particular host, the higher the number of shares, and thus the higher the number of IOs that that particular host can issue. The throttling is done by modifying the device queue depth of the various hosts sharing the datastore. When the period of contention passes, and latency returns to normal values, the device queue depths are allowed to return to default values on each host.
SIOC V2 Introduction
Before describing SIOC V2, it should be highlighted that SIOC V1 and SIOC V2 can co-exist on vSphere 6.5. This makes it much simpler when considering upgrades, or migrations between versions. With that in mind, SIOC V2 is considerably different from a user experience perspective when compared to V1. SIOCv2 is implemented using IO Filter framework Storage IO Control category. SIOC V2 can be managed using SPBM Policies. What this means is that you create a policy which contains your SIOC specifications, and these policies are then attached to virtual machines.
Creating an SIOC policy based
Creating an SIOC policy is done is exactly the same way as building a storage policy for VSAN or Virtual Volumes. Select the VM Storage Policy from the vSphere client home page, and from there select the option to create a new VM Storage Policy. VM Storage Policies in vSphere 6.5 has a new option called “Common Rules”. These are used for configuring data services provided by hosts, such as Storage I/O Control and Encryption.
Use common rules in the VM storage policy
The first step is to click on the check box to enabled common rules. This will then allow you to add components, such as SIOC, to the policy.

Add Component – Storage I/O Control
In vSphere 6.5, there are two components available for common rules, Encryption and Storage I/O Control. Select Storage I/O Control in this case. Now you can select Normal, High, Low or Custom shares allocation.

This table describes the different Limits,Shares and Reservations associated with each setting:
HIGH
NORMAL
LOW
Limits
100,000
10,000
1,000
Reservation
100
50
10
Shares
2,000
1,000
500
When the policy has been created, it may be assigned to newly deployed VMs during provisioning,or to already existing VMs by assigning this new policy to the whole VM (or just an individual VMDK) by editing its settings. One thing to note is that IO Filter based IOPS does not look at the size of the IO. For example, there is no normalization so that a 64K IOP is not equal to 2 x 32K IOPS. It is a fixed value of IOPS irrespective of the size of the IO.
Custom Allocation
If neither of the values in the Normal, High, Low allocations is appropriate, there is the ability to create custom settings for these values. In a custom setting, IOPS limit and IOPS reservation are both set to -1, implying unlimited. These may be modified as required.

Advanced Options
SchedCostUnit
This is an advanced parameter that was created for SIOC V1 only. SIOC V2 does not have SchedCostUnit implemented. For V1, SchedCostUnit determines the unit size (normalized size) of an IO operation for scheduling, and it is currently a constant value of 32K. This constant value, however, may not satisfy different requirements from different customers. Some customers may want to set this unit size to 4K. Other customers may want to set it up to 256K.
To satisfy these different requirements, SchedCostUnit is now configurable. It defaults to an IO size value of 32K, and allowable values range between 4K to 256K.
The SchedCostUnit dictates how requests are counted. A request with size <= SchedCostUnit counts as a single I/O. Anything greater than SchedCostUnit will be counted as 2 or more requests.
For example, by changing the SchedCostUnit from 32K to 64K, the number of IOPS observed will halve. The size of the IO can be set using the:
“esxcli system settings advanced set -o /Disk/SchedCostUnit -i 65536”
and verified by using the”
“esxcli system settings advanced list -o /Disk/SchedCostUnit”
command. SIOC V2 counts guest IO directly. IOPS will be counted based on IO count, regardless of the IO size.
SchedReservationBurst
When limits are set on VMDKs, requests could have high average latency because the limit was enforced at a high (per request) granularity. This was due to the strict enforcement on a VM getting its share of IOs in interval of 1 second/L, where L is the user specified limit. The issue is more visible in fast storage, such as flash arrays. It was noted that SIOC V2 did not perform well when presented with a “bursty” workload on fast storage.
This SchedReservationBurst setting relaxes that constraint so a VM get its share of IOs at any time during a 1 second window, rather than enforce strict placement of IOs in intervals of 1/L. BURST option is turned-on by default.
SIOC V2 Limitations
In this initial release of SIOC V2 in vSphere 6.5, there is no support for vSAN or Virtual Volumes. SIOC v2 is only supported with VMs that run on VMFS and NFS datastores.

Select Menu > Policies and Profiles, and under Policies and Profiles, click VM Customization Specifications.
Click the Create a new specification icon.
The New VM Guest Customization Specification wizard opens.
On the Name and target OS page, enter a name and description for the customization specification and select Windows as a target guest OS.
(Optional) Select the Generate a new security identity (SID) option and click Next.
A Windows Security ID (SID) is used in some Windows operating systems to uniquely identify systems and users. If you do not select this option, the new virtual machine has the same SID as the virtual machine or template from which it was cloned or deployed.
Duplicate SIDs do not cause problems when the computers are part of a domain and only domain user accounts are used. However, if the computers are part of a Workgroup or local user accounts are used, duplicate SIDs can compromise file access controls. For more information, see the documentation for your Microsoft Windows operating system.
On the Set Registration Information page, enter the virtual machine owner’s name and organization and click Next.
On the Computer name page, enter a computer name for the guest operating system and a domain name.
The operating system uses the computer name to identify itself on the network. On Linux systems, it is called the host name.
Option
Action
Use the virtual machine name
Select this option to use the virtual machine name. The computer name that vCenter Server creates is identical to the name of the virtual machine on which the guest operating system is running. If the name exceeds 63 characters, it is truncated.
Enter a name in the Clone/Deploy wizard
Select this option to be prompted to enter a name during cloning or deployment.
Enter a name
Enter a name.
The name can contain alphanumeric characters and a hyphen (-). It cannot contain a period (.), blank spaces, or special characters, and cannot contain digits only. Names are not case-sensitive.
(Optional) To ensure that the name is unique, select the Append a numeric value check box.
This action appends a hyphen followed by a numeric value to the virtual machine name. The name is truncated if it exceeds 63 characters when combined with the numeric value.
Generate a name using the custom application configured with vCenter Server Optional: Enter a parameter that can be passed to the custom application.
On the Windows license page, provide licensing information for the Windows operating system and click Next.
Option
Action
For nonserver operating systems
Type the Windows product key for the new guest operating system.
For server operating systems
Type the Windows product key for the new guest operating system.
Select Include Server License Information.
Select either Per seat or Per server.
If you select Per server, enter the maximum number of simultaneous connections for the server to accept.
On the Set Administrator Password page, configure the administrator password for the virtual machine and click Next.
Enter a password for the administrator account and confirm the password by typing it again.
(Optional) Select the Automatically logon as Administrator check box to log users in to the guest operating system as Administrator, and select the number of times to log in automatically.
On the Time zone page, select the time zone for the virtual machine and click Next.
(Optional) On the Run Once page, specify commands to run the first time a user logs in to the guest operating system and click Next.
See the Microsoft Sysprep documentation for information about RunOnce commands.
On the Network page, select the type of network settings to apply to the guest operating system and click Next.
Select Use standard network settings so that vCenter Server configures all network interfaces from a DHCP server by using the default settings.
Select Manually select custom settings and configure each network interface yourself.
Select a network adapter from the list or add a new one.
For the selected NIC, click the vertical ellipsis icon and select Edit.
The Edit Network dialog box opens.
Click the IPv4 tab to configure the virtual machine to use IPv4 network.
You can configure all the settings at that stage or you can select the Prompt the user for an IPv4 address when the specification is used option. In that case, vCenter Server prompts for an IP address when you select to apply that customization specification during cloning or deployment. With that option, you can also configure the gateways during cloning or deployment.
Click the IPv6 tab to configure the virtual machine to use IPv6 network.
You can configure all the settings at that stage or you can select the Prompt the user for an address when the specification is used option. In that case, vCenter Server prompts for an IP address when you select to apply that customization specification during cloning or deployment. With that option, you can also configure the gateways during cloning or deployment.
Click the DNS tab to specify DNS server details.
Click WINS to specify primary and secondary WINS server information.
Click OK to close the Edit Network dialog box.
On the Set Workgroup or Domain page, select how the virtual machine participates in the network and click Next.
Option
Action
Workgroup
Enter a workgroup name. For example, MSHOME.
Windows Server Domain
Enter the domain name.
To add a computer to the specified domain, enter the user name and password for a user account that has permission.
On the Ready to complete page, review the details and click Finish to save your changes.

Do the following before you start to troubleshoot a problem using esxtop: 1. Log on to the VMware Management Interface for the ESX Server machine in question. Refer to the online document, Logging Into the VMware Management Interface, for details. In the status monitor, under Virtual Machines, note the virtual machine IDs (or VMIDs) for all virtual machines running on the server.

2. Make certain you have an secure shell (SSH) client. Windows users can get a free SSH client from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. 3. If you have ESX Server version 2.0.x, refer to the VMware Knowledge Base Answer ID 1078 for instructions on downloading and installing the VMware performance monitoring tools, esxtop and vmkusage. ESX Server version 2.1 and higher include esxtop and vmkusage. See Using vmkusage to Isolate Performance Problems on page 6 for a description of vmkusage. Starting esxtop Perform the following steps to start and set up esxtop
1. Using a secure shell (SSH), log on to the ESX Server machine as root. 2. Enter esxtop in the SSH command line. The esxtop display appears.

Note: The esxtop tool includes several interactive commands. To view a list of the interactive commands, enter h. 3. Enter the f command. The Field Select page appears

4. Enter r to toggle on the SWPD field.
5. Press any key other than a through x to see the esxtop display again

Note: You can also run esxtop in batch mode. For example, use the command: [root]# esxtop -b -n iterations > logfile. For detailed command reference information, enter man esxtop on the SSH command line. Examining CPU Usage This section describes how to assess system CPU loading, percentage of individual CPU use and individual virtual machine CPU use. Load Average Line Examine the load average on the first line to determine the amount of use for all physical CPUs on the ESX Server machine. The load averages are displayed for five-second, and one-, five- and fifteen-minute intervals. A load average of 1.00 means that the ESX Server machine’s physical CPUs are fully utilized, and a load average of 0.5 means they are half utilized. On the other hand, a load average of 2.00 means that you either need to increase the number of CPUs or decrease the number of virtual machines running on the ESX Server machine because the system as a whole is overloaded.
PCPU Line Examine the PCPU line for the percentage of individual physical CPU use for CPU0 and CPU1 respectively (for a dual-processor machine). The last value is the average percentage for all of the physical CPUs. As a rule of thumb, 80.00% is a desirable usage percentage, but bear in mind that different organizations have varying standards with respect to how close to capacity they run their servers. 90% should be considered a warning that the CPUs are approaching an overloaded condition.
You can enter the interactive c command to toggle the display of the PCPU line. If hyperthreading is enabled, the LCPU line appears whenever the PCPU line is displayed. The LCPU line shows the logical CPU use. Virtual Machine CPU Usage A virtual machine world is listed as vmm in the WTYPE column. The world ID (WID) corresponds to the VMID in the VMware Management Interface Status Monitor (see Getting Started on page 2). For virtual machines with one virtual CPU (VCPU), the VCPUID and WID is the same. For virtual machines with two VCPUs, there are two VCPUIDs associated with one WID. For example:

Use the WID and VMID values as cross references to identify a specific virtual machine’s display name. Use the following steps to assess virtual machine CPU usage. 1. Examine the %READY field for the percentage of time that the virtual machine was ready but could not get scheduled to run on a physical CPU. Under normal operating conditions this value should remain under 5%. 2. Examine the %USED field for the percentage of physical CPU resources used by a VCPU. If the physical CPUs are running at full capacity, you can use %USED to identify a virtual machine that is using a large amount of physical CPU resources. 3. Examine the %EUSED field for the percentage of the maximum physical CPU resource usage a virtual machine is currently using. The %EUSED value is calculated as follows: %EUSED = %USED *((# of VCPUs * 100)/ max) In this formula: # of VCPUs is the number of VCPUs configured in a virtual machine. max is the maximum percentage of CPU resources allocated to a virtual machine. The default is 100. The following table shows how %EUSED increases as the value of max decreases for a virtual machine with a constant physical CPU usage of 40% (i.e., %USED = 40).

%EUSED is an useful indicator of how close a virtual machine is to saturating the physical CPUs. If a virtual machine consistently saturates the physical CPUs (i.e., it uses 100% of the physical CPU resources), you can fix it by either: * Decreasing the number of virtual machines running on an ESX Server machine.
* Moving the virtual machine to a different ESX Server machine that has more physical CPU resources, and increasing the value of max. Assessing Memory Usage Look at the percentage of maximum memory actively used by a virtual machine listed in the %MEM column. Also, note the amount of swapped memory used by a virtual machine listed the SWPD column. Using some swap space is not necessarily bad, because the pages swapped out could be inactive; however, the use of swap space is a clue that you could be losing performance due to active swapping. If the swap percentage increases, there could be a performance problem. If the maximum system memory or swap space is exceeded, additional virtual machines will not power up and you may need to reconfigure memory and swap space. Assessing Disk and Network Usage Examine the DISK and NIC lines for disk activity. If the throughput is either not meeting expectations or approaching the maximum capacity of the hardware device, there is a potential for a performance bottleneck. Exiting esxtop Enter q to exit esxtop