Blue Flower

Time to renew my VCP qualification, no time to lose, only got barely 3 months to do it!

This page contains the notes that I have taken for my study, PLEASE turn off the flipping music!!!! It's time to revise hard

Passed 25th August 2017, 410 out of 500 (82%), oh yeath, what a relief! :)

Step1: Installing vCenter

Installing VCSA (vCenter Server Appliance) is actually a nightmare with version 6.0, have not tested with 6.5 but have heard there are mayor improvements, let's hope so.

  1. Before installing the VMware Client Integration Plug-in 6.0, be sure to turn off your antivirus! The program will actually modify your hosts file, and most antivirus solutions will block that I managed to install the Plug-in without issues in a Win 2008 R2 with IE11 version 11.0.9600.17207
  2. Needless to say, you need to right-click on the Plug-in and choose to install with admin rights
  3. Ensure you set the IE11 to "Automatically detect intranet network"

 

vCenter/vsphere tips:

  • To check the proper certificate of the ESXi host server, visit the "View Support Information" in the DCUI
  • VMware does not support concurent deployments of PSC in the same SSO domain
  • Enhanced Link Mode is only available from Standard up license (not foundation or essentials)
  • Shares are ignore ift there is no contention
  • Enterprise Plus license is required for Network IO control
  • Cross-vCenter vMotion only comes with Enterprise Plus. Enterprise+ also includes Storage DRS, Storage IO Control (SIOC), Single Rool IO Virtualizacion (SR-IOV), Network IO Control (NIOC), Flash Read Cache (vFRC), vDS, Host Profiles, Auto Deploy

 

VMCA (VMware Certificate Authority)

  • VMCA does not support CRLs
  • VMCA does not have the concept of certificate revocation, you'll have to replace all
  • VECS (VMware End Point Certificate Store) is a local repository for certificates and private keys VECS does not store ESXi host certificates, those are locally in /etc/vmware/ssl folder

    VMCA (VMware Certification Authority), runs in the PSC, responsible for issuing certificates for solution users, machine certificates and ESXi certificates

    VECS (VMware Endpoint Certificate Storage) is a local repository for certificates and private keys

    ESXi certificates are stored locally on the ESXi hosts "etc/vmware/ssl" locationvecs-

 

Ensure that when you export VM, they have no DVD/CD attached

  • OVF (Open Virtualisation Format), set of files
  • OVA (Open Virtualisation A ), all in one file

Multipath iSCSI (two nic) and use separeate physical network (if not, defenitely separeate VLANs). Also, iSCSI uses no routing at all, so ensure all stays in the same VLAN

Do multipath for iSCSI

New stuff to know for vSphere 6

 Harden virtual machines

  • isolation.tools.diskWiper.disable = TRUE
  • isolation.tools.diskShrink.disable = TRUE
  • isolation.tools.copy.disable =TRUE
  • isolation.tools.paste.disable = TRUE
  • isolation.device.connectable.disable = TRUE
  • isolation.device.edit.disable = TRUE
  • vmx.log.keepOld = 10 //to disable logging all together, add the following line instead: logging = FALSE
  • tools.setInfo.sizeLimit = "2000000" /set the size of the VMX file to 2MB instead of the default 1MB, so more info can be added
  • tool.guestlib.enableHostInfo = FALSE /PerfMon counter disabled by default
  • These are the unexposed features:
    • isolation.tools.unity.push.update.disable -TRUE
    • isolation.tools.ghi.launchmenu.change =TRUE
    • isolation.tools.ghi.autologon.disable =TRUE
    • isolation.tools.hgfsServerSet.disable =TRUE
    • isolation.tools.memeSchedFakeSampleState.disable -TRUE
    • isolation.tools.getCreds.disable =TRUE

The following 2 lines are added by default, but ensure they're there otherwise VMTools can be use to eject devices

  • isolation.device.connectable.disable = TRUE
  • isolation.device.edit.disable = TRUE

FT and HA are not supportetd for VMs that have 3D graphics enable

If the machine is slow you can disable Accelerated 3D Graphics or add the following to the VMX file:

  • vga.vgaOly = TRUE

If the VMware Tools installation hangs on a VM, use this command on the host to cancel it:

  • vim-cmd vmsvc/tools.cancelinstall "vm.id"

VMware tools in Linux use "sudo apt-get install open-vm-tools"

If when typing characters are repeated, add this line to the VMX file, increasing the threshold for auto-repeat

  • keyboard.typematicMinDelay = "2000000"

To use FT Legacy on a VM, add this entry:

  • vm.uselegacyft = TRUE

 

Harden ESXi host

  • logging = FALSE //add this line to the file /etc/VMware/config to disable VM logging
  • Security.PasswordQualityControl = control the strengh of the DCUI password
    • min=disabled,disabled,disabled,7,7
    • min character accepted for 1 class, min accepted for 2 classes, min accepted for passphrases, min accepted for 3 classes (in the example above 7), min accepted for 3 classes plus passphared, meaning 4 classes
    • Uppercase at begging and number at the end don't count
  • MOB (Managed Object Browser) is disable by default to prevent attacks, but you have to enable it if you want to extract an old certificate from the host
    • Config.HostAgent.plugins.solo.enableMob = FALSE

You can control the behavious of SSH by modifying /etc/ssh/sshd_config file, the paremeter PermitRoolLogin

Net.IOControlPnicOptOut = vmnic1, vmnic2, etc //exclude the nic for participating in NIOC

 

Harden vCenter Server

The following filters are by default set to TRUE (enable) so that vCenter cannot detect the present of the related object when scanning for storage

  • For RDM objects = config.vpxd.filter.rdmFilter
  • For VMFS objects = config.vpxd.filter.vmfsFilter
  • For Host Rescan = config.vpxd.filter.hostRescanFilter
  • For Same Host and Transports = config.vpxd.filter.SameHostAndTransportsFilter

To setup MSCS (Microsoft Cluster) the RDM filter must the set to FALSE, so that vCenter can detect the LUN and add it as an RDM even though is being use as an RDM by another VM

If the performance graphics are not displaying correctly, or if the statistics collection level is higher than 1, you many need to truncate these two tables in the vCenter database:

  • truncate table VPX_HIST_STAT1
  • truncate table VPX_SAMPLE_TIME1

There is a script that you can donwload from KB2110031 to reduce the amount of historical data on the PostgressSQL database

#service-control --stop --all ;stops all vCenter services, use then the --start to initiate then

To remove the warning about management network redundancy, set this on the advance options:

  • das.ignoreRedundantNetWarning = true

 

ESXi Networking

Load-based teaming is only offered in vDS. Standard switches only offer:

  • Originating virtual port ID
  • Source MAC address
  • Source and Destination IP hash

Dynamic binding = NO, it's been decreciated from vSphere 5.0; dynamic binding means the adapter is only connected to the vDS when the machine is on

Ephemeral = YES, ports are created and deleted on demand, just like Elastic

When LACP is in used, you cannot configure port mirroning

Remember that a dvPort group can be used both by a VM and by a VMkernel port

PVLANs need to be configure at the vDS level .

  • Isolated PVLAN, only talk to the promiscuous
  • Community PVLAN, talk to other communities as well as the promiscuos

TSO (TCP Segmentation Offload) is enable by default, and if you change it reverts to its default after a reboot. To change it permanentely the command needs to be added to /etc/rc.local/local.sh

  • To identify whether it's enable or disable run: esxcli system settings advanced list -o /Net/UseHwTSO (if it says 1 is enable)
  • To disable TSO in Linux run #ethtool -K vmnic# tso off
  • To enable TSO in Windows add this line to .vmx: enthernet#.features = "0x2"

Network recovery (rollback) is not suported on stateless configuration of auto deploy hosts (with the configuration installed in RAM)

ESXi firewall commands: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-7A8BEFC8-BF86-49B5-AE2D-E400AAD81BA3.html

 

VMware Knowledge base - useful articles

How to add the certificates of vCenter for all green approval https://kb.vmware.com/kb/2108294

Virtual Machines Hardware Versions https://kb.vmware.com/kb/1003746 Here a list of newly added features of HW version 11 http://www.running-system.com/vsphere-6-new-virtual-hardware-version-11-vhw11/

Determine your build version: https://kb.vmware.com/kb/1022196 , then check your build numbers against this list https://kb.vmware.com/kb/1014508

Editing files on an ESX host using vi or nano https://kb.vmware.com/kb/1020302

Integration Plug-in fails to install: https://kb.vmware.com/kb/2130672 Rubbish, use the flat client to deploy ova for the time being

System Logs are stored on non-persistant storage https://kb.vmware.com/kb/2032823

Upgrading from vSphere 5.x to 6 https://kb.vmware.com/kb/2057795

Backup and Restore the vPosgree database https://kb.vmware.com/kb/2091961

Restart the ESXi host management agents through CLI https://kb.vmware.com/kb/1003490

Understanding network rollback https://kb.vmware.com/kb/2032908

All TCP and UDP ports use in the vSphere Universe: https://kb.vmware.com/kb/1012382

After a Windows installation of vCenter, you have NO ACCESS AT ALL after logging in: You need to override the path as stated in here: https://communities.vmware.com/thread/507933

Please try the following. (before that please create a backup from the vCenter server) In regedit system wide path is defined here: Computer\HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment->Path Local system account overridden Path is defined under: Computer->HKEY_USERS->S-1-5-18\Environment->Path (first step, second location can be verified if exists; and if exists values can be compared to see differences) You can either remove(rename) the existing override (Computer->HKEY_USERS->S-1-5-18\Environment->Path) completely. This will make the system wide Path to take effect. Or if this override was specified on purpose (for some reason), then modify Computer->HKEY_USERS->S-1-5-18\Environment->Path to make sure to include MIT Kerberos installation (such as c:\Program Files\MIT\Kerberos\bin), and possibly other vSphere paths (like OpenSSL) for completeness.

VMware ESXi 6 Password Policy: https://www.ivobeerens.nl/2015/10/07/vmware-esxi6-password-policy/

Configure vFlash Read Cache (vFRC) https://www.vladan.fr/vmware-vflash-read-cache-vfrc/

Visit the VMware Labs (flings) to keep an eye in future develops: https://labs.vmware.com/flings/

Data Collections Levels: https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.monitoring.doc_50%2FGUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html

Configuration: https://docs.vmware.com/en/VMware-vSphere/6.0/vsphere-esxi-vcenter-server-601-appliance-configuration-guide.pdf

All TCP/UDP ports used in the vSphere universe: https://kb.vmware.com/kb/1012382

 

Resources

The 5 states of ESXi hosts are based on the "minFree" value, which is 899MB of the first 28GB of the host, plus 1% of any additional memory; the states are:

  • High - equals 400% of minFree
  • Clear - equals 100% of minFree
  • Soft - 64%
  • Hard - 32%
  • Low - 16%

This is what the host does at when the threshold at the different stages is crossed:

  • When below High (< 400%), large pages are broken into 4KB pages
  • When below Clear, TSP is actively called instead of waiting for the next schedule TSP run
  • When below Soft, balloning begins
  • When below Hard, compression and swapping begins
  • At Low, blocking begins, and certain VMs are preventing from allocating memory

The mechanism:

  • Transparent Page Sharing (TSP); identify identical pages and move them into a single file; intra-VM TPS is enable by default (but for security inter-VM TPS is disable
  • Balloning; memory pages used by the VM are reclaimed by the host and given to another VM; VMware Tools ballon driver (vmemclt) is used for that. The host will only claim up to 65% of the VM memory, but this can be change in the VM vmx file Mem.ctlMaxPercent
  • Compression is a sign of contention, it only engages at hard and low
  • Swapping move ram from the physical ram to the disk, two types of swap can happens:
    • Guest OS swap; inside the guest, the famout pagefile.sys, this can occur at any time
    • VM Swapping; the VMkernel moves ram to the .vswp file of the VM, this happens at Hard and Low
  • Memory Overhead; the amount of ram the VMkernel uses to actually run the VM
  • Memory Consumed = assigned mem + overhead

Storage Metrics:

  • Physical Device Latency; should be less than 10 milliseconds, the time the device takes to process iSCSI commands
  • Kernel Latency; should be less than 1 millisecond, the time VMkernel process commands from VM to storage
  • Guest Latency; it should be less than 15 milliseconds

vCenter  tips

Cluster HA

To define additional isolation addresses for HA to ping to, add the advance command das.isolationaddress0 ;you can configure as well this setting: das.usedefaultisolationaddress

HA requires the following ports to be open

  • Inbound TCP and UDP ports 8042 to 8045
  • Outboud TCP and UDP ports 2050 to 2250

If HA takes a while to be configure on the host, it may timeout, if it does please extend the timeout periof of HA by adding this advance command to a value greater than the default of 240 seconds vpxd.das.electionWaitTimeSec

To increase the amount of time HA wait for a VM to shutdown after the power off command has been given, add a value greater than 300 seconds to the advance command das.isolationShutdownPeriod

To prevent false positive increase the grace period for network isolation here: fdm.isolationpolicydelaySec

HA uses a feature called Fault Domain Manager (FDM), and therefore it is not dependeable of vCenter to work, e.g. vCenter server can be down and HA will still work

If you don't have enough storage for the heartbeats, you can disable the warning by seeting das.ignoreInsufficientHbDatastore to true

HA, unlike DRS, does not respect anti-affinity rules unless: das.respectvmvmantiaffinityrules is set to true (default is false)

Here you can find advance settings for HA: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033250

 

DRS

Requires Enterprise edition

 

vSAN

Check this out for the default vSAN policy: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.virtualsan.doc/GUID-C228168F-6807-4C2A-9D74-E584CAF49A2A.html

Also here: https://docs.vmware.com/en/VMware-vSphere/5.5/com.vmware.vsphere.storage.doc/GUID-C8E919D0-9D80-4AE1-826B-D180632775F3.html

 

To access the VAMI, typ the port 5480 on the vCenter and logon as root: https://myvcenter.local:5480

The PSC (Platform Service Controller) GUI: https://myvcenter.local/psc

The Applicance Support Bundle (in case VC is unavailable): https://mycenter.local/applicance/support-bundle

VCSA typical CLI commands (applicanesh)

  • From a esxi host ssh command line, you can go to the vCenter by typing: # ssh root@myVcenterIPAddress
  • #df -h ; check the hard drive capacity of vCenter
  • #vimtop ;similar to esxtop

 

 

Fault Tolerance

Legacy FT and the new version of FT (Symetric Multi-Processing) called SMP-FT can coexist side by side

  • SMP-FT needs a 10Gbit network
  • Enterprise Plus is required to protect VM with 4 x vCPUs

 

AutoDeploy

It needs the following components

  • DHCP Server
  • PXE boot enable
  • TFTP server for the images
  • vCenter for the host profiles
  • Auto Deploy server for rules engines
  • Answer file (those are managed only through the vSphere Client, and not the web client)

A stateless installation; the hosts needs auto deploy to install on every single boot, thouhg it catches the install on USB drive, etc

A stateful installation; the host only needs auto deploy for the 1st boot, there after run from local installation

To connet to vCenter user PowerCLI run:

Get-ExecutionPolicy, and if needed Set-ExecutionPolicy Unrestricted

Connect-VIServer vcsa6.nazaudy.internal

 

CLI commands and Storage theory

For the ESXi hosts, a list of useful Linux commands:

#cat /etc/hosts ; shows the content of the hosts file

#cp /etc/hosts /var/tmp ;copy the file hosts to the location var > tmp

#vi /etc/hosts ;open the file with the vi text editor

  • Press Shit+Alt to start editing in a line above the cursor
  • Press "i" to start editing where the cursor is
  • Press "o" to start editiing in a line below the cursor
  • Press the letter "d" twice to delete a whole line
  • ESC + :q' > exit without saving
  • ESC + :wq > save changes, if you get error that file is read only, add ! at the end (:wq!)

When logging on to the vCenter by ssh, run this command to enable the shell

  • Command> shell.set --enable true
  • Command> shell
  • vol#

To fix problems converting host drives from SSD to HDD run: http://techhead.co/cannot-change-the-host-configuration-error-message-when-adding-disk-storage-to-a-vmware-vsphere-esxi-host/

  •  #ls- lha /vmfs/devices/disks
  • #partedUtil getptbl /vmfs/devices/disks/vml.0000000000766d686261313a333a30
  • #partedUtil setptbl /vmfs/devices/disk/vml.0000000000766d686261313a333a30 msdos

To shutdown a virtual machine using ssh from the host where it is running http://nigelhickey.com/power-vm-via-ssh/

  • #esxcli vm process list
  • #esxcli vm process kill -t [soft,hard,force] -w WorldNumber

To reboot a host:

  • esxcli system maintenanceMode set --enable true
  • esxcli system

 

Storage

iSCSI Software Initiator is the only one that support biridectional CHAP

Designate separates network adapters for iSCSI for performance and security

VSAN does not support IPv6

NFS4.1 is not compatible with SDRS, SIOC, SRM and VVOLs

NFS4.1 native multipathing for nic teaming policy is IP Hash; NFS4.1 is a big improvement in vSphere6, as before only 1 x IP was use to connect to a NFS share, now you can use Session Trunking and Multipahting

VASA (vSphere APIs for Storage Awareness), the storage send info about health, performace, etc to the vCenter; each vendor can setup its VASA provider differently

PSA (Pluggable Storage Architecture) configures and manages multipathing failover, it has the task of assigning to each storage device a MPP (Multi-Path Plugin) by using pre-define rules. There can be 2 types of MPP:

  1. NMP (Native Mutipathing Plugin), provided by VMware, it contains the following:
    1. SATP (Storage Array Type Plug), created by VMware for every array on the HCL
    2. PSP (Path Selection Policy), it selects which physical path to use for storage transport. VMware has 3 built-in PSPs:
      1. MRU (Most Recently Used) - VMW_PSP_MRU ;by default MRU is also selected for ALUA devices; this is the default for active/passive arrays
      2. Fixed - VMW_PSP_FIXED ;it's the default to active/active arrays, iSCSI and Fibre Channel
      3. Round Robin - VMW_PSP_RR; the default use for active/active arrays when multipathing is involved
  2. Third-Party MPP, supplied by the vendor storage

More info about multipathing considerations here: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-4D64F3DA-9701-4210-B34A-0A44D3A0100C.html and here too: https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.storage.doc/GUID-C1C4A725-8BE4-4875-919E-693812961366.html

 VAAI (vSphere APIs for Array Integration), first introduced in 4.1; is a method for offloading operations from the host to the array. VAAI has 3 main capabilities, called primitives:

  • Full Copy; the scsi extended copy command is replaced by the XCOPY command provided by VAAI
  • Block Zeroing
  • Hardware-assisted locking

VMCP (Virtual Machine Component Protection), it protects VMs when APD or PDL occurs

  • APD (All Paths Down), defaults to 140 seconds
  • PDL (Permanent Device Lost, added from vSphere5.0), after 140 seconds of APD the host stop sending I/O requests

VAAI - Storage APIs for Array Integration

Another improvement in vSphere6 is Bus Sharing: now you can set the HDD on a VM to be share with others

When "multi-writer locking" is enable, a VMDK is accessible by several computers, perfect for Microsoft Clustering

 

vFlash Read Cache (vFRC)

  • Act as tier between the VM and the Storage
  • Introduced in vSphere 5.5
  • The host needs to have at least 1 x SSD
  • Enterprise Plus license accepted only
  • The VM needs to be at least hardware version 10 (5.5 compatible)

 

Storage commands

#esxcli storage core adapter list ;list all storage drives

Any device to which you want to change the PSP, needs to have its path unclaim and then reclaim, for example the below command changes the default MRU to Round Robin, remember that after the change the host needs to be rebooted:

#esxcli storage nmp satp set -s VMW_SATP_CX -P VMW_PSP_RR

VASA = Awareness

 

VMware mosaic of appliances

vRealize Orchestator, how to access it

After you import the "VMware-vRO-Appliance-7.0.1.17606-3571217_OVF10.ova" file, open the console and take note of the Orchestator Server IP address and port:

 

 Then visit that address and click on "Start Orchestator Client"

 

 Open the aplication with Java, as prompted, and logon with the details "vcoadmin" and "vcoadmin"

You can also visit https://10.10.10.25:8283/vco-config/ and use "vmware" and "vmware" OR root and your own password

vRealize Operations, how to access it

Start by configuring all to DHCP, otherwise you may find this error about ./install.sh KB2150424

After the deploy, reduce the RAM size to 8GB or something that accomodates your host

It uses a few databases, Global xDB and File System Database (FSDB)

A vRealize Ops Manager cluster can contain multiple nodes:

  • Master
  • Master Replica
  • Data
  • Remote Collector

The 3 architectures components of vROps are:

  • Administrative Server
  • Analytics Server
  • Database Server

 

VDP (VMware Data Protection)

  • Visit https://10.10.10.90:8543/vdp-configure after installation
  • Initial password: changeme
  • Make sure time is set to: Europe\London
  • Run the installation wizard and attach to the vCenter
  • Use the vCenter plugin to connect to the VDP

NOTE: After pwoering on a recovered Windows VM running PSC, do not restart the server until you run the psc-restore script

 

VRA (vSphere Replicaton Appliance)

First of all import the OVF "vSphere_Replication_Server_SRM_OVF10" into vCenter, that will create the vRA Server from which you'll replicate to a remote site

After you logon, visit System > Time Zone and make sure time is set to: Europe\London

At the remote site, import hte vRA_AddOn_OVF10.ovf, and then register to it the vRA Server

vSphere Replication does not support VSS quiescing on VMs

 

vMA (Management Assistant)

The username for the vMA is "vi-admin"

How to add the vCenter and ESXi host to the vMA: https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vma.doc_50%2Fvima_get_start.4.12.html

Basicallly, you need to enter the command: ./credstore_admin.pl add --server [FQDN] --username [ME] --password [PASSWD]

  • To join domain: sudo domainjoin-cli join nazaudy.internal This email address is being protected from spambots. You need JavaScript enabled to view it.
  • After it has successfully joined, run sudo reboot to restart vMA
  • Ensure the servers (and obviously vCenter) are added to the domain first
  • Type vifp addserver host1.nazaudy.internal --authpolicy adauth --username nazaudy\\Administrator
  • Type vifp listservers to ensure the server has been added succesfully
  • Type vifptarget -s host1.nazaudy.internal to connect to the server and start issuing commands

 

London, 25 August 2017

Print Friendly, PDF & Email