VMware Ruby Virtual Console for vSAN 6.6

VMware Ruby Virtual Console for vSAN 6.6

The Ruby vSphere Console (RVC) is an interactive command-line console user interface for VMware vSphere and Virtual Center.

The Ruby vSphere Console comes bundled with both the vCenter Server Appliance (VCSA) and the Windows version of vCenter Server. RVC is quickly becoming one of the primary tools for managing and troubleshooting Virtual SAN environments

How to begin

  • To begin using the Ruby vSphere Console to manage your vSphere infrastructure, deploy the vCenter Server Appliance and configure network connectivity for the appliance.
  • Afterwards, SSH using Putty or an app you prefer to the dedicated vCenter Server Appliance and login as a privileged user. No additional configuration is required to begin.
  • Commands such as ‘cd’ and ‘ls’ work fine and if you want to return to the previous directory type ‘cd .. and press Enter

How to Login

RVC credentials are directly related to the default domain setting in SSO (Single Sign-On). Verify the default SSO Identity Source is set to the desired entity.

So there are a few different ways to logon potentially either locally or with domain credentials. Examples below

  • rvc administrator@vsphere.local@localhost
  • rvc root@localhost
  • rvc administrator@techlab.local@localhost

Where to go from here

You are now at the root of the virtual filesystem.

  • To access and navigate through the system type ‘cd 0‘ to access the root (/) directory or ‘cd 1‘ to access the ‘localhost/’ directory. You can type the ‘ls’ command to list the contents of a directory. I am going to type ‘cd 1‘ to access my localhost directory so lets see what we have.

  • Type ls to see what directory structure we have now. You should now see your datacenter or datacenters

  • Change directory by typing cd 0 to the relevant datacenter and you will now see the following folder structure.

  • Type ls to see the structure of this folder

  • Type cd 1 to change to the Computers folder where we will see the cluster and then type ls

  • We can now use a command to check the state of the vSAN cluster. You don’t want to enter the command ‘vsan.check_state vsan-cluster’ as that will not work. The number ‘0’ is what you need to use to look at the state of the cluster so type vsan.check_state 0

  • Next look at the vSAN Object Status Report. Type vsan.obj_status_report 0

  • We can also run the command vsan.obj_status_report 0 -t which displays a table with more information about vSAN objects

  • Next look at a detailed view of the cluster. Type vsan.cluster_info 0

  • Next we’ll have a look at disk stats. Type vsan.disks_stats 0

  • Next have a look at simulating a failure of a host on your vSAN cluster. type vsan.whatif_host_failures 0

  • You can also type vsan.whatif_host_failures -s 0

  • You can also view VM performance by typing vsan.vm_perf_stats “vm” This command will sample disk performance over a period of 20 seconds. This command will provide you with ‘read/write’ information IOPS, throughput and latency

Using vSAN Observer

Click me –> https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2064240

To generate a performance statistics bundle over a one hour period at 30 second intervals for a vSAN cluster named vSAN and save the generated statistics bundle to the /tmp folder, run this command:

  • Log into rvc
  • Navigate down to Computers
  • Type the following vsan.observer ~/computers/clustername(fill this in)/ –run-webserver –force –generate-html-bundle /tmp –interval 30 –max-runtime 1
  • While this is running, you can log into a web browser and run http://vCentername:8010 which will provide multiple graphs and information you can view
  • Press Control C to stop if you want to stop this prior to the test ending.

Inaccessible objects or orphaned objects

If you get an issue like I did with an orphaned object then browse through the vSAN datastore in the Web Client and find the GUID of the object and run the following command on the hosts. Take care you have the correct GUID!

  • /usr/lib/vmware/osfs/bin/objtool delete -u 5825a359-2645-eb1e-b109-002564f9b0c2 -f -v 10
  • Give it a minute and you will see it vanish from your vSAN datastore

Useful Commands

Within the RVC Console, type in vsan. then press tab twice to get the whole list of vsan commands you can use.

On the hosts the following commands can be useful

  • cd /etc/init.d/vsanmgmtd status
  • cd /etc/init.d/vsanmgmtd restart
  • services.sh restartcd  /vmfs/volumes then ls
  • vsish -e set /vmkmodules/vsan/dom/ownerabdicate “naa id”



Using HCI Bench v1.6.3 to performance test vSAN 6.6

vSAN Load Testing Tool: HCI Bench

VMware has a vSAN Stress and Load testing tool called HCIBench, which is provided via VMware’s fling capability. HCIbench can be run in versions 5.5 and upwards today as a replacement for the vSAN Proactive tests which are inbuilt into vSAN currently. I am running this against vSphere 6.5/vSAN 6.6 today. HCIBench provides more flexibility in defining a target performance profile as input and test results from HCIBench can be viewed in a web browser and saved to disk.

HCIBench will help simplify the stress testing task, as HCIBench asks you to specify your desired testing parameters (size of working set, IO profile, number of VMs and VMDKs, etc.) and then spawns multiple instances of Vdbench on multiple servers. If you don’t want to configure anything manually there is a button called Easyrun which will set everything for you. After the test run is done, it conveniently gathers all the results in one place for easy review and resets itself for the next test run.

HCIBench is not only a benchmark tool designed for vSAN, but also could be used to evaluate the performance of all kinds of Hyper-Converged Infrastructure Storage in vSphere environment.

Where can I can find HCI Bench?

There is a dedicated fling page which will provide access to HCIBench and its associated documentation. A zip file containing the Vdbench binaries from Oracle will also be required to be downloaded which can be done through the configuration page after the appliance is installed. You will need to register an account with Oracle to download this file but this doesn’t take long.

HCIBench Download: labs.vmware.com/flings/hcibench

HCIBench User Guidehttps://download3.vmware.com/software/vmw-tools/hcibench/HCIBench_User_Guide.pdf


  • Web Browser: IE8+, Firefox or Chrome
  • vSphere 5.5 and later environments for both HCIBench and its client VMs deployment

HCIBench Tool Architecture

The tool is specifically designed for running performance tests using Vdbench against a vSAN datastore.
It is delivered in the form of Open Virtualization Appliance (OVA) that includes the following components:

The test Controller VM is installed with:

  • Ruby vSphere Console (RVC)
  • vSAN Observer
  • Automation bundle
  • Configuration files
  • Linux test VM template

The Controller VM has all the needed components installed. The core component is RVC (https://github.com/vmware/rvc) with some extended features enabled. RVC is the engine of this performance test tool, responsible for deploying Vdbench Guest VMs, conducting Vdbench runs, collecting results, and monitoring vSAN by using vSAN Observer.


Before deploying this performance test tool packaged as OVA, make sure the environment meets the following requirements:

The vSAN Cluster is created and configured properly

  • The network for Vdbench Guest VMs is ready, and needs to have DHCP service enabled; if the network doesn’t have DHCP service, “Private Network” must be mapped to the same network when HCIBench being deployed.
  • The vSphere environment where the tool is deployed can access the vSAN Cluster environment to be tested
  • The tool can be deployed into any vSphere environment. However, we do not recommend deploying it into the vSAN Cluster that is tested to avoid unnecessary resource consumption by the tool.

What am I benchmarking?

This is my home lab which runs vSAN 6.6 on 3 x Dell Poweredge T710 servers each with

  • 2 x 6 core X5650 2.66Ghz processors
  • 128GB RAM
  • 6 x Dell Enterprise 2TB SATA 7.2k hot plug drives
  • 1 x Samsung 256GB SSD Enterprise 6.0Gbps
  • Perc 6i RAID BBWC battery-backed cache
  • iDRAC 6 Enterprise Remote Card
  • NetXtreme II 5709c Gigabit Ethernet NIC

Installation Instructions

  • Download the HCIBench OVA from https://labs.vmware.com/flings/hcibench and deploy it to your vSphere 5.5 or later environment.
  • Because the vApp option is used for deployment, HCIBench doesn’t support deployment on a standalone ESXi host, the ESXi host needs to be managed by a vCenter server.
  • When configuring the network, if you don’t have DHCP service on the VLAN that the VDBench client VMs will be deployed on, the “Private Network” needs to be mapped to the same VLAN because HCIBench will be able to provide the DHCP service.
  • Log into vCenter and go to File > Deploy OVF File

  • Name the machine and select a deployment location

  • Select where to run the deployed template. I’m going to run it on one of my host local datastores as it is recommended to run it in a location other than the vSAN.

  • Review the details

  • Accept the License Agreement

  • Select a storage location to store the files for the deployed template

  • Select a destination network for each source network
  • Map the “Public Network” to the network which the HCIBench will be
    accessed through; if the network prepared for Vdbench Guest VM doesn’t have DHCP service, map the “Private Network” to the same network, otherwise just ignore the “Private Network”.

  • Enter the network details. I have chosen static and filled in the detail as per below. I have a Windows DHCP Server on my network which will issue IP Addresses to the worker VMs.
  • Note: I added the IP Address of the HCIBench appliance into my DNS Server

  • Click Next and check all the details

  • The OVF should deploy. If you get a failure with the message. “The OVF failed to deploy. The ovf descriptor is not available” then redownload the OVA and try again and it should work.

  • Next power on the Controller VM and go to your web browser and navigate to your VM using http://<Your_HCIBench_IP>:8080. In my case Your IP is the IP address you gave it during the OVF deployment or the DHCP address it picked up if you chose this option. If it asks you for a root password, it is normally what you set in the Deploy OVF wizard.
  • Log in with the root account details you set and you’ll get the Configuration UI

  • Go down the whole list and fill in each field. The screen-print shows half the configuration
  • Fill in the vCenter IP or FQDN
  • Fill in the vCenter Username as username@domain format
  • Fill in the Center Password
  • Fill in your Datacenter Name
  • Fill in your Cluster Name
  • Fill in the network name. If you don’t fill anything in here, it will assume the “VM Network” Note: This is my default network so I left it blank.
  • You’ll see a checkbox for enabling DHCP Service on the network. DHCP is required for all the Vdbench worker VMs that HCIBench will produce so if you don’t have DHCP on this network, you will need to check this box so it will assign addresses for you. As before I have a Windows DHCP server on my network so I won’t check this.

  • Next enter the Datastore name of the datastore you want HCIBench to test so for example I am going to put in vsanDatastore which is the name of my vSAN.
  • Select Clear Read/Write Cache Before Each Testing which will make sure that test results are not skewed by any data lurking in the cache. It is designed to flush the cache tier prior to testing.
  • Next you have the option to deploy the worker VMs directly to the hosts or whether HCIBench should leverage vCenter

If this parameter is unchecked, ignore the Hosts field below, for the Host Username/Password fields can also be ignored if Clear Read/Write Cache Before Each Testing is unchecked. In this mode, a Vdbench Guest VM is deployed by the vCenter and then is cloned to all hosts in the vSAN Cluster in a round-robin fashion. The naming convention of Vdbench Guest VMs deployed in this mode is
If this parameter is checked, all the other parameters except EASY RUN must be specified properly.
The Hosts parameter specifies IP addresses or FQDNs of hosts in the vSAN Cluster to have Vdbench Guest VMs deployed, and all these hosts should have the same username and password specifed in Host Username and Host Password. In this mode, Vdbench Guest VMs are deployed directly on the specified hosts concurrently. To reduce the network traffic, five hosts are running deployment at the same time then it moves to the next five hosts. Each host also deploys at an increment of five VMs at a time.

The naming convention of test VMs deployed in this mode is “vdbench-<HOSTNAME/IP>-<DATASTORE_NAME>-batch<VM#>-<VM#>”.

In general, it is recommended to check Deploy on Hosts for deployment of a large number of testVMs. However, if distributed switch portgroup is used as the client VM network, Deploy on Hosts must be unchecked.
EASY RUN is specifically designed for vSAN user, by checking this, HCIBench is able to handle all the configurations below by identifying the vSAN configuration. EASY RUN helps to decide how many client VMs should be deployed, the number and size of VMDKs of each VM, the way of preparing virtual disks before testing etc. The configurations below will be hidden if this option is checked.

  • You can omit all the host details and just click EASYRUN

  • Next Download the vDBench zip file and upload it as it is. Note: you will need to create yourself an Oracle account if you do not have one.

  • It should look like this. Click Upload

  • Click Save Configuration

  • Click Validate the Configuration.Note at the bottom, it is saying to “Deploy on hosts must be unchecked” when using fully automated DRS. As a result I changed my cluster DRS settings to partially automated and then I got the correct message below when I validated again.

  • If you get any issues, please look at the Pre-validation logs located here – /opt/automation/logs/prevalidation

  • Next we can start a Test. Click Test

  • You will see the VMs being deployed in vCenter

  • And more messages being shown

  • It should finish and say Test is finished


  • After the Vdbench testing finishes, the test results are collected from all Vdbench instances in the test VMs. And you can view the results at http://HCIBench_IP/results in a web browser and/or clicking the results button from the testing window.
  • You can also click Save Result and save a zip file of all the results
  • Click on the easy-run folder

  • Click on the .txt file

  • You will get a summarized results file

  • Click on the other folder

  • You can also see the individual vdBench VMs statistics by clicking on

  • You can also navigate down to what is a vSAN Observer collection. Click on the stats.html file to display a vSAN Observer view of the cluster for the period of time that the test was running

  • You will be able to click through the tabs to see what sort of performance, latency and throughput was occurring.

  • Enjoy and check you are getting the results you would expect from your storage

Useful Links

  • Comments from the HCIBench fling site which may be useful for troubleshooting


  • If you have questions or need help with the tool, please email VSANperformance@vmware.com
  • Information about the back-end scripts in HCIBench thanks to Chen Wei

Use HCIBench Like a Pro – Part 2



Backup and Restore of VCSA 6.5

Backing up and Restoring VCSA

Backup steps

  • Log into the VCSA appliance screen. In my case
  • Select Backup from the Summary Screen

  • Put in the backup details. As a test I have set up a Filezilla FTP server on a Windows box to use as my backup location

You can choose to optionally encrypt the backup with a password. This uses AES256 encryption

  • Click Next and it will validate the inputs

  • Choose what to backup

  • Click Next and Complete the Backup

  • The Backup should run and complete successfully

  • Check the backup files exist in the ftp directory you specified

Restoring a VCSA 6.5 appliance

The one thing to remember is that is my lab environment. I am going to power off and unregister my current VCSA to simulate that there has been an unrecoverable failure as I am going to need to restore an identical named and IP addressed machine. Obviously you can’t have 2 VMs registered with the same name in the inventory or the same IP address so I have temporarily shutdown and unregistered my current VCSA.

  • Mount the vCSA installer ISO and run installer.exe from \vcsa-ui-installer\win32 assuming you’re running this on Windows.
  • Click Restore

  • The Restore Wizard will start and you will see the below screen
  • Click Next

  • Accept the License Agreement

  • Fill in the details for the backup file

  • Check the details

  • Click Next
  • Put in the host details for deploying the new VCSA

  • Accept the Certificate

  • Set up the target appliance VM

  • Set the deployment size

  • Select the datastore to restore to. In my case I have a vSAN or a Local datastore to restore to

  • Enter the IP Address details.

  • Click Next
  • Check the Details and click Ready to Complete

  • It will say initializing

  • It will finally complete and should say

  • Click Continue

  • Check the details

  • Complete the second stage of the restore

  • Take note of the warning

  • The restore should start

  • Once the restore has fully finished, you should see the below screen

  • Check the host for the newly restored VCSA

Useful Links

3 x Dell Poweredge T710 lab with a bootstrapped install of vCenter 6.5, embedded PSC and vSAN 6.6

vCenter 6.5/vSAN 6.6 new install

This is a blog based on my Dell Poweredge T710 lab which I’ve set up to take advantage of testing vSphere 6.5 and vSAN 6.6 as a combined install of a new installation which should bootstrap vSAN, create a vCenter and then place the vCenter on the vSAN automatically.

Note: vSAN will be a hybrid configuration of 1 x SSD and 6 SATA hot plug drives per server.

New integrated bootstrapping feature explained

In some environments where new hardware being deployed, high availability shared storage may not be accessible during day-zero installation meaning if you were building a greenfield deployment, it was almost a catch 22 scenario. How did you build your vSAN with a vCenter server when you only had the disks for a vSAN deployment. There were ways around this via command line but it has now been built into the functionality of vSphere 6.5/vSAN 6.6.

Local disk, if available, can be used as a temporary location for vCenter installation, but vCenter migration after bringing up the cluster could be time consuming and error prone. Bootstrapping vSAN without vCenter can solve this problem and remove the requirement to have high availability storage or temporary local disk at day-zero operations. This could be applicable to a greenfield deployment scenario. With the Bootstrapping vSAN method, a vSAN based datastore can be made available at day-zero operation to bring-up all management components.

Lab Setup

3 x Dell Poweredge T710 servers each with

  • 2 x 6 core X5650 2.66Ghz processors
  • 128GB RAM
  • 6 x Dell Enterprise 2TB SATA 7.2k hot plug drives
  • 1 x Samsung 256GB SSD Enterprise 6.0Gbps
  • Perc 6i RAID BBWC battery-backed cache
  • iDRAC 6 Enterprise Remote Card
  • NetXtreme II 5709c Gigabit Ethernet NIC

Initial Steps for each 3 hosts

  • The Perc 6i controller is not on the vSAN HCL but vSAN can still be setup using RAID0 passthrough which involves configuring a RAID0 volume for each drive in the BIOS (Ctrl + R at bootup) Always make sure the drive is initialized in the BIOS which clears any previous content because vSAN requires the drives to be empty. Press Control > R during boot up and access the Virtual Disk Management screen to create disks as RAID0. See the link below for full information


  • In the System Setup BIOS screen you will need to enable Virtualization Technology.  Not enabled by default and will stop any VMs from powering on if not enabled

  • Make sure you have an AD/DNS Server with entries for your hosts and vCenter
  • Put in your license keys
  • Disks may not come up marked as SSD. In this case I had to run the following commands on each server (Replace your disk naa id with yours and the SATP Type)

Find your disk information as per below command but you can also find the disk ID’s in the host client


  • Your SSD disks should then come up marked as SSD. I didn’t have to reboot.

Install the vCenter Appliance

  • Make sure you have the software downloaded. I’m using the VMware-VCSA-all-6.5.0-5705665.iso
  • On another machine, mount the VMware-VCSA-all-6.5.0-5705665.iso. I connected this to my Windows 10 laptop as a virtual drive. Start the vCenter Server Appliance 6.5 installer located at \vcsa-ui-installer\win32

  • Select Install from the VMware vCenter Server Appliance 6.5 Installer.

  • You will see the Introduction screen

  • Accept the License Agreement
  • Select Deployment Type. For now I’m going to use an embedded Platform Service Controller

  • Enter the details for the appliance target. Try an IP Address if a FQDN doesn’t work.

  • Accept the certificate

  • Put in a root password for the vCenter Server Appliance

  • Select a deployment size

  • There are now 2 deployment types. You can install as normal or you can “Install on a new Virtual SAN cluster containing the target host”

  • I am going to test this new feature of a combined install of vCenter and vSAN placing the vCenter on vSAN
  • Put in a name for your Datacenter and Cluster and click Next. It will say Loading

  • Claim disks for Virtual SAN. You can see it has picked up all my disks on my first host and recognizes the SSD and sets it as a cache Disk while the other non SSD Disks are set as Capacity Disks

  • Next enter your network settings

  • You are now ready to complete at Stage 1. Check the settings and click Finish

  • It will now show the following screen

  • When it has finished you should see the below screen

  • Click Continue and we will be on to Stage 2

  • Next Enter the time settings. You can use NTP Servers or sync with the local host. You can also enable SSH

  • Next set up the embedded PSC

  • Next decide if you want to join the Customer Experience Program

  • Finish and check the config

  • You should now see the below screen

  • When it has finished you will see the below screen

  • Next connect to the vCenter appliance with the administrator@vsphere.local account and the password you set up previously


  • Select the Host > Select Configure > Select Networking > VMkernel Adapters

  • Select a switch

  • Add a VMkernel adapter for vSAN

  • Specify VMkernel networking drivers

  • Check Settings and Finish

  • Next I need to add my other 2 hosts to the Datacenter and create a vSAN VMkernel port on each host followed by adding them into the cluster

  • Click on the cluster > Select Configure > vSAN > Disk Management and select your disks on the other servers and make them either the cache disk or capacity disk

  • This process is normally quite quick and once complete you should have your vSAN up and running!

  • Click on the cluster > Select Configure > Services and Edit Settings to turn on HA and DRS

  • Once everything is looking ok click on the cluster > vSAN > General > Configuration Assist to check any errors or warnings about any issues so you can fix these.

Procedure to shutdown the vSAN cluster and start it up again.

So it crossed my mind that as this is my lab, it is not going to be running 24×7 or my house is going to be rather warm and my electricity bill will definitely rise! I need to power it off therefore what is the correct way to shut everything down and power up again?

Normally to shutdown an ESXi Cluster, using vCenter Webclient, ESXi hosts have to be put into maintenance mode and then ESXi hosts are powered off. Also to start ESXi Cluster, vCenter Server is used to remove them from maintenance mode after powering on the hosts. However if the VSAN cluster is running management components such as vCenter Server and other management VMs, the ESXi host that is running vCenter Server cannot be put into maintenance mode. So vSAN Cluster shutdown and starting procedures have to be properly sequenced.

VMware KB


Steps for Power Off

  • Start by powering off all virtual machines on ESXi cluster except the vCenter Server. If your Management cluster has ActiveDirectory which provides services to vCenter Server, then do not power off Active Directory VM as well
  • Migrate the vCenter Server VM and ActiveDirectory VM(s) to a single ESXi Host
  • Place all the remaining hosts in the cluster into maintenance mode. When confirming the maintenance mode for ESXi Host, ensure the following selection is made ( deselect checkbox for Move powered-off VMs and “No data migration” is chosen for Virtual SAN data migration)
  • You can put the hosts in Maintenance Mode manually as per the baove step or you can use a command line command. You can run the ESXCLI command below to put a host in Maintenance mode. However, you must perform this operation through one of the CLI methods that supports setting the VSAN mode when entering Maintenance Mode. You can either do this by logging directly into the ESXi Shell and running ESXCLI.

esxcli system maintenanceMode set -e true -m noAction

other options are

esxcli system maintenanceMode set -e true -m ensureObjectAccessibility

esxcli system maintenanceMode set -e true -m evacuateAllData

  • Power off the vCenter Server VM and Active Directory VM. At this point, the vSphere WebClient access is lost.
  • Shutdown all ESXi Hosts. This will complete the Shutdown procedure for VSAN Cluster.

Starting the ESXi Hosts and the vSAN back up

The procedure to start a vSAN Cluster begins with the ESXi host where vCenter Server and Active Directory VMs are running.

  • Power on all ESXi hosts in the cluster.
  • Take the hosts out of maintenance mode
  • Identify ESXi host where vCenter Server and Active Directory VMs are located
  • Power on AD servers
  • Power on vCenter server
  • Note: If the vCenter Server VM has a vmnic that is tied to a VDS  network, then vCenter Server can’t be powered on because VM power-on operation on VDS requires vCenter Server to be running. So it is recommended to move any vmnic on vCenter Server to a vSwitch-based network. This can be moved back to the VDS Network afterwards
  • Log into vCenter and check vSAN

Useful Troubleshooting Tools

  • rvc console on vCenter
  • Putty
  • esxcli commands

I had an issue at a customer site where they had put some hosts in Maintenance Mode and when they brought them out again, the hosts came out of Maintenance Mode but the vSAN didn’t resulting in the misreporting of storage in your cluster. As a result storage policies will error and you won’t be able to put any more hosts in maintenance mode if there isn’t the visible storage to move them. Note: you won’t have lost any storage. The system will just think it’s not there until you put the host into Maintenance Mode and take it out again for a second time! VMware are aware of this issue which seems to be present in 6.5U1 however this was a customers automated system. I haven’t seen this happen in my home lab!

By running the command vsan.cluster_info 0 in rvc, you are able to see for every disk whether the node is evacuated or not. If you have taken the host out of Maintenance Mode and the vSAN has also come out of Maintenance Mode then it will say Node evacuated: no. If it hasn’t come out properly it will say Node evacuated: yes (won’t accept any new components)




Rolling back and recovering from failed vSphere 6 and external multisite PSC upgrades

Recovering from failed vSphere 6 and external multisite PSC upgrades 

So what happens if you have the following situation within a failed vSphere upgrade with a multi-site system?

The Scenario

The requirement was to upgrade 3 vCenters from 5.1U3 to 6.0U2. The vCenter servers previously had embedded SSO but have been repointed to the external 6.0 U2 PSCs. Note, we had an intermediary stage of repointing to 5.5U3 external SSOs first before we could upgrade the PSCs to 6.0U2. So once the PSCs are in a multisite 6.0U2 configuration, the upgrade of the vCenters can start and this is where we need to take care as the systems are interlinked at this point.

So we now have

  • 3 x vSphere 6.0U2 PSCs set up in a multisite configuration
  • 3 x 5.1 U3 vCenters pointing to a vSphere 6.0U2 PSC
  • 3 x SQL 2008 R2 Databases running the vCenter Databases and the Update Manager Databases. Not seen in the example above

Why did it fail initially?

Never assume anything. We had already upgraded 2 environments without any issue so we overlooked checking the SQL environments which were meant to be a replica but clearly weren’t!

  • DB password had special characters in the password.
  • Needed to give db owner rights on the vCenter DB and the MSDB database prior to upgrading
  • The ODBC Connection was not the right version. We needed SQL Native Client 10 or 11.
  • Additional Database Permissions for VMware (Can be seen in the vSphere 6.0 Documentation Center
  • SQL Server 2008 R2 needed Service Pack 2 installing. (No problem with this)
  • An informational messages regarding the vCenter FQDN
  • An informational message about the ALL_SERVICES accounts. (New in vSphere. Depends if you want to use Local System for all your vCenter Services or use the multiple vSphere accounts for individual services

So how do you recover from one failed vCenter Upgrade in this scenario?

We do need to take into account at what step has it failed and check the logs to verify this. Can it be an issue that can be worked though and fixed forward or do you need to rollback and depending on how far the installer has got before failing, it might open variations in the options for a rollback.  For example, if it fails to authenticate to the database, then it is unlikely it would have made any DB changes. The installer logs will give us this information.

You can retrieve the installation log files manually for examination.



Navigate to the installation log file locations.

%PROGRAMDATA%\VMware\vCenterServer\logs directory, usually C:\ProgramData\VMware\vCenterServer\logs

%TEMP% directory, usually C:\Users\ username\AppData\Local\Temp

The files in the %TEMP% directory include vminst.log, pkgmgr.log, pkgmgr-comp-msi.log, and vim-vcs-msi.log.


Open the installation log files in a text editor for examination.


  • Make sure you have been through all the pre-requisites prior to starting the upgrade and test in a lab as well. Give yourself the best start at not failing.
  • Snapshot the whole environment (All VCs, All DBs and all All PSCs) Make sure the environment is quiesced or if you want a more consistent image vs crash consistent, shutdown the whole environment and snapshot it cold
If the upgrade of any VC 5.x to 6.0 fails in this environment, you must roll back ALL PSC.  The reason is that the solution user format differs between 5.x and 6.x. During the upgrade of the VC, the VC 5.x solution users will be removed from the PSC and replaced with 6.x solution users early in the VC 6.x upgrade (during vmafd firstboot I believe). If you have a failure and only roll back the VC then you have a VC with 5.x solution users talking to a PSC that no longer is aware of those users. You must roll back ALL PSCs in the SSO Domain as they replicate.
  • If you encounter an unrecoverable upgrade installer error then in the event of rolling back to snapshots, the order will be all PSCs, the vCenter DB and vCenter Server
  • In the event the rollback above fails, all servers should be rolled back again with the order of power on being all PSCs, all vCenter DBs and all vCenter servers

Questions we have been asked

During an upgrade, could we stop/break the multisite replication agreements between PSCs to avoid any replication of issues in the event of an upgrade problem on one vCenter? 

There’s no issue with breaking replication agreements generally but it not something that should be done for an upgrade. The vdcrepadmin command line tool does allows the breaking and creating of agreements and it actually protects the customer by not allowing them to delete the only agreement available. This prevents a customer from inadvertently creating an isolated PSC. What you would do is go through and create the new agreements and once they are in place just delete the extra ones. There is nothing special about the replication agreements that are created during PSC deployment. They are the same as ones created with vdcrepadmin.




PSC 6 Replication for multi-site setups

PSC 6 Replication 

VMware Validated Designs

First of all I’d like to talk about VMware Validated Designs before going on to talk about how these can be incorporated into replicated PSC designs.

VMware Validated Designs deliver holistic data center-level designs that span compute, storage, networking and management, defining the gold standard for how to deploy and configure the complete VMware SDDC stack in a wide range of use cases. VMware Validated Designs include detailed guidance that combine with best practices on how to operate a VMware SDDC optimally. The benefits are

  • Accelerate time to market
  • Increase efficiency
  • De risk deployments and operations
  • Drive agility
  • Repeatable proven solutions

When looking to implement any software from the SDDC suite, we are now referring to the VMware Validated Designs for a repeatable proven solution. Of course there may always be modifications required to fit customers environments but the idea going forwards is to maintain the same standards of installation and configuration

Platform Services Controller Topology Decision Tree

Adam Eckerle from VMware has very kindly designed a poster containing a decision tree for anyone thinking of deploying PSCs and guides you through a set of decisions before deployment


PSC and Replication Points

  • All PSC replication is bi-directional but ring topology setups have to be manually created
  • Ring topology is now the recommended method, not mesh as it was previously
  • By default each PSC is replicating with only a single other PSC (the one you select when installing the additional secondary or tertiary PSC)
  • Site names do not have anything to do with replication today they are a logical construct for load balancers and future usage
  • Changes are not unique to a site but to a domain assuming they are part of the vsphere.local domain
  • vCenter only points to one PSC at a time or a load balanced address
  • PSC’s have to be part of the same domain together to use enhanced linked mode

Why Ring Topologies and not Mesh Topologies?

  • Ring Topologies allow for easier scaling out
  • They are easier to set up and maintain. There’s no issue with breaking and recreating replication agreements. vdcrepadmin does this and it actually protects the customer by not allowing them to delete the only agreement available. This prevents a customer from inadvertently creating an isolated PSC. What you would do is go through and create the new agreements and once they are in place just delete the extra ones. There is nothing special about the replication agreements that are created during PSC deployment. They are the same as ones created with vdcrepadmin
  • The failure of any one component in a ring does not cause replication partitions
  • Minimal links can be created for maximum redundancy which is increased when load balancers are used.

2 Site VVD Design

In the diagram below, you can see within each region/site, the VMware Validated Design instantiates two Platform Services Controllers and two vCenter Server systems in the appliance form factor. This includes one PSC and one vCenter for the management pod and one PSC and one vCenter for the shared edge and compute pod. The design also joins the Platform Services Controller instance to its respective Platform Services Controller instance.

3 Site VVD Design

When we start getting into a 3 site multisite design, we can then see how keeping it simple makes life a great deal easier. Below is a series of diagrams which show how this can be set up with a recommended ring topology.

This particular example uses 3 sites with 2 PSCs in each. There is no load balancer.


  • When initially installing the external PSCs, PSCA will be the first standalone PSC. PSCB will be installed next and partnered with PSCA. PSCC is then installed and partnered with PSCB. This is 2 way automatic replication in an inline configuration

  • If we wanted to make it a ring topology with just these 3 servers we could do the following and create a replication agreement between PSCA and PSCC

  • Now we add the second PSCs into each site and we get the following scenario. 2 way automatic links are created between the first and second PSC in each site (A and D) (B and E) and (C and F)

  • Now we see we have kind of lost our inline topology and how do we create a ring now? We want an inline topology of D > A > B > E > C > F
  • We actually need to create a new agreement between PSCE and PSCC followed by breaking the link between PSCB and PSCC. Remember the system will not let us break a replication agreement which would leave a PSC isolated so we need to create a link first

  • We will now get the following inline configuration

  • Next we want to create our ring topology. A replication agreement is created between PSCD and PSCF to create the ring


Stratoscale explains IaaS, PaaS and SaaS – “The Good”, the “Bad” or the “Ugly”?

Stratoscale asked 32 IT experts to share their insights on the differences between IaaS, PaaS and SaaS in the great article below.

Please enjoy the insights on “The Good”, the “Bad” or the “Ugly”? within IaaS, PaaS and SaaS.

IaaS, PaaS, and SaaS: The Good, the Bad and the Ugly

Cloud computing is a broad term for the various IT-related services that can be provided on demand using a consumption based model.

The three most common cloud computing models:

  • Infrastructure as a Service (IaaS)
  • Platform as a Service (PaaS)
  • Software as a Service (SaaS)

There are countless articles covering cloud services. Yet, you might still be confused about what the heck all these “as a service” terminologies are. You are not alone, we hope that after you read what 30+ cloud experts have to say about their value, advantages, benefits and best practices things will be much clearer.


Replacing vROps 6.4 self signed certificates

Replacing vROps 6.4 self signed certificates

Certificate requirements – Click the link

  • File encoded in PEM format
  • Certificate is valid for Server Authentication
  • All certificates in the chain are included
  • The private key is included (2048 bit RSA or higher)
  • The private key is not secured with a password

VMware KB 2046591 Link – Click here

Step 1

  • To create the Certificate Request first download OpenSSL for Windows and install it in the default location – C:\OpenSSL-Win64
  • Note: Path and name may be slightly different depending on the version
  • I used Win64OpenSSL_Light-1_1_0e.exe

Step 2

  • Copy and paste the following into a text file and save as vrops.cfg in a folder called c:\OpenSSL-Win64\Certs. You will need to create the certs folder. You will be able to place other certs, keys and requests in here also

  • Customize the bold fields.

[ req ]

default_md = sha512
default_bits = 4096
default_keyfile = vropsprivate.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: Loadbalancedfqdn, IP: Loadbalancedip, DNS: Loadbalancedshortname, DNS: FirstNodeFQDN, IP: FirstNodeip, DNS: FirstNodeShortname, DNS: SeondNodeFQDN, IP: SecondNodeip, DNS: SecondNodeShortName

[ req_distinguished_name ]
countryName = countryName
stateOrProvinceName = stateOrProvinceName
localityName = localityName
0.organizationName = organizationName
organizationalUnitName = organizationalUnitName
commonName = Loadbalancedfqdn

In my case I don’t have a load balancer. I just have a Master node, a Replica Node and a Remote Collector Node so it looks like this

[ req ]

default_md = sha512
default_bits = 4096
default_keyfile = vropsprivate.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: techlabops001.techlab.local, IP:, DNS: techlabops001, DNS: techlabops002.techlab.local, IP:, DNS: techlabops002, DNS: techlabops003.techlab.local, IP:, DNS: techlabops003

[ req_distinguished_name ]
countryName = UK
stateOrProvinceName = London
localityName = Norfolk
0.organizationName = Techlab
organizationalUnitName = vROps
commonName = techlabops001.techlab.local

Step 3

Open a command prompt and change directory to where OpenSSL is installed.  You need to first generate a new key pair with a 2048 or 4096 key length. Leave this file in the path you specify here for later

openssl genrsa -out key_filename.key 4096

c:\openSSL\bin>openssl.exe genrsa -out c:\openSSL\certs\vropsprivate.key 4096

Step 4

Use the key to generate a certificate signing request by running this command. I have added -sha512 but this is optional

openssl req -new -key key_filename.key -out certificate_request.csr -config config_file.cfg

openssl req -new -key c:\certs\vropsprivate.key -out c:\certs\vropscsr.csr -config C:\OpenSSL\certs\vrops.cfg -sha512

Step 5

  • Click Windows Start > Run, enter certtmpl.msc, and click OK.
  • Follow the below link to create a certificate template (If you haven’t got one already)

vCSA (vCenter Server Appliance) – Part 4 – Deploy CA-signed certificates

Step 6

Submit the CSR file to your Certificate Authority (CA) to obtain a signed certificate

  • Browse to your CA – https://servername/certsrv – Request a certificate. In my case I have a Microsoft CA installed on my Domain Controller
  • Choose Request a Certificate

  • Chose Request a Certificate

  • Open the request file in a notepad and copy the contents into the request box, In my case this is the vropscsr.csr file.
  • Select a certificate template that meets the requirement. In this case vROps Template

  • Submit the request and download as Base 64 Encoded

  • Download certificate and save it locally to your machine as “certnew.cer”

Step 7

Now we need to download the certificate chain from our CA

  • Browse to your CA – https://servername/certsrv
  • Select Download a CA certificate, certificate chain or CRL

  • Click Download CA certificate chain

  • It will download a certnew.p7b file or if it downloads a .cer file then use this
  • Change the name to root64.p7b and Click Open or save it and then open it. The Certificate manager Console will open

  • We now need to extract the certificates for the root and all intermediate authorities in the certificate chain. For each CA, export the certificate to a file.
  • Right click on the root certificate and select Export

  • In the export wizard, select Base-64 and save the file to the same folder as vrops.cfg. Do that for each authority!

  • Once you receive the certificate you have all the components needed to create the required .pem certificate.

Note: The order of CA’s certs in the .PEM file: Private Key, Cert, Intermediate Cert and then Root Cert which can seen above and described below

  • Private Key = vropsprivate.key
  • Cert = certnew.cer
  • Intermediate Cert – I don’t have one
  • Root CA – root64.cer

Step 8

  • Check you have all your certs in C:\OpenSSL\Certs. Note I already have a .pem file from other testing so ignore this as you won’t have one yet.

Step 9

Next go back to SSL and generate the .pem certificate. Example below

type key_filename.key server_cert.cer cacerts.cer > multi_part.pem

  • Change directory to where you saved your certificates
  • Type the following (including the word type!)

type vropsprivate.key certnew.cer root64.cer > vrops.pem

  • You should now see your .pem file which in my case is vrops.pem

  • If you open it, it will look something like this

Step 10

We can now navigate to the vROps page and upload the .pem file into the system

  • You have to log in specifically to the small administration website (https://vrops-server/admin), and click on the small icon on the top right of the screen as seen below. Then you can choose to Install new certificate.

  • When you click Install, you will be disconnected and reconnected. Better close the tab and reopen it to check that the custom certificate has been imported properly.
  • You should be able to open the vROps web pages without any certificate errors 🙂 See the green lock on the Firefox browser.

  • If it is still saying it is insecure then you may have to import the root64.cer into the Firefox browser. Go to Options > Advanced > Certificates > View Certificates and import the root64.cer
  • You can see my Microsoft CA Authority here which is now trusted

  • Note: I couldn’t get this working in Chrome even after importing the root CA certificate. I think Google have locked down their websites for vulnerabilities in the SHA1 algorithm however I haven’t figured out a way to get around this yet

If you do have problems and cannot get into the vROps webpages for any reason, VMware has KB2144949 which will allow you to revert back to the default certificates.


Using JXplorer to connect to vSphere PSC Servers

Using JXplorer to connect to vSphere PSC Server

JXplorer is a cross platform LDAP browser and editor. It is a standards compliant general purpose LDAP client that can be used to search, read and edit any standard LDAP directory, or any directory service with an LDAP or DSML interface.

Note: Please take extreme care when connecting to the vmdird database. This is not a recommended way of viewing this data but it can be very useful

JXplorer Download Location

I installed this on a Windows 10 workstation which had connectivity to my PSC Servers


Configuration Steps

  • Open JXplorer and open a new Connection
  • In the Host field put in the same of your PSC/SSO Server
  • Don’t put anything in Base DN. It will let you connect if you do put a Base DN in but will error when you try and expand the tree
  • In Security Level, choose User + Password
  • In User DN, type CN=Administrator,CN=Users,DC=vsphere,DC=local
  • Put in your PSC/SSO password
  • Save the template

  • You should now see the following screen

  • Expand vSphere > Configuration > Sites and you should be able to see all the replication agreements.
  • I’ve been playing around with multi-site scenarios which is why you can see Default-First-Site, Default-Second-Site and Default-Third-Site which are my 3 PSCs in a multisite scenario.

Other Observations

Information from Sung Rao (VMware) Thank Q

In 5.5. the only secure LDAP communication between SSO/PSC nodes are via LDAPS.  Thus in automatic replication agreements establishment, LDAPS is used.

In 6.0 and after, we introduced LDAP SASL/SRP binding which go through port 389. LDAP SASL/SRP (or KRB) is the simple and safe to manage between LDAP nodes. This binding mechanism is preferable to LDAPS as the SSL port is difficult to manage/deploy correctly as it depends on PKI. Also, LDAP layer sites below certificate. You need an ID before you can get a cert

Regardless, in 6.0 and after, the server will try SASL/SRP first and fall back to LDAPS if necessary regardless of LDAP/LDAPS in the labeledURI in the replication agreement definition. You also cannot force the replication agreements to use LDAPS in 6.0 and after

IaaS, PaaS and SaaS explained

“IaaS, PaaS and SaaS explained

IaaS, PaaS, SaaS… It can sometimes feel like we are in acronym hell. What are the correct definitions and how does each service differ?

IT organizations have historically been using their own Private Cloud Infrastructure either on premise or inside their own external datacenters to host servers, applications and data etc. Move forward 20 years and we have moved into a service provider era containing Public Cloud Infrastructure and Hybrid Public Cloud Infrastructure. Organizations called service providers exist specifically to provide, manage and maintain the infrastructure on which their client organization’s servers, application or data are hosted. The client organization gets access controls to manage their servers, applications and data hosted on the remote server. This is the basis behind cloud computing so where does IaaS, PaaS, SaaS fit into this scenario?

In an IaaS (Infrastructure as a Service) model sometimes called Hardware as a Service, a third-party service provider hosts hardware, software, servers, storage, networking and other infrastructure components on behalf of an organization; including managing tasks such as system maintenance, upgrading, backup and disaster recovery planning.

Some of the main vendors of IaaS are Amazon Elastic Compute Cloud, Rackspace Cloud Hosting, Microsoft Windows Azure, Google Compute and Openstack Open Source Cloud There are also numerous other IaaS Vendors with their own offerings such as IBM, VMWare, HP, SAP and Oracle and many more.

IaaS Benefits

As a client of cloud service providers, you can focus on your own applications & databases, websites and IT systems without the obligation and capital expenditure of managing your own IT hardware, maintenance, datacentre space and extensive support.

  • Cloud Service Providers have the ability to provide a variety of hardware and technologies such as Linux and Open Source to cater for all organizations requirements
  • Cloud Service Providers offer pricing models which allow for companies to only pay for the storage and hours they need to use servers for. This billing method offers a significant cost saving against having to buy, maintain and then run your own hardware and networks
  • Location independent
  • Cloud Providers have the ability to offer scalability and sustainability. The ability to have an immediate increase in web services or resources in line with end of year financial runs or busy work periods such as Christmas can prove invaluable.
  • Highly automated. Regular task and daily tasks can be automated saving time and increasing productivity in other areas
  • Service Level agreements for standards of service.

IaaS Negatives

  • In regards to company sensitive data, there may be an issue as to who within the Service Provider could potentially have access to or monitor your data.
  • When customer data resides in with external service providers, there is the question of how security compliant the service provider is or can they offer you the type of physical and virtual security required including adhering to external data security regulations required by certain customers. Does the provider have security measures in place to manage communications outages such as denial of service and attacks including authentication issues such as IP spoofing, DNS poisoning, arp poisoning and RIP attacks?
  • The cloud service provider needs to make sure the system is always available for its clients. Companies need to be assured of relying on the high availability and performance of another providers’ systems.
  • Can a service provider verify the security of your data from end to end during transit, at rest or backup?
  • Often you will be sharing space with other organisations on the same infrastructure. The service provider should be able to clearly show secure segregation between customers.
  • If the service provider has a lack of employee processes and procedures such as how it monitors its employees and how access is granted and used, there could be a risk of malicious insiders who having access to client’s infrastructure and data could cause a significant security breach given the level of access and ability to infiltrate organizations and assets. Thus, damage to company brands, financial implications or penalties and productivity issues may cause damage to the service provider and the client’s reputation and business.
  • Different countries have different regulations and security considerations. Certain companies will need data to reside in countries which adhere to their own regulations and know the data is not able to be transferred into countries where data could be at considerable risk of exposure or data loss/leaking.


The PaaS (Platform as a Service) model, is a computing platform or environment allows developers to have the complete tools, operating systems, middleware and programming languages to build software or website applications. Everything is then hosted and stored by the PaaS Service Provider. PaaS offers developers a solution that is a complete software development, testing and deployment environment. In addition it has the benefit that the operating system, virtual machines, and infrastructure are hidden and not a concern to the developer. PaaS service models have automatic scalability to allow for increased usage or spikes in activity – therefore making PaaS a really useful way to build high traffic applications.

Some of the main vendors of PaaS include Amazon Web Services, Cloud Foundry (Open Source project run by VMware), AppHarbour and Heroku owned by Salesforce. There are various other vendors such as IBM Smartcloud, Redhat Openshift, openStack, Google App Engines and Engine Yard

A PaaS Service Provider generally offers the following

  • Operating System – Windows, Linux or Open Source
  • Programming Languages – C#, Python, Java, Ruby and Node.js etc.
  • Virtual Machines – Servers to run the applications
  • Databases – SQl or Oracle databases
  • Web Servers – Apache or IIS etc.
  • Distributed Computing – Messaging and Big Data technologies

PaaS Benefits

  • Developers have no concern for the underlying infrastructure. They don’t need to buy, implement, manage and maintain the hardware that the applications run on.
  • Bandwidth and resources are instantly scalable with increased workload and similar to IaaS, pricing models to allow for the peaks and troughs of workloads at certain times of the week, month or year.
  • Databases, VMs and complete environments can be commissioned very quickly reducing build times, developer costs and allowing applications to enter the market quickly.
  • Developers can collaborate worldwide very easily
  • Developers can use their own software on the platform

PaaS Disadvantages

  • Developers responsible for the updating and upgrading of the applications
  • Often you will be sharing space with other organisations on the same infrastructure. The service provider should be able to clearly show secure segregation between customers
  • Not as cost effective as SaaS and not as much control over VM as IaaS.
  • Changing providers may prove difficult
  • Compliance with all applicable regulations concerning security, privacy, and data retention needs the same considerations as IaaS

Software as a Service (SaaS)

The SaaS (Software as a Service) model is the simplest and most straightforward model for clients which is hosted on a Service Providers infrastructure, not a company’s own private network. Users basically log on to an application via a web browser or dedicated desktop icon. CRM, E-mail, Games and almost any application could potentially be hosted by the SaaS Cloud platform including virtual desktops


  • Users can log on to the application anytime/anywhere with an internet connection
  • The Cloud provider will monitor, maintain, upgrade and backup the underlying infrastructure and software versions automatically
  • Good pricing models including the ability to scale up and scale down users sometimes on a month by month basis
  • Minimal planning and easy to set up. Solutions can be implemented in weeks rather than months
  • Cloud Providers have an extremely resilient infrastructure tied into service level agreements for quality and uptime of service
  • No license fees to manage, just subscription fees


  • Little control over deployment, upgrade and testing methodology
  • The SaaS Cloud provider has full access to customers’ data, unless encryption is used
  • It is important to ensure that the application is compliant in terms of the location it is hosted due to certain countries and industries having strict regulations as to where data is stored
  • Currently there are limited applications which are not available on a hosted platform although IBM for example has over 100 SaaS hosted applications
  • Reliant on clients having a stable internet connection
  • Hosted applications can be variable in their features and functionalities compared to being managed and adjusted in-house
  • Currently SaaS applications are seen to be slightly slower than an in-house application however the comparison will be minimal and continually improving.
  • It is important to make sure a proper recovery and backup plan should be in place. The infrastructure hosting the application should be highly available and replicated across potentially multiple locations. In addition, the timeframe to recover from any potential attacks or failures must be clearly stated or part of the service agreement held with the SaaS provider

In this age of “Anything as a Service”, Cloud providers offer clients the pretty picture of unlimited computing, network, and storage capacity. One of the main concerns behind these outsourced cloud models is the location, security and safety of the data being created, transmitted and managed.  It is critical to ensure that there are correct processes and compliance of the internal security procedures, segregation of services, configuration hardening, patching, upgrading, auditing, and logging. The development software including APIs supplied for PaaS needs to be as secure as possible whilst working in injunction with any other software in the environment. Criminals continue to leverage new technologies and ways to penetrate the service providers in order to tamper with data causing loss and theft. IaaS offerings have hosted the Zeus botnet, Infostealer Trojan horses and Adobe PDF exploits. Aside from outside security concerns, it is vital to know who is managing your applications and data internally. Security is not just restricted to IT Processes. People also need to be subject to stringent security policies and procedures to prevent malicious attempts to infiltrate organizations and confidential data. Service providers must be able to stay one step ahead to continually maintain confidentiality, integrity and availability of those services.

These models are all evolving into exciting offerings with the potential to streamline IT, increase the use of automation and create secure highly available controlled environments with minimal disruption and ease of use. Although there are obvious learning curves for Cloud Infrastructure specialists including an understanding of multiple different platforms alongside an understanding of the increasing integration of automated services, these XaaS service models seem to be the way forward into a new era of Public Cloud Computing.




Optimization WordPress Plugins & Solutions by W3 EDGE