Rolling back and recovering from failed vSphere 6 and external multisite PSC upgrades

Recovering from failed vSphere 6 and external multisite PSC upgrades 

So what happens if you have the following situation within a failed vSphere upgrade with a multi-site system?

The Scenario

The requirement was to upgrade 3 vCenters from 5.1U3 to 6.0U2. The vCenter servers previously had embedded SSO but have been repointed to the external 6.0 U2 PSCs. Note, we had an intermediary stage of repointing to 5.5U3 external SSOs first before we could upgrade the PSCs to 6.0U2. So once the PSCs are in a multisite 6.0U2 configuration, the upgrade of the vCenters can start and this is where we need to take care as the systems are interlinked at this point.

So we now have

  • 3 x vSphere 6.0U2 PSCs set up in a multisite configuration
  • 3 x 5.1 U3 vCenters pointing to a vSphere 6.0U2 PSC
  • 3 x SQL 2008 R2 Databases running the vCenter Databases and the Update Manager Databases. Not seen in the example above

Why did it fail initially?

Never assume anything. We had already upgraded 2 environments without any issue so we overlooked checking the SQL environments which were meant to be a replica but clearly weren’t!

  • DB password had special characters in the password.
  • Needed to give db owner rights on the vCenter DB and the MSDB database prior to upgrading
  • The ODBC Connection was not the right version. We needed SQL Native Client 10 or 11.
  • Additional Database Permissions for VMware (Can be seen in the vSphere 6.0 Documentation Center
  • SQL Server 2008 R2 needed Service Pack 2 installing. (No problem with this)
  • An informational messages regarding the vCenter FQDN
  • An informational message about the ALL_SERVICES accounts. (New in vSphere. Depends if you want to use Local System for all your vCenter Services or use the multiple vSphere accounts for individual services

So how do you recover from one failed vCenter Upgrade in this scenario?

We do need to take into account at what step has it failed and check the logs to verify this. Can it be an issue that can be worked though and fixed forward or do you need to rollback and depending on how far the installer has got before failing, it might open variations in the options for a rollback.  For example, if it fails to authenticate to the database, then it is unlikely it would have made any DB changes. The installer logs will give us this information.

You can retrieve the installation log files manually for examination.



Navigate to the installation log file locations.

%PROGRAMDATA%\VMware\vCenterServer\logs directory, usually C:\ProgramData\VMware\vCenterServer\logs

%TEMP% directory, usually C:\Users\ username\AppData\Local\Temp

The files in the %TEMP% directory include vminst.log, pkgmgr.log, pkgmgr-comp-msi.log, and vim-vcs-msi.log.


Open the installation log files in a text editor for examination.


  • Make sure you have been through all the pre-requisites prior to starting the upgrade and test in a lab as well. Give yourself the best start at not failing.
  • Snapshot the whole environment (All VCs, All DBs and all All PSCs) Make sure the environment is quiesced or if you want a more consistent image vs crash consistent, shutdown the whole environment and snapshot it cold
If the upgrade of any VC 5.x to 6.0 fails in this environment, you must roll back ALL PSC.  The reason is that the solution user format differs between 5.x and 6.x. During the upgrade of the VC, the VC 5.x solution users will be removed from the PSC and replaced with 6.x solution users early in the VC 6.x upgrade (during vmafd firstboot I believe). If you have a failure and only roll back the VC then you have a VC with 5.x solution users talking to a PSC that no longer is aware of those users. You must roll back ALL PSCs in the SSO Domain as they replicate.
  • If you encounter an unrecoverable upgrade installer error then in the event of rolling back to snapshots, the order will be all PSCs, the vCenter DB and vCenter Server
  • In the event the rollback above fails, all servers should be rolled back again with the order of power on being all PSCs, all vCenter DBs and all vCenter servers

Questions we have been asked

During an upgrade, could we stop/break the multisite replication agreements between PSCs to avoid any replication of issues in the event of an upgrade problem on one vCenter? 

There’s no issue with breaking replication agreements generally but it not something that should be done for an upgrade. The vdcrepadmin command line tool does allows the breaking and creating of agreements and it actually protects the customer by not allowing them to delete the only agreement available. This prevents a customer from inadvertently creating an isolated PSC. What you would do is go through and create the new agreements and once they are in place just delete the extra ones. There is nothing special about the replication agreements that are created during PSC deployment. They are the same as ones created with vdcrepadmin.




PSC 6 Replication for multi-site setups

PSC 6 Replication 

VMware Validated Designs

First of all I’d like to talk about VMware Validated Designs before going on to talk about how these can be incorporated into replicated PSC designs.

VMware Validated Designs deliver holistic data center-level designs that span compute, storage, networking and management, defining the gold standard for how to deploy and configure the complete VMware SDDC stack in a wide range of use cases. VMware Validated Designs include detailed guidance that combine with best practices on how to operate a VMware SDDC optimally. The benefits are

  • Accelerate time to market
  • Increase efficiency
  • De risk deployments and operations
  • Drive agility
  • Repeatable proven solutions

When looking to implement any software from the SDDC suite, we are now referring to the VMware Validated Designs for a repeatable proven solution. Of course there may always be modifications required to fit customers environments but the idea going forwards is to maintain the same standards of installation and configuration

Platform Services Controller Topology Decision Tree

Adam Eckerle from VMware has very kindly designed a poster containing a decision tree for anyone thinking of deploying PSCs and guides you through a set of decisions before deployment

PSC and Replication Points

  • All PSC replication is bi-directional but ring topology setups have to be manually created
  • Ring topology is now the recommended method, not mesh as it was previously
  • By default each PSC is replicating with only a single other PSC (the one you select when installing the additional secondary or tertiary PSC)
  • Site names do not have anything to do with replication today they are a logical construct for load balancers and future usage
  • Changes are not unique to a site but to a domain assuming they are part of the vsphere.local domain
  • vCenter only points to one PSC at a time or a load balanced address
  • PSC’s have to be part of the same domain together to use enhanced linked mode

Why Ring Topologies and not Mesh Topologies?

  • Ring Topologies allow for easier scaling out
  • They are easier to set up and maintain. There’s no issue with breaking and recreating replication agreements. vdcrepadmin does this and it actually protects the customer by not allowing them to delete the only agreement available. This prevents a customer from inadvertently creating an isolated PSC. What you would do is go through and create the new agreements and once they are in place just delete the extra ones. There is nothing special about the replication agreements that are created during PSC deployment. They are the same as ones created with vdcrepadmin
  • The failure of any one component in a ring does not cause replication partitions
  • Minimal links can be created for maximum redundancy which is increased when load balancers are used.

2 Site VVD Design

In the diagram below, you can see within each region/site, the VMware Validated Design instantiates two Platform Services Controllers and two vCenter Server systems in the appliance form factor. This includes one PSC and one vCenter for the management pod and one PSC and one vCenter for the shared edge and compute pod. The design also joins the Platform Services Controller instance to its respective Platform Services Controller instance.

3 Site VVD Design

When we start getting into a 3 site multisite design, we can then see how keeping it simple makes life a great deal easier. Below is a series of diagrams which show how this can be set up with a recommended ring topology.

This particular example uses 3 sites with 2 PSCs in each. There is no load balancer.


  • When initially installing the external PSCs, PSCA will be the first standalone PSC. PSCB will be installed next and partnered with PSCA. PSCC is then installed and partnered with PSCB. This is 2 way automatic replication in an inline configuration

  • If we wanted to make it a ring topology with just these 3 servers we could do the following and create a replication agreement between PSCA and PSCC

  • Now we add the second PSCs into each site and we get the following scenario. 2 way automatic links are created between the first and second PSC in each site (A and D) (B and E) and (C and F)

  • Now we see we have kind of lost our inline topology and how do we create a ring now? We want an inline topology of D > A > B > E > C > F
  • We actually need to create a new agreement between PSCE and PSCC followed by breaking the link between PSCB and PSCC. Remember the system will not let us break a replication agreement which would leave a PSC isolated so we need to create a link first

  • We will now get the following inline configuration

  • Next we want to create our ring topology. A replication agreement is created between PSCD and PSCF to create the ring


Stratoscale explains IaaS, PaaS and SaaS – “The Good”, the “Bad” or the “Ugly”?

Stratoscale asked 32 IT experts to share their insights on the differences between IaaS, PaaS and SaaS in the great article below.

Please enjoy the insights on “The Good”, the “Bad” or the “Ugly”? within IaaS, PaaS and SaaS.

IaaS, PaaS, and SaaS: The Good, the Bad and the Ugly

Cloud computing is a broad term for the various IT-related services that can be provided on demand using a consumption based model.

The three most common cloud computing models:

  • Infrastructure as a Service (IaaS)
  • Platform as a Service (PaaS)
  • Software as a Service (SaaS)

There are countless articles covering cloud services. Yet, you might still be confused about what the heck all these “as a service” terminologies are. You are not alone, we hope that after you read what 30+ cloud experts have to say about their value, advantages, benefits and best practices things will be much clearer.


Replacing vROps 6.4 self signed certificates

Replacing vROps 6.4 self signed certificates

Certificate requirements – Click the link

  • File encoded in PEM format
  • Certificate is valid for Server Authentication
  • All certificates in the chain are included
  • The private key is included (2048 bit RSA or higher)
  • The private key is not secured with a password

VMware KB 2046591 Link – Click here

Step 1

  • To create the Certificate Request first download OpenSSL for Windows and install it in the default location – C:\OpenSSL-Win64
  • Note: Path and name may be slightly different depending on the version
  • I used Win64OpenSSL_Light-1_1_0e.exe

Step 2

  • Copy and paste the following into a text file and save as vrops.cfg in a folder called c:\OpenSSL-Win64\Certs. You will need to create the certs folder. You will be able to place other certs, keys and requests in here also

  • Customize the bold fields.

[ req ]

default_md = sha512
default_bits = 4096
default_keyfile = vropsprivate.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: Loadbalancedfqdn, IP: Loadbalancedip, DNS: Loadbalancedshortname, DNS: FirstNodeFQDN, IP: FirstNodeip, DNS: FirstNodeShortname, DNS: SeondNodeFQDN, IP: SecondNodeip, DNS: SecondNodeShortName

[ req_distinguished_name ]
countryName = countryName
stateOrProvinceName = stateOrProvinceName
localityName = localityName
0.organizationName = organizationName
organizationalUnitName = organizationalUnitName
commonName = Loadbalancedfqdn

In my case I don’t have a load balancer. I just have a Master node, a Replica Node and a Remote Collector Node so it looks like this

[ req ]

default_md = sha512
default_bits = 4096
default_keyfile = vropsprivate.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: techlabops001.techlab.local, IP:, DNS: techlabops001, DNS: techlabops002.techlab.local, IP:, DNS: techlabops002, DNS: techlabops003.techlab.local, IP:, DNS: techlabops003

[ req_distinguished_name ]
countryName = UK
stateOrProvinceName = London
localityName = Norfolk
0.organizationName = Techlab
organizationalUnitName = vROps
commonName = techlabops001.techlab.local

Step 3

Open a command prompt and change directory to where OpenSSL is installed.  You need to first generate a new key pair with a 2048 or 4096 key length. Leave this file in the path you specify here for later

openssl genrsa -out key_filename.key 4096

c:\openSSL\bin>openssl.exe genrsa -out c:\openSSL\certs\vropsprivate.key 4096

Step 4

Use the key to generate a certificate signing request by running this command. I have added -sha512 but this is optional

openssl req -new -key key_filename.key -out certificate_request.csr -config config_file.cfg

openssl req -new -key c:\certs\vropsprivate.key -out c:\certs\vropscsr.csr -config C:\OpenSSL\certs\vrops.cfg -sha512

Step 5

  • Click Windows Start > Run, enter certtmpl.msc, and click OK.
  • Follow the below link to create a certificate template (If you haven’t got one already)

vCSA (vCenter Server Appliance) – Part 4 – Deploy CA-signed certificates

Step 6

Submit the CSR file to your Certificate Authority (CA) to obtain a signed certificate

  • Browse to your CA – https://servername/certsrv – Request a certificate. In my case I have a Microsoft CA installed on my Domain Controller
  • Choose Request a Certificate

  • Chose Request a Certificate

  • Open the request file in a notepad and copy the contents into the request box, In my case this is the vropscsr.csr file.
  • Select a certificate template that meets the requirement. In this case vROps Template

  • Submit the request and download as Base 64 Encoded

  • Download certificate and save it locally to your machine as “certnew.cer”

Step 7

Now we need to download the certificate chain from our CA

  • Browse to your CA – https://servername/certsrv
  • Select Download a CA certificate, certificate chain or CRL

  • Click Download CA certificate chain

  • It will download a certnew.p7b file or if it downloads a .cer file then use this
  • Change the name to root64.p7b and Click Open or save it and then open it. The Certificate manager Console will open

  • We now need to extract the certificates for the root and all intermediate authorities in the certificate chain. For each CA, export the certificate to a file.
  • Right click on the root certificate and select Export

  • In the export wizard, select Base-64 and save the file to the same folder as vrops.cfg. Do that for each authority!

  • Once you receive the certificate you have all the components needed to create the required .pem certificate.

Note: The order of CA’s certs in the .PEM file: Private Key, Cert, Intermediate Cert and then Root Cert which can seen above and described below

  • Private Key = vropsprivate.key
  • Cert = certnew.cer
  • Intermediate Cert – I don’t have one
  • Root CA – root64.cer

Step 8

  • Check you have all your certs in C:\OpenSSL\Certs. Note I already have a .pem file from other testing so ignore this as you won’t have one yet.

Step 9

Next go back to SSL and generate the .pem certificate. Example below

type key_filename.key server_cert.cer cacerts.cer > multi_part.pem

  • Change directory to where you saved your certificates
  • Type the following (including the word type!)

type vropsprivate.key certnew.cer root64.cer > vrops.pem

  • You should now see your .pem file which in my case is vrops.pem

  • If you open it, it will look something like this

Step 10

We can now navigate to the vROps page and upload the .pem file into the system

  • You have to log in specifically to the small administration website (https://vrops-server/admin), and click on the small icon on the top right of the screen as seen below. Then you can choose to Install new certificate.

  • When you click Install, you will be disconnected and reconnected. Better close the tab and reopen it to check that the custom certificate has been imported properly.
  • You should be able to open the vROps web pages without any certificate errors 🙂 See the green lock on the Firefox browser.

  • If it is still saying it is insecure then you may have to import the root64.cer into the Firefox browser. Go to Options > Advanced > Certificates > View Certificates and import the root64.cer
  • You can see my Microsoft CA Authority here which is now trusted

  • Note: I couldn’t get this working in Chrome even after importing the root CA certificate. I think Google have locked down their websites for vulnerabilities in the SHA1 algorithm however I haven’t figured out a way to get around this yet

If you do have problems and cannot get into the vROps webpages for any reason, VMware has KB2144949 which will allow you to revert back to the default certificates.

Using JXplorer to connect to vSphere PSC Servers

Using JXplorer to connect to vSphere PSC Server

JXplorer is a cross platform LDAP browser and editor. It is a standards compliant general purpose LDAP client that can be used to search, read and edit any standard LDAP directory, or any directory service with an LDAP or DSML interface.

Note: Please take extreme care when connecting to the vmdird database. This is not a recommended way of viewing this data but it can be very useful

JXplorer Download Location

I installed this on a Windows 10 workstation which had connectivity to my PSC Servers

Configuration Steps

  • Open JXplorer and open a new Connection
  • In the Host field put in the same of your PSC/SSO Server
  • Don’t put anything in Base DN. It will let you connect if you do put a Base DN in but will error when you try and expand the tree
  • In Security Level, choose User + Password
  • In User DN, type CN=Administrator,CN=Users,DC=vsphere,DC=local
  • Put in your PSC/SSO password
  • Save the template

  • You should now see the following screen

  • Expand vSphere > Configuration > Sites and you should be able to see all the replication agreements.
  • I’ve been playing around with multi-site scenarios which is why you can see Default-First-Site, Default-Second-Site and Default-Third-Site which are my 3 PSCs in a multisite scenario.

Other Observations

Information from Sung Rao (VMware) Thank Q

In 5.5. the only secure LDAP communication between SSO/PSC nodes are via LDAPS.  Thus in automatic replication agreements establishment, LDAPS is used.

In 6.0 and after, we introduced LDAP SASL/SRP binding which go through port 389. LDAP SASL/SRP (or KRB) is the simple and safe to manage between LDAP nodes. This binding mechanism is preferable to LDAPS as the SSL port is difficult to manage/deploy correctly as it depends on PKI. Also, LDAP layer sites below certificate. You need an ID before you can get a cert

Regardless, in 6.0 and after, the server will try SASL/SRP first and fall back to LDAPS if necessary regardless of LDAP/LDAPS in the labeledURI in the replication agreement definition. You also cannot force the replication agreements to use LDAPS in 6.0 and after

IaaS, PaaS and SaaS explained

“IaaS, PaaS and SaaS explained

IaaS, PaaS, SaaS… It can sometimes feel like we are in acronym hell. What are the correct definitions and how does each service differ?

IT organizations have historically been using their own Private Cloud Infrastructure either on premise or inside their own external datacenters to host servers, applications and data etc. Move forward 20 years and we have moved into a service provider era containing Public Cloud Infrastructure and Hybrid Public Cloud Infrastructure. Organizations called service providers exist specifically to provide, manage and maintain the infrastructure on which their client organization’s servers, application or data are hosted. The client organization gets access controls to manage their servers, applications and data hosted on the remote server. This is the basis behind cloud computing so where does IaaS, PaaS, SaaS fit into this scenario?

In an IaaS (Infrastructure as a Service) model sometimes called Hardware as a Service, a third-party service provider hosts hardware, software, servers, storage, networking and other infrastructure components on behalf of an organization; including managing tasks such as system maintenance, upgrading, backup and disaster recovery planning.

Some of the main vendors of IaaS are Amazon Elastic Compute Cloud, Rackspace Cloud Hosting, Microsoft Windows Azure, Google Compute and Openstack Open Source Cloud There are also numerous other IaaS Vendors with their own offerings such as IBM, VMWare, HP, SAP and Oracle and many more.

IaaS Benefits

As a client of cloud service providers, you can focus on your own applications & databases, websites and IT systems without the obligation and capital expenditure of managing your own IT hardware, maintenance, datacentre space and extensive support.

  • Cloud Service Providers have the ability to provide a variety of hardware and technologies such as Linux and Open Source to cater for all organizations requirements
  • Cloud Service Providers offer pricing models which allow for companies to only pay for the storage and hours they need to use servers for. This billing method offers a significant cost saving against having to buy, maintain and then run your own hardware and networks
  • Location independent
  • Cloud Providers have the ability to offer scalability and sustainability. The ability to have an immediate increase in web services or resources in line with end of year financial runs or busy work periods such as Christmas can prove invaluable.
  • Highly automated. Regular task and daily tasks can be automated saving time and increasing productivity in other areas
  • Service Level agreements for standards of service.

IaaS Negatives

  • In regards to company sensitive data, there may be an issue as to who within the Service Provider could potentially have access to or monitor your data.
  • When customer data resides in with external service providers, there is the question of how security compliant the service provider is or can they offer you the type of physical and virtual security required including adhering to external data security regulations required by certain customers. Does the provider have security measures in place to manage communications outages such as denial of service and attacks including authentication issues such as IP spoofing, DNS poisoning, arp poisoning and RIP attacks?
  • The cloud service provider needs to make sure the system is always available for its clients. Companies need to be assured of relying on the high availability and performance of another providers’ systems.
  • Can a service provider verify the security of your data from end to end during transit, at rest or backup?
  • Often you will be sharing space with other organisations on the same infrastructure. The service provider should be able to clearly show secure segregation between customers.
  • If the service provider has a lack of employee processes and procedures such as how it monitors its employees and how access is granted and used, there could be a risk of malicious insiders who having access to client’s infrastructure and data could cause a significant security breach given the level of access and ability to infiltrate organizations and assets. Thus, damage to company brands, financial implications or penalties and productivity issues may cause damage to the service provider and the client’s reputation and business.
  • Different countries have different regulations and security considerations. Certain companies will need data to reside in countries which adhere to their own regulations and know the data is not able to be transferred into countries where data could be at considerable risk of exposure or data loss/leaking.


The PaaS (Platform as a Service) model, is a computing platform or environment allows developers to have the complete tools, operating systems, middleware and programming languages to build software or website applications. Everything is then hosted and stored by the PaaS Service Provider. PaaS offers developers a solution that is a complete software development, testing and deployment environment. In addition it has the benefit that the operating system, virtual machines, and infrastructure are hidden and not a concern to the developer. PaaS service models have automatic scalability to allow for increased usage or spikes in activity – therefore making PaaS a really useful way to build high traffic applications.

Some of the main vendors of PaaS include Amazon Web Services, Cloud Foundry (Open Source project run by VMware), AppHarbour and Heroku owned by Salesforce. There are various other vendors such as IBM Smartcloud, Redhat Openshift, openStack, Google App Engines and Engine Yard

A PaaS Service Provider generally offers the following

  • Operating System – Windows, Linux or Open Source
  • Programming Languages – C#, Python, Java, Ruby and Node.js etc.
  • Virtual Machines – Servers to run the applications
  • Databases – SQl or Oracle databases
  • Web Servers – Apache or IIS etc.
  • Distributed Computing – Messaging and Big Data technologies

PaaS Benefits

  • Developers have no concern for the underlying infrastructure. They don’t need to buy, implement, manage and maintain the hardware that the applications run on.
  • Bandwidth and resources are instantly scalable with increased workload and similar to IaaS, pricing models to allow for the peaks and troughs of workloads at certain times of the week, month or year.
  • Databases, VMs and complete environments can be commissioned very quickly reducing build times, developer costs and allowing applications to enter the market quickly.
  • Developers can collaborate worldwide very easily
  • Developers can use their own software on the platform

PaaS Disadvantages

  • Developers responsible for the updating and upgrading of the applications
  • Often you will be sharing space with other organisations on the same infrastructure. The service provider should be able to clearly show secure segregation between customers
  • Not as cost effective as SaaS and not as much control over VM as IaaS.
  • Changing providers may prove difficult
  • Compliance with all applicable regulations concerning security, privacy, and data retention needs the same considerations as IaaS

Software as a Service (SaaS)

The SaaS (Software as a Service) model is the simplest and most straightforward model for clients which is hosted on a Service Providers infrastructure, not a company’s own private network. Users basically log on to an application via a web browser or dedicated desktop icon. CRM, E-mail, Games and almost any application could potentially be hosted by the SaaS Cloud platform including virtual desktops


  • Users can log on to the application anytime/anywhere with an internet connection
  • The Cloud provider will monitor, maintain, upgrade and backup the underlying infrastructure and software versions automatically
  • Good pricing models including the ability to scale up and scale down users sometimes on a month by month basis
  • Minimal planning and easy to set up. Solutions can be implemented in weeks rather than months
  • Cloud Providers have an extremely resilient infrastructure tied into service level agreements for quality and uptime of service
  • No license fees to manage, just subscription fees


  • Little control over deployment, upgrade and testing methodology
  • The SaaS Cloud provider has full access to customers’ data, unless encryption is used
  • It is important to ensure that the application is compliant in terms of the location it is hosted due to certain countries and industries having strict regulations as to where data is stored
  • Currently there are limited applications which are not available on a hosted platform although IBM for example has over 100 SaaS hosted applications
  • Reliant on clients having a stable internet connection
  • Hosted applications can be variable in their features and functionalities compared to being managed and adjusted in-house
  • Currently SaaS applications are seen to be slightly slower than an in-house application however the comparison will be minimal and continually improving.
  • It is important to make sure a proper recovery and backup plan should be in place. The infrastructure hosting the application should be highly available and replicated across potentially multiple locations. In addition, the timeframe to recover from any potential attacks or failures must be clearly stated or part of the service agreement held with the SaaS provider

In this age of “Anything as a Service”, Cloud providers offer clients the pretty picture of unlimited computing, network, and storage capacity. One of the main concerns behind these outsourced cloud models is the location, security and safety of the data being created, transmitted and managed.  It is critical to ensure that there are correct processes and compliance of the internal security procedures, segregation of services, configuration hardening, patching, upgrading, auditing, and logging. The development software including APIs supplied for PaaS needs to be as secure as possible whilst working in injunction with any other software in the environment. Criminals continue to leverage new technologies and ways to penetrate the service providers in order to tamper with data causing loss and theft. IaaS offerings have hosted the Zeus botnet, Infostealer Trojan horses and Adobe PDF exploits. Aside from outside security concerns, it is vital to know who is managing your applications and data internally. Security is not just restricted to IT Processes. People also need to be subject to stringent security policies and procedures to prevent malicious attempts to infiltrate organizations and confidential data. Service providers must be able to stay one step ahead to continually maintain confidentiality, integrity and availability of those services.

These models are all evolving into exciting offerings with the potential to streamline IT, increase the use of automation and create secure highly available controlled environments with minimal disruption and ease of use. Although there are obvious learning curves for Cloud Infrastructure specialists including an understanding of multiple different platforms alongside an understanding of the increasing integration of automated services, these XaaS service models seem to be the way forward into a new era of Public Cloud Computing.




Decommissioning a Windows 2012 vCenter 6 Server and a Windows 2012 PSC 6 Server

Decommissioning a vCenter 6 Server and a PSC 6 Server

After you deploy a vCenter Server with an embedded Platform Services Controller or a vCenter Server with an external Platform Services Controller, if you no longer need any of the appliances or if an appliance stops responding, you can decommission and delete the appliance from vSphere inventory and domain.

: The process for removing a vCenter Server or a Platform Services Controller from the vSphere domain is irreversible. After you remove an appliance from the domain, you cannot rejoin it to the same domain. You must perform a re-install or a re-deploy of  vCenter Server or Platform Services Controller system in order to re-join it to the domain.


In my scenario I have 2 PSCs in multisite mode and 1 vCenter connected to each PSC. Note: These are Windows Server 2012 servers. The link at the bottom of the page details the process for the VCSA appliance

  • techlabsso002.techlab.local (Windows 2012 PSC)
  • techlabsso003.techlab.local (Windows 2012 PSC)
  • techlabvcs002.techlab.local (Windows 2012 vCenter connected to techlabsso002)
  • techlabvcs003.techlab.local (Windows 2012 vCenter connected to techlabsso003)

I am going to remove 1 vCenter and 1 PSC from this scenario which will be techlabsso003.techlab.local and techlabvcs003.techlab.local.

Step 1 Decommission the vCenter Server

First of all I want to decommission my vCenter Server from the Platform Services Controller

  • Log into the first PSC server; in my case techlabsso003
  • Browse to C:\ProgramData\VMware\vCenterServer\cfg\install-defaults.
  • Open the vmdir.ldu-guid file to find the hostid.

  • On the Platform Service Controller, click Start > Run, type cmd.exe, and click OK. The Command Prompt window open.
  • Navigate to C:\Program Files\VMware\vCenter Server\bin
  • Run the cmsso-util unregister command to unregister the vCenter Server. Where, vCenter_Server_System_Name is the FQDN or IP address of vCenter Server that you want to decommission. You must run this command only on the Platform Services Controller which your vCenter Server is registered.

cmsso-util unregister --hostId host_Id --node-pnid vCenter_Server_System_Name --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

  • You should see the following asking you to confirm you want to unregister the vCenter Server. Note I had to remove the –hostid argument as the command didn’t recognise it

  • You should then see it say Success

  • Power off the vCenter Server

Step 2 Decommission the PSC Server

  • Personally I would shut down the PSC or simply stop the Platform Services Controller services. Note below that I’ve only stopped the Platform Service but all services would need stopping

  • On the PSC to be decommissioned, browse to C:\ProgramData\VMware\vCenterServer\cfg\install-defaults.
  • Open the vmdir.ldu-guid file to find the hostid.

  • On one of the other Platform Service Controllers, click Start > Run, type cmd.exe, and click OK. The Command Prompt window open.
  • Navigate to C:\Program Files\VMware\vCenter Server\bin
  • Run the cmsso-util unregister command to unregister the stopped Platform Services Controller
  • Where, Platform_Services_Controller_System_Name is the FQDN or IP address of the Platform Services Controller that you want to decommission. You must run this command only on one of the Platform Services Controller replication partners, as the synchronization removes the entries from all other Platform Services Controller replication partners.

cmsso-util unregister --hostId host_Id --node-pnid PSC_System_Name --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

  • Now you may get errors such as the below

Could not find a host ID which maps to “Servername” in Component Manager

Leave federation cleanup failed. Error [1] – Operations error and Error registering Computer account

  • If either of these occur then make sure you have switched off the PSC to be decommissioned
  • Navigate to C:\Program Files\VMware vCenter Server\vmdird and run the below command

.\vdcleavefed -h -u administrator -w

  • If you get a message to say Leave Federation cleanup done then you can then go ahead and delete the Platform Services Controller appliance that you no longer need from the vSphere inventory.
  • Reviewing the Nodes > Objects tab in the Web Client successfully removed the object from inventory.

Notes on vdcleavefed

cmsso-util leverages vdcleavefed to clean up the machine account, but also goes through and cleans up the solution users and anything else that may have been registered in component manager. It also has some pre-flight checks involved. vdcleavefed on its own will just rip out the machine account, and really should not be used unless in case of an emergency where we are in such a state where cmsso-util will not suffice (e.g. doesn’t work). cmsso-util is the preferred method of cleanup.



Deploying Orchestrator 6.0.3 into vSphere 6


Deploying Orchestrator 6.0.3 into vSphere 6

Software versions in my lab environment

  • vCenter v6.0.0, 3018524
  • vSphere Hosts v6.0.0, 3029758
  • VMware-vCO-Appliance-




  • Download and deploy VMware-vCO-Appliance- into vCenter – File > Deploy ovf template

screen-shot-2016-11-22-at-20-26-33 screen-shot-2016-11-22-at-20-26-47 screen-shot-2016-11-22-at-20-28-11 screen-shot-2016-11-22-at-20-28-35 screen-shot-2016-11-22-at-20-29-07 screen-shot-2016-11-22-at-20-29-36 screen-shot-2016-11-22-at-20-29-59 screen-shot-2016-11-22-at-20-30-47 screen-shot-2016-11-22-at-20-31-58 screen-shot-2016-11-22-at-20-35-35 screen-shot-2016-11-22-at-20-37-18

  • Power on the VM
  • Log into a web browser using the Orchestrator appliance web address. In my case


  • Change the time zone to Europe/London or whichever your timezone is and click Save Settings


  • Click the Network tab and check the settings on the 3 tabs

screen-shot-2016-11-22-at-20-54-10 screen-shot-2016-11-22-at-20-56-20 screen-shot-2016-11-22-at-20-57-02

  • Click the Admin tab and click Time Settings are correct. I have Use Host Time but you can use Time Server

screen-shot-2016-11-22-at-20-57-49 screen-shot-2016-11-22-at-20-58-59

  • Click Save Settings


  • Log into a web browser using the Orchestrator web address. In my case
  • Use the vmware username and the password you set up in the OVF deployment


  • You will reach the below screen


  • Click on the Network tab on the left hand side and select your IP Address and check all other details are correct. Click Apply Changes at the bottom right of the screen


  • Click on Authentication and scroll down the screen until you see a link for SSL Certificates. Click on this link


  • Put in your vCenter server in the following format – techlabvcs001.techlab.local:7444 and click Import

screen-shot-2016-11-22-at-21-08-01 screen-shot-2016-11-22-at-21-21-36

  • Put in your Single Sign On/PSC server in the following format – techlapsc001.techlab.local:7444 and click Import

screen-shot-2016-11-22-at-21-12-04 screen-shot-2016-11-22-at-21-14-28

  • Go back to the Authentication tab
  • Put in your Single Sign On server and click Advanced
  • put in your Admin username and password
  • Click Register Orchestrator


  • It should look like the below with further configuration to do


  • Choose your SSO Domain which can be the local domain, LAN domain or the vsphere.local domain.
  • In my case I chose my main domain techlab.local where I have set up a group called vro-group which contains user accounts I want to use as Admins


  • Click Accept Orchestrator Configuration


  • Click Test login and try one of your users


  • Check your license


  • Check the Plugins are all ok


  • Click Startup options and restart both services
  • Log back in and check everything is green

screen-shot-2016-11-22-at-21-36-17 screen-shot-2016-11-22-at-21-37-48 NEXT

  • Open a web page and navigate to your Orchestrator configuration page which in my case is https://techlaborc001.techlab.local:8281
  • Click on Start Orchestrator Client


  • Click on the drop down to Design
  • Navigate to Library > Microsoft > Active Directory > Configuration > Add an Active Directory server
  • Add in the relevant details for your AD server and add others as necessary

screen-shot-2016-11-22-at-23-19-53 screen-shot-2016-11-22-at-23-17-19

  • Next navigate to Library > vCenter > Configuration > Add a vCenter Instance


  • Click Next and fill in the next screen


  • Next we need to run the workflow Register vCenter Orchestrator as a vCenter extension


  • Next type in the external address to advertise this Orchestrator
  • this needs to be for example https://techlaborc001.techlab.local:8281


  • It should say it has been registered as per below


  • We can check it has been registered by opening a web browser and putting in the vCenter server address as per below
  • https://techlabvcs001.techlab.local/mob
  • Click on Content


  • Click on ExtensionManager


  • Look for extensionList[“com.vmware.vco”] which should only exist when the workflow has run correctly.


  • Click on Client


  • You should see the below in url string. This will also appear in the Web Client which we’ll see further on in the instructions
  • You can put this link into a web browser and it should try and download the zip


  • If you need to remove an extension, follow this useful blog below

Removing extensions link

  • You now need to restart the web client
  • When the Web Client has restarted and come up again, Go to the Home screen and select the Orchestrator icon


  • You should now see the vCenter and the Orchestrator server listed and you’ll see the information which we saw in the mob web page


  • If you click on Workflows under Inventory trees, you will see the whole library of workflows


  • You can then use the inbuilt workflows or create your own in Orchestrator
  • If you run the List the vCenter Orchestrator extensions of vCenter server, you will see it will pop up in the Recent Tasks list at the bottom of vCenter


  • Pretty funky stuff 🙂


  • In the vSphere Web Client > Click Home > Orchestrator, click on the Workflow icon and expand vCenter > Virtual Machine Management > Basic



  • Right click “Create simple virtual machine”, here is where you can run a workflow directly from within vSphere Web Client.



Installing the Linux Log Insight agent on the vCenter Orchestrator appliance


Installing the Linux Log Insight agent on vCenter Orchestrator

The Log Insight agent now gets pre-installed on some of the vRealize appliances which is very useful which means there is no need to install agents manually.  Some of the VMware products which have the agent pre-installed:

vRealize Business
vRealize Operations Manager (beginning from 6.1)
vRealize Orchestrator (beginning from 7.0.1)
vRealize Automation (beginning from 7.0.1)
vRealize Log Insight

However this version of Orchestrator is 6.0.3 due to work testing so we need to install the agent manually.

vCO Details

  • Name = techlabvco001.techlab.local
  • IP Address =
  • SSH enabled
  • vCO Config Page =
  • vCO Getting started page and Orchestrator client download =
  • vCO Appliance login =

Log Insight Details

  • Name = techlabvrl001.techlab.local
  • IP Address =
  • SSH enabled
  • Log Insight Configuration =

Useful link to Log Insight Documentation Center


Note: You may already have Orchestrator installed. If so go from connecting WinScp to the Orchestrator appliance.

  • Download and install the vCenter Orchestrator OVF. In my case this was version 6.0.3 as I was doing some testing for work.
  • Import the OVF into vCenter and follow the wizard to set all the relevant configuration information. Note: You will need to set a root password and a default password for the vmware user account in the wizard in order to access the configuration page
  • Power on the vCO appliance
  • Navigate to the vCO Config Page = and log in with the account vmware and the password you set during installation
  • In the General Page you can reset the vmware account password if you wish


  • Click on Network and check all the details are correct


  • You will need to put in an authentication source (LDAP, Active Directory etc) This is required as you will need to have authentication sources to log in to the Orchestrator client



  • After configuring an authentication source, you may need to restart the vRO Server and the vRO Configuration Server.


  • Add your license in. Options are below


  • Check all other options and configure as relevant. Basically everything should look green.
  • Next Log into Log Insight


  • Click on the Administration icon (Top right in Log Insight)


  • Click on Agents


  • Click on Download Log Insight Agent Version 3.0.1
  • Choose Linux RPM


  • Using Winscp, log into the vCO appliance


  • We now need to copy the Linux Log Insight agent to a directory on the vCO server
  • Copy the agent to the /tmp folder


  • Putty in to the vCO box
  • Switch to the /tmp folder – cd /tmp
  • To set the target vRealize Log Insight server during installation run the sudo command and replace hostname with the IP address or hostname of the vRealize Log Insight server.
  • sudo SERVERHOST=hostname rpm -i VMware-Log-Insight-Agent-VERSION-BUILD_NUMBER.rpm
  • In my case
  • sudo SERVERHOST=techlabvrl001.techl;ab.local rpm -i VMware-Log-Insight-Agent-3.0.0-2985111.noarch_192.168.1.122.rpm


  • You should see the following


  • Go back into WinSCP and open the file liagent.ini from /etc/liagent.ini
  • Check the LogInsight hostname has been added in and check all other options. We will not be modifying this liagent file as the recommended way to modify these settings is via the Linux Content Pack which needs to be imported into Log Insight and configured from within here. Instructions below in further steps


  • Go back into Log Insight and refresh the page and check the agent has been picked up.


Next we need to install the Linux Content Pack – Linux__v1.0.vlcp currently

  • Go to the Administration icon and click on Content Packs


  • Find the Linux Content Pack


  • When you click on the Content Pack, the below information will come up


  • Click Install and the below message will come up


  • Now that you have installed the content pack you can create groups with specific configurations. Go back to Administration > Agents and create your first group for Linux computers.
  • Select Linux in the pull-down menu and click on the copy template button (2 rectangles). (Note you can’t see the 2 triangles until you hover over the agent)


  • Put a name in for the agent group


  • Adjust the filter to reflect what machine/machines you want to use
  • In this case I have just added a filter for the hostname of my vCO server


  • This adds the following to the Agent Configuration for the agent on your Linux machines.
  • If you want to view the Orchestrator Workflow Information then you need to add another section in the Agent Configuration (



  • Click Save New Group
  • Log into the Orchestrator client and test a Workflow (I used Add an Active Directory Server and Remove an Active Directory Server but you could try anything)


  • You should then see the below Workflow being logged in log Insight if you filter by vCO hostname


  • Voila, you have logging set up for the vCO in Log Insight

Adding queries to Dashboards

  • We were using a Workflow which changed VM vDS Port Groups. Within this Workflow, it is set to output a string to the scripting log called PORTGROUP Change – Update completed successfully
  • We can create a favourite query using this query text contains PORTGROUP Change – Update completed successfully – See highlighted below
  • You can now add this query to a Dashboard. Whilst in the query, you can click on the icon to the right (highlighted in yellow) which means Add current query to dashboard


  • Fill in the Dashboard details and then you should be able to view this anytime and adjust the time over which work has taken place


vSphere HTML 5 Web Client Fling


vSphere HTML 5 Web Client

The vSphere HTML5 Web Client is here! It is written using HTML5 and Javascript

The following features are available at the moment

  • VM Power Operations (common cases)
  • VM Edit Settings (simple CPU, Memory, Disk changes)
  • VM Console
  • VM and Host Summary pages
  • VM Migration (only to a Host)
  • Clone to Template/VM
  • Create VM on a Host (limited)
  • Additional monitoring views (Performance charts, Tasks, Events)
  • Global Views (Recent tasks, Alarms–view only)

This Fling has been designed to work with your existing vSphere 6.0 environments. The new client is deployed as a new VM from the downloadable OVA.  Currently the installation instructions are command line-based, but VMware are working on a GUI installation and plan to release it as an update to this Fling once it is ready.

Download and Information

System requirements

  • 2 vCPU, 4 GB RAM, 14 GB
  • An existing VC6.0 installation (VCSA or Windows). The H5 client appliance will need 4 GB RAM, 2 vCPUs and the hard disk will grow up to 14 GB
  • Recommended browsers: Chrome, Firefox, IE11. Others may work, with some functional or layout issues.
  • Windows vCenter: Was tested with a vCenter on Windows Server 2012 R2, but should work with other versions as well.


Note: I have a Windows 2012 R2 server running vCenter Server 6 and a Windows 2012 R2 server running an external PSC version 6. There are different instructions for running different vCenter/PSC setups.

First of all download the H5 Client Deployment Instructions and Helpful Hints.

  • Download the OVA and server-configure.bat

Screen Shot 2016-08-23 at 11.35.34

  • In vCenter, go to File > Deploy OVF Template

Screen Shot 2016-08-23 at 12.07.38

  • Check OVF Template details

Screen Shot 2016-08-23 at 12.08.37

  • Accept the License Agreement
  • Put in a name and Location

Screen Shot 2016-08-23 at 12.09.30

  • Choose a host and cluster

Screen Shot 2016-08-23 at 12.10.15

  • Select the Resource Pool if any

Screen Shot 2016-08-23 at 12.11.03

  • Choose your storage

Screen Shot 2016-08-23 at 12.11.43

  • Check Disk Format

Screen Shot 2016-08-23 at 12.12.24

  • Check VM Networking and choose a Port Group

Screen Shot 2016-08-23 at 12.13.03

  • Choose IP Address allocation

Screen Shot 2016-08-23 at 12.13.45

  • Put in an IP address

Screen Shot 2016-08-23 at 12.14.28

  • Click Finish

Screen Shot 2016-08-23 at 12.15.34

  • I then had to create an IP Pool in vCenter
  • Click on the Datacenter object > IP Pools

Screen Shot 2016-08-23 at 12.17.09

  • Click on the tabs and fill in the relevant information. In my case I needed to add some DNS and Association information to associate this resource pool with my networks and in particular the network my HTML 5 client is going to be on

Screen Shot 2016-08-23 at 12.18.27

Screen Shot 2016-08-23 at 12.19.26

  • Power on the VM
  • If you click on the console, you should see the below screen

Screen Shot 2016-08-23 at 14.09.20

  • SSH or WINScp as root into the H5 client appliance VM (Note: Username is root and password is demova)
  • Create the following folders

Screen Shot 2016-08-23 at 12.37.00

Screen Shot 2016-08-23 at 12.36.19

  • Copy the provided ‘server-configure.bat’ to any directory on the vCenter and PSC for Windows. (This file is one of the Fling downloads on the top left) NOTE: If you have installed vCenter into any folder other than default (%PROGRAMFILES%), the script may not find the appropriate files. You will need to edit the file and replace the two references to %PROGRAMFILES% with the appropriate path so that the “KEYTOOL” and “VECS_CLI” paths line up. These two variables are at the top of the file.
  • You may also need to change this at the end of the file to the correct path (this is for the file): SET CLIENT_DIR=%PROGRAMDATA%\VMware\vCenterServer\cfg\vsphere-client
  • My PSC was all installed on the C Drive but I had my vCenter installed on the D Drive so I had to change the file below which is highlighted in yellow to my correct path


  • Run the server-configure.bat on your PSC server as Administrator
  • The store.jks and file will be created
  • Ignore the Creating error message

Screen Shot 2016-08-23 at 15.11.17

  • Copy the files store.jks and which are generated to the below locations
  • /etc/vmware/vsphere-client/store.jks
  • /etc/vmware/vsphere-client/vsphere-client/
  • In the Windows VC machine, open an Administrator Command Prompt and run the ‘server-configure.bat’ script. The following files will get generated:

Screen Shot 2016-08-23 at 15.18.12

  • Copy the file to H5 client virtual appliance at the following location
  • /etc/vmware/vsphere-client/config/
  • Log into the H5 appliance and run this command to start the server:
  • /etc/init.d/vsphere-client start

Screen Shot 2016-08-23 at 15.23.06

  • It should come up and say started in xxx seconds

Screen Shot 2016-08-23 at 15.27.14

  • Once the installation steps above are completed, point your browser to this URL, and log in with your normal vCenter credentials:
  • https://H5_Appliance_Address:9443/ui

Screen Shot 2016-08-23 at 15.30.15

Optimization WordPress Plugins & Solutions by W3 EDGE