I am experiencing the exact same problem. vCenter Server 5.5 installation is on a brand new VM running Windows Server 2012 R2. vCenter Server is also new. I've been following Derek Seaman's excellent blog post about two-tier PKI infrastructure and vSphere 5.5 installation.
I am able to replace the vCenterSSO certificate. When I move to replacing Web Client certificate with a self-signed CA certificate, I receive the error below in vCenter Certificate Automation Tool v5.5.
==================================================================
7. Update the vSphere Web Client and Log Browser SSL Certificates
1. Update the Web Client Trust to Single Sign-On
2. Update the Web Client Trust to Inventory Service
3. Update the Web Client Trust to vCenter Server
4. Update the Web Client SSL Certificate
5. Update the Log Browser Trust to Single Sign-On
6. Update the Log Browser SSL Certificate
7. Rollback to the previous Web Client SSL Certificate
8. Rollback to the previous Log Browser SSL Certificate
9. Return to the main menu to update other services
The chosen service is: 4
[2014-06-08 - 1:25:36,70]: The services that are restarted as a part of this op
eration are: vSphere Web Client
Enter location to the new Web Client SSL chain (default value is: C:\Tools\Certi
ficates\vCenterWebClient\chain.pem):
Enter location to the new Web Client private key (default value is: C:\Tools\Cer
tificates\vCenterWebClient\rui.key):
Enter Single Sign-On Administrator user (default value is: administrator@vsphere
.local):
Enter Single Sign-On Administrator password (will not be echoed):
[.] The supplied certificate chain is valid.
[2014-06-08 - 1:25:50,86]: Last operation update vSphere Web Client SSL certifi
cate failed :
[2014-06-08 - 1:25:50,86]: Cannot validate the Lookup Service connection - erro
rlevel is 1
I get the below error when executing ssolscli.cmd.
C:\Program Files\VMware\Infrastructure\VMware\CIS\vmware-sso>ssolscli.cmd listSe
rvices https://192.168.12.8:7444/lookupservice/sdk
Intializing registration provider...
Getting SSL certificates for https://192.168.12.8:7444/lookupservice/sdk
com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certi
ficate chain not verified
com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certi
ficate chain not verified
Return code is: OperationFailed
100
Were you or anybody else able to resolve the above issue? Any thoughts and/or suggestions are greatly appreciated.