I'm kind of surprised at the lack of resolution to this (and google doesn't give me a lot of results regarding this specific problem) - for the longest time I thought I had working ESXi hosts that used SSL certificates signed by an internal CA - anybody actually had a case opened with VMware on this issue which still exist for both 5.1 and 5.5?
My first set of installs: vCenter Server Appliance 5.1 (original build 5200, have since updated to the latest build in the 5.1 release) with a single ESXi 5.1 host. The host was already added into vCenter before the certificate was replaced (so technically vCenter database would have the thumbprint for the original self-signed certificate generated during install). All operations seem to be fine
My second set of installs is where I discovered that my first set might not be working properly - same sequence so the host certificate was replaced AFTER it was added into vCenter. When I try to deploy a template with customization, that failed. However, if you do it without customization, the operation succeeded.
Seeing the symptoms from my 2nd set, I went to build a fresh lab where the host certificate is replaced BEFORE adding it into the lab vCenter - that failed with an SSLVerify error so basically, you cannot add a host that doesn't have self signed certificate.
What is interesting is that none of the existing KB addresses this issue (KB 2036744 addresses vCenter appliance 5.1 certificate change, KB 2015499 addresses ESXi 5.1 host certificate change, but no actual sequence of which should go first), turning on trivia logging in vCenter doesn't add much more info, and it seems to be failing the verification when it invokes the OpenSSL libraries - this is what I see in vpxd.log:
2013-11-13T00:18:40.247Z [7F2FF2E2E700 info 'vpxdvpxdMoLicenseManager' opID=31582879] [LicMgr] Downloading Dlfs for Host 'VMware ESX Server', Version: '5.0', File Version: '5.1.1.0', Dlf Directory Location: '/etc/vmware-vpx//licenses/site//VMware ESX Server/5.0/5.1.1.0'
2013-11-13T00:18:40.258Z [7F2FF2E2E700 warning 'Default' opID=31582879] SSL Error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
-->
2013-11-13T00:18:40.258Z [7F2FF2E2E700 warning 'Default' opID=31582879] SSL: connect failed
-->
2013-11-13T00:18:40.258Z [7F2FF2E2E700 error 'provisioningvpxNfcClient' opID=31582879] [VpxNfcClient] Unable to connect to NFC server: The remote host certificate has these problems:
-->
--> * unable to get local issuer certificate
and proceeds to dump the stack trace.
For a host that doesn't use self-signed certificate, vCenter never stores the thumbprint into the database (it should be empty). Now when vpxd starts up, I noticed this message in the log:
2013-11-12T23:43:18.403Z [7F2FFF8B2720 info 'vpxdvpxdMain'] [VpxdMain] Setting OpenSSL verify locations CAFile= CAPath=/etc/ssl/certs
Which suggests to me that OpenSSL libraries should able to the certificate directory /etc/ssl/certs to verify a remote server when making an SSL connection - I had my internal CA cert in that directory, soft-linked to its subject hash, and did the same even for the intermediate CA certs (when that is technically not necessary because I already have those certs returned by the ESXi host in a certificate chain)
Reading through the KB article still doesn't tell me any of the steps will actually help this verification - any ideas in the community about using non-self-signed cert on an ESXi host? Right now, for the 2nd set of install I actually reverted to the self-signed cert on the host.