So I have found a work around if you really need a non-self-signed cert on the ESXi host - do not install the root CA cert and intermediate certs that signed the ESXi host cert in the vCenter appliance under /etc/ssl/certs. What this will do is cause vCenter to fail SSL verification "early", so that it will prompt you to trust the ESXi host cert. If you do remove the CA certs (root and intermediate), you will have to at least restart vpxd (/etc/init.d/vmware-vpxd restart) - you might be able to get away without doing so (it's OpenSSL libraries that does the first SSL verification) but I always restarted during my test
The caveat with this, by my guess, is that if you replaced the internal certs on vCenter appliance with certs that are signed by the same CA chain, things could break - I haven't tried going without the CA certs in /etc/ssl/certs on my production instance yet
On the other hand - the conclusion I came to is solely based on all the test cases I ran on my own after vCenter magically added my host during a troubleshooting session with VMware - I had to revert to a fresh vCenter appliance 2 or 3 times to verify that this is repeatable. I would speculate that somewhere in the internal routines it forgot to remind OpenSSL that the CA certs are in /etc/ssl/certs, and from the error messages I got out of the logs, I'm fairly certain it's OpenSSL doing all verification rather than some Java routines (which would have required a JKS keystore to be modified) as the error code returned matches an existing OpenSSL error code