I have pin pointed the cause of this memory issue in my 5.1 environment and stopped the issue by disabling the Single Sign on component. The steps that we performed are NOT supported by VMwrae so use at your own risk:
1. RDP to vcenter
2. Locate vpxd.cfg C:\ProgramData\VMware\VMware VirtualCenter\
3. Edit this file with notpad++ (google it if you don't have it)
4. Modify line 54 or find code:
<sso>
<admin>
<url>https://vcenter FQDN/sdk/uri>
</admin>
<enabled>true</enabled>
<groupcheck>
<lookupService>
<serviceId>{Large String here}:5</serviceId>
<lookupService>
Change the Value "true" to "false" (without quotation marks) and then re-boot your vcenter. In doing this our vcenter has gone from a woolly mammoth using 40GB of ram to using 7.5 - 8GB and it now stable and responding fast and not producing false alerts.
I hope this may be of some small help to someone but as I said at the start use at your own risk as this is not supported configuration by VMware.