We are using 5.0 and found out that the problem was caused by the performance monitor in vsphere client.
If there is a vsphere client keeps monitoring the graph from vsphere client, the vCenter memory will be eaten up by the vpxd,exe.
Once I shutdown the vsphere client machine and restart vcenter server everything becomes normal again.
So somehow the performance monitoring on vcenter has memory leak I guess.
Any idea?