You have to differentiate between memory consumption and active memory usage. The hypervisor does not have access to the VM guest OS free list and does not know about internal memory mapping of these guests.
It therefore will usually not release memory that was once claimed by the guest until memory pressure is kicking in. So while a VM might have been showing alot of consumed memory the actual working set of this VM is lower most of the time.
This of course needs to be evaluated on a case by case basis as there is no general rule of thumb and it is extremely workload dependent if having high memory consumption on the host is an actual issue or just a cosmetic issue (especially since ESXi 5.0 uses large pages and TPS is kicking in way later than it used to do in 4.x).