Hi, I have been battling with this specific issue for more than a year. I've just upgraded our farm to 5.5 and was experiencing exactly the same issue again.
Eventually after trying everything I've been able to identify the cause.
I had VMware Workstation (version 10) installed on my Windows client. As soon as I uninstalled VMware workstation, console access of VMs (either via the Web client or the vSphere client) is more or less instant (it was consistently around 10-15 seconds before).
VMware - please can you urgently look at this conflict. I've wasted so much time trying to get to the bottom of this!
Hope this helps someone else.