There can be a couple of things going on
1. Issues with the NIC cards used for the management network (drivers/firmware etc)
2. Issues with the traffic passing through the ports of the physical switch intermittently.
What is the hardware (host and nics) you are running ESX on? Also are only the management nic's affected?
Capturing the host logs (vm-support) from one of the affected hosts at the time of the incident would allow us to see what's happening at the OS level.
Also an easy thing to do would be to upgrade the drivers and firmware for the NIC cards to the latest levels available and reboot the host and check