Ok, so you are using vSphere 4.1, but which edition? Essentials, Essentials plus, Standard, Enterprise, ...?
What type of physical switch do you use and - if managed - how are the uplink ports configured? To me this looks like an unmanaged switch, because you are using different IP subnets without VLAN tagging!?
André