I eventually managed to figure it out.
It looks like my vmware workstation did not connect the esxi host as a physical host in my LAN, and so I was unable to ping it from the vCenter server appliance (however it worked the other way around and that's why I thought it was ok).
After connecting it to the physical LAN from the settings of VMware workstation, everything worked well.