Yes it is possible to directly manage the ESXi host via vsphere client, but you will not be able to get the unique and good features of vCenter server like HA and DRS.
So which means if your host goes down, your runnning VM's will not have a fail over host.