I created a new group with an underscore instead of a space ("vcenter_Admins" instead of "domain admins"). I added that group to SSO and to vCenter and everything seems to work fine now.
I could never find a good explanation for the login issues (other than the space character hunch). I could login via web client, but not full client so I think permissions were fine. I added the domain admins group to everything I thought it was needed. I already know about the nested group issue in 5.5.