Hi,
You stated that DOMAINA\GROUPA contains user that are not member of DOMAINA. But, when declaring a user directly, it works. That means that, SSO works correctly, or at least, as expected.
Your "problem" is in fact an expected behavior of SSO 5.5. Nested groups are not recognized anymore. That means that Groups of DOMAINA have to contain members of DOMAINA. If you want user from DOMAINB to be recognized, declare them directly or use groups from DOMAINB with users from DOMAINB.
You should have a look at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2059528
This is not exactly your problem but seems to be related.
Hope it helps.