TylerDurden77 Try to tweak the Java heap size for the vCenter services, especially for the Inventory Service. Default value is 3GB.
Hereby useful links with more details:
Tweaking Java.exe memory usage on vCenter Server 5.1 « Default Reasoning
After the change you need to restart the services.
Even if those parameters in general affect the memory usage, they can help reducing the CPU consumption.
Br,
Angel