Process Explorer is a very good tool which I use often.
BUT it will not help you in any way to find if a RootKit exists on your Server unless you happen to know specifically what process name the RootKit might be running under.
Even then, it will not help you to permanently remove the RootKit.
If you do a Google search for RootKit you will find a number of tools which all advertize that do the job better than their competition. With that in mind, I'd suggest that you try one or two.
The one that I have used is ThreatFire
It also claims to be better than the others, but cannot certify that claim.
I am guessing that if your other tools have not found a RootKit problem there is a high probability that there is none and that your SQL Server performance problems come from another source. But it doesn't hurt to keep looking.
Did your SQL Server use to work with acceptable performance and now no longer does so?
If so, then what changed?
I'd look towards that as at least a contributing factor (if not the absolute cause itself) in less performance.
In general most SQL Server performance issues are caused by a poorly 'tuned' SQL Server and/or an application developed in a manner which does not efficiently utilize the SQL Server - a problem which is magnified as data table size increases.
With that in mind I'd recommend getting a SQL Server tune-up (there should be some contractor/consultant in your area who can do this).
Check to see if the various SQL Server database parameters are set to optimally use the Server resources so that you don't 'bottle neck' your performance with un-expected resource utilization.
Good Luck