DB2 on Windows server, IBM AIX server, Solaris Server, HP-UX server or Linux (Red Hat Enterprise) server, which better? compatible and perfomance?

I would say that AIX and Linux are above the rest. Windows is OK if you don't have a lot of CPU cores (Windows OS doesn't scale as well as UNIX/Linux if you have a lot of CPU cores). Sun Solaris performs well, but not many users so you tend to find more bugs. HP/UX has very few users and I would definitely avoid that one.

Price wise (partially because it runs on AMD and Intel processors which means cheaper hardware and lower DB2 PVU pricing), Linux is probably the winner.

Hard to say. I'd give AIX the benefit of the doubt, escpially if you are going to exloit the new column format "decfloat". Numeric data stored in this format is natively implemented in the instruction-set of the new Power6 CPU. This should benefit.
On the other hand: the new ZFS filesystem in solaris looks really impressive.

If your hardware and software budgets are unlimited, AIX on IBM P6 hardware will perform better, but for a given amount of money you can buy more AMD/Intel processors and more AMD/Intel DB2 PVU licenses than you can on AIX and P6.

The PVU pricing is supposed to even-out the differences, but I still think for a given amount of money you would come out ahead with Linux on AMD/Intel.

I have run Linux on a P5 IBM box (not my decision), and I don't know what the performance difference is between that and running AIX, but probably not much. But If I had a P Series box, I would prefer to run AIX because that is what the vast majority of other customers are running on P-Series hardware and there are likely to be fewer bugs in that particular DB2 code.

Performance tuning is done on all platforms that DB2 LUW ships on. Likewise, all platforms are equally regression-tested. So I wouldn't expect much differences aside from different hardware/operating system support and special processor instructions as was already mentioned above. However, the major performance issue is typically not the underlying platform but rather the tuning of the database system. For example, a system with slower disks may still easily outperform another system with faster disks if indexes or statistics are missing.

Typically, the deciding factor is what you are most familiar with and which skills are available in your shop. If you run a pure-Windows shop, it may not be wise to start with a Solaris server.