You can start with checking installation log files for any errors. If there are no errors in install log, you can check the differences (like firewall , anti-virus , etc. ) between working Windows 2008 R2 x64 and new server.
For information on errors encountered during installation, review the installation log file located in the My Documents\DB2LOG\ directory. The log file uses the following format: DB2-ProductAbrrev-DateTime.log, for example, DB2-ESE-Tue Apr 04 17_04_45 2008.log.