Check this site, but in short, - Named Pipes is not supported on MSDE. You need to create an alias on your server for MSDE instance using TCP/IP protocol, and specify the alias during server registration in EM. You can also use the IP address for registration.
I setup alias with TCP/IP and try to connect with alias name as well with tcp/ip, but same error message is comming. by the way what cold be my instance, i just install msde with setup.exe from command prompt.