How to find which table is deadlock in sp_who ?
What R the setting done after installation to increse the performance ?
What R the memory setting done after installation ?
What is default size of system D/b?
To determine deadlocks in sp_who, look at the blk_spid value for the blocked process. Then look to see if the blocker spid process is being blocked. If you find two processes that have eachothers spid in blk_spid, then they are deadlocked.
There are many configuration options that can be used to tune performance. Most depend on your application and usage patterns. I would suggest reviewing the Sybase Performance and Tuning Guide available on their site.
You will need to tune the memory parameters based on the amount of available memory in your system. Run sp_configure 'memory' to get a list of available memory tuning options. More is generally better.
Not sure what you mean by system db. There are several default system databases. I size the master db at 60MB, have never had to extend it.