Sybase is a company. Adaptive Server Enterprise, SQLAnywhere, and IQ are (some of) Sybase's products.
The short-short version of history is that MS needed a database platform on OS/2, and Sybase was really the first to have a commercial client-server DBMS so MS chose them and their 'SQL Server' product. After several years of a weird partnership they split and MS took the 'SQL Server' name.
The differences are probably not easily summed up in a single post (or thread!) but both follow the same basic architecture (log and data devices, locking, T-SQL basics, etc.) whereas MS is Windows only but ASE lives on Windows, Linux, Solaris, AIX, etc.