As far as I know there is no sp_dumptrans stored procedure provided by Sybase. may be either its a customised procedure of your environment or some sp that sybase internally use while dumping the trans.
for clearing up the log system, you need to take a transaction dump by using below command
dump tran <db_name> to 'file name with complete path>
or set the trunc log on chkpt db option for the database in question.