what exactly you execute?
When you need simple dump file you should executy only:
dbexport dbs -o path_to_some_directory
dbs- is name of database you want to backup
In path_to_some_directory directory is created the new one "dbs.exp" which contains the dump files.
does output_dir already exist? The exportdirectory has to exist before using dbexport this way. It's a strange notification though and maybe it's caused by an omitted TAPEDEV value in the onconfig file. Check this out in UNIX with
onstat -c | grep TAPEDEV
If it has no value because you don't use a tapedevice set the value to /dev/null, stop and re-initialize the server.