I don't know a clue on this error but I have successfully installed UDB 8.1 on Redhat 2.1 Advanced Server with neccessary Kernal Upgrades. Why don't you check the supported OS versions for your UDB installation from IBM site.
Modifying kernel parameters (Linux)
Before installing DB2, you may want to update Linux kernel parameters. DB2 will automatically raise the IPC limits where necessary. You may still want to raise these limits depending on your particular needs.
You must have root authority to modify kernel parameters.
To update kernel parameters:
RedHat and SuSE
Systems using a 2.4.x series kernel have a default value for the message queue parameter (msgmni), which allows only a few simultaneous connections to DB2. Semaphore array parameters also have to be changed for DB2 to run successfully. To check shared memory segment, semaphore array, and message queue limits, issue the ipcs -l command.
The following is the output from the the ipcs -l command.
# ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096 // SHMMNI
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 1024 // SEMMNI
max semaphores per array = 250
max semaphores system wide = 128000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 1024 // MSGMNI
max size of message (bytes) = 65536
default max size of queue (bytes) = 16384 // MSGMAX
Modify the kernel parameters by adding the following entries to the default system control configuration file, /etc/sysctl.conf: