Is anyone familiar with this kind of error message in db2diag.log

2009-06-16-13.20.04.406174-240 E3579141A1413 LEVEL: Error (OS)
PID : 520384 TID : 31098 PROC : asnqapp
INSTANCE: rptyesnz NODE : 000
APPID : 10.2.200.49.34911.090616151649
EDUID : 31098
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED : OS, -, malloc
OSERR : ENOMEM (12) "There is not enough memory available now."
MESSAGE : Private memory and/or virtual address space exhausted, or data ulimit
exceeded
DATA #1 : Soft data resource limit, PD_TYPE_RLIM_DATA_CUR, 8 bytes
251657728
DATA #2 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 8 bytes
266240
DATA #3 : Current set size, PD_TYPE_SET_SIZE, 8 bytes
107479040
CALLSTCK:
[0] 0x0900000002A76C50 pdLogSysRC + 0xB0
[1] 0x0900000002AA8758 sqloLogMemoryCondition + 0x278
[2] 0x0900000002AA7F04 sqlogmblkEx + 0xB24
[3] 0x09000000031C8DF4 sqlak_callbDrdaInput__FP14db2UCinterface + 0x49D4
[4] 0x0900000002D75BC0 sqljrGenSqlDta__FP14db2UCinterfaceUi + 0xCC0
[5] 0x0900000002BCB7FC sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo + 0x67C
[6] 0x0900000002FAE280 sqlacall + 0x19A0
[7] 0x00000001001097F0 runSQL__7subsSQLFP5sqldaP4stmtP5sqlca + 0x250
[8] 0x0000000100130080 insertRow__5QArowFP7appAgntP7agntSubPC10encodeInfo PC12conflictInfoC7ASNBOOL + 0x8A0
[9] 0x0000000100133588 applyRow__5QArowFP7appAgntP7agntSubPC10encodeInfoP C12conflictInfo + 0x428

I don't know replication, but check user data limit (ulimit -a) if rptyesnz is a 64-bit instance on AIX

The memory got exhausted on your machine. May be you can increase ulimit size to the following.

ulimit -Hn > 65536 or unlimited
ulimit -Hd == unlimited
ulimit -Hf == unlimited

ulimit -a is unlimited and the system has lot amount of free memory. I even checked the memory limit for the Q replication its more than enough. But still getting that error.

You don't give your version or OS, but I ran into a bug in an older version (AIX and v8.2) where ulimit settings were not picked up properly if the instance was started by a user who did not have underlying root authority. This was several years ago and I don't remember the details any more.