db2 is getting ENOMEM (out of memory) from pwrite() system call, but there is enough free memory as per free/vmstat. Linux system message log doesn't contain any memory related errors.


From db2diag.log:

FUNCTION: DB2 UDB, oper system services, sqloseekwrite64, probe:40
MESSAGE : ZRC=0x870F0024=-2029060060=SQLO_MEM "out of memory"
DIA8533C The system memory limit was reached.
CALLED : OS, -, pwrite OSERR: ENOMEM (12)

...

FUNCTION: DB2 UDB, buffer pool services, sqlbWriteBlocks, probe:20
MESSAGE : ZRC=0x870F0024=-2029060060=SQLO_MEM "out of memory"
DIA8533C The system memory limit was reached.
DATA #1 : String, 35 bytes
No more memory available in system.

....

FUNCTION: DB2 UDB, buffer pool services, sqlbWritePage, probe:20
MESSAGE : ZRC=0x85020005=-2063466491=SQLB_NO_MEMORY
"No pageread because no-mem error from OSS"
DIA8305C Memory allocation failure occurred.




Memory Usage Info, '/usr/bin/free'

total used free shared buffers cached
Mem: 16433336 9566916 6866420 0 388560 8411152
-/+ buffers/cache: 767204 15666132
Swap: 1020088 0 1020088


Memory Usage Info, '/usr/bin/vmstat 1 5'

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 6866428 388560 8411108 0 0 1 23 1 1 0 1 99 0 0
0 0 0 6866428 388560 8411148 0 0 0 46 1188 978 0 0 100 0 0
0 0 0 6866428 388560 8411148 0 0 0 66 1081 964 0 0 100 0 0
0 0 0 6866492 388560 8411152 0 0 0 117 1117 941 0 0 100 0 0
0 0 0 6866420 388560 8411152 0 0 8 53 1076 962 0 0 100 0 0



So, it could that some memory is pinned at the OS level??? On AIX, I can check for v_pinshm and use vmstat -v / svmon to check amount of pinned memory. Does anyone know how to check for the same on Linux?


RHEL 5.2 , x86_64

I would make sure your Linux kernel parms are set correctly per the following output:
ipcs –l
Check the online doc for recommendations on setting kernel parms, depending on which DB2 version you have.

I would also recommend that you consider turning off STMM, or at least hard-coding memory for bufferpools, LOCKLIST, MAXLOCKS, and SORTHEAP.

Yes, I checked kernel parameters and they look ok:


IPC Resource Limits, '/usr/bin/ipcs -l'


------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 1024
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536



Instance "xxxxxxxxx" uses "64" bits and DB2 code release "SQL09055"
with level identifier "06060107".
Informational tokens are "DB2 v9.5.0.5", "s091123", "MI00316", Fix Pack "5".
DATA #2 : System Info, 440 bytes
System: Linux x88cpdc 6 2 x86_64
CPU: total:4 online:4 Cores per socket:4 Threading degree per core:1
Physical Memory(MB): total:16049 free:6648
Virtual Memory(MB): total:17046 free:7645
Swap Memory(MB): total:997 free:997
Kernel Params: msgMaxMessageSize:65536 msgMsgMap:65536 msgMaxQueueIDs:1024
msgNumberOfHeaders:65536 msgMaxQueueSize:65536
msgMaxSegmentSize:16 shmMax:68719476736 shmMin:1 shmIDs:4096
shmSegments:4096 semMap:32000 semIDs:1024 semNum:32000
semUndo:32000 semNumPerID:250 semOps:32 semUndoSize:20
semMaxVal:32767 semAdjustOnExit:32767
Cur cpu time limit (seconds) = 0xFFFFFFFF
Cur file size limit (bytes) = 0xFFFFFFFF
Cur data size (bytes) = 0xFFFFFFFF
Cur stack size (bytes) = 0x00A00000
Cur core size (bytes) = 0x00000000
Cur memory size (bytes) = 0xFFFFFFFF
nofiles (descriptors) = 0x00000400



STTM should be off, I don't see any activity in the STMM logs.


I also made sure that their Linux env meets system requirements as per:
IBM developerWorks: Wikis - Information Management - Red Hat Enterprise Linux (RHEL) 5 - DB2 9.5


This problem occurs during db migration from v8 to v9.5 (via restore), during system catalog migration step - migrating existing tables or creating new ones. The db is very small (backup image is ~160MB if compressed during backup and ~1GB if uncompressed) so it's not clear why db2 is getting an out of memory from the OS... Restoring the same backup image back to v8 completed successfully (cat migration is not needed).

They use VCS and some vormetric FS guard software (for encryption), don't know if this is related or not.


Any idea on how to check for amount of pinned memory at the OS level?