This gives the actual number of buffers sent and received. But I wanted to know theorotically what does it mean by setting the value of FCM_NUM_BUFFERS and FCM_NUM_CHANNELS ? How much memory is allocated per partition for FCM based on these values ?
In our environment FCM_NUM_BUFFERS is set to a value of 200,000 4KB pages this should be equal to 762 MB per partition. Totally we have 17 partitions spanned across 3 physical nodes so the maximum FCM memory should come around 17*0.762 GB = 12.96 GB. When I see memory usage in db2top, it gives that the total FCM memory usage as 58 GB. So where is this extra memory coming in to picture? Does FCM_NUM_CHANNELS come in to picture while calculating the memory ?
FCM buffers are shared by all logical partitions on the server. For each physical server, FCM_NUM_BUFFERS x (# of logical partitions + 2 failover partitions)
(2 failover partitions is by default, controlled with DB2_NUM_FAILOVER_NODES)
For Linux, db2 calculates a value based on the above + 25% for larger configuration. It's rounded up to multiple of 2GB and then we add on an additional 2GB
(not exactly sure if this is only for automatic setting, I think this is for automatic and fixed value but don't quote me here).
You have 3 physical / 17 logical partitions. I assume one partition is on its own server (for coordinator node), leaving 2 servers with 8 logical partitons each.
For each server with 8 logical partitons:
200000 * (8 + 2) = 2000000 4K = ~7.6GB
If 25% plus rounding... is done for non-automatic setting, then:
7.6GB x 1.25 = 9.5GB, round up to 10GB + 2GB = 12GB
For server with 1 logical partition
200000 * (1 + 2) = 600000 4K = ~2.3GB
... = 6GB
db2 will allocate a fixed size segment based on the above (one segment per server). You can see the size of this segment in the ipcs output - this is allocated but not committed in RAM. You can check the segment size using:
- Find the Id value (3rd column) for FCM in db2pd -memsets
- ipcs | grep for that ID value. The bytes column is the segment size
I think db2top is doing double counting for FCM memory (comes from global snapshot?), should only count once per server. Try using db2pd.
If you notice, size on the operating system level is 6442450944 bytes which is 6 GB. This is exactly as calculated by you. But why is the size in db2pd –memsets showing 3314112 KB ? This is just 3.1 GB where is the rest of the memory going ?
Data Node1 : [Eight logical partitions]
db2pd -memsets | head -5; db2pd -memsets | grep -i fcm
Database Partition 1 -- Active -- Up 5 days 05:48:14 -- Date 11/14/2011 00:19:54
Name Address Id Size(Kb) Key DBP Type Unrsv(Kb) Used(Kb) HWM(Kb) Cmt(Kb) Uncmt(Kb)
FCM 0x0000000220000000 344195095 11130240 0xE164E962 0 11 7284352 3845888 6682624 3845888 7284352
This is similar to that of Data Node 1 where OS size is 15032385536 [14 GB] and db2pd –memsets showing a value of 11130240 kb [10.61 GB]. Not sure where the rest of the memory is used up? And also not sure why it is 14 GB at OS level unlike your calculation which should be 12 GB.
My calculations were based on FCM_NUM_BUFFERS value. There are also channels... The pre-allocated segment size is very close to what I estimated. Note that this is a pre-allocated size (fixed size - can't grow beyond this value), memory in not actually being used/committed unless it's needed. Cmt from memsets is what you need to look at to see the actual memory usage (only count once per physical server).
I see you opened a pmr this morning. You can tell them I'm familiar with this and to engage me, if needed.