OS: AIX 5.3.0.0
DB2/AIX64 9.5.4

When I try to select from SYSIBMADM.DBMCFG, getting the "SQL1042C An unexpected system error occurred. SQLSTATE=58004"

No useful information being logged in diaglog file.

Any one has come across this issue?

Thanks,
-Ram

I do not know what the problem is, but you are over a year out of date. FP 4 came out in September of 2009. You might want to update to the latest FP and see if they fixed it.

Andy

Thank you for the response Andy.
Yes, it is already in our plan.
But the thing is, this issue is only on one server with 2 instances. Same issue on any of the databases on this server.
We have other servers and instances with the same DB level and configurations and no issue.
Only the difference in this server I can see is, all the tablespace containers are of NFS and other servers have them as local filesystem containers.
Am I missing anything else to look into?

What's logged in the diag.log?

Do the instances/databases that report this error have the fenced ID setup the same way the instances/databases that work ok (ie. was the fenced ID used when creating an instance, is it part of the same group as the instance owner ID or not...)? Compare how the fenced ID is setup, if it's used, between working and non-working systems.

Its a while that these instances were created and I was not present then.
Here is the content from the diaglog when I execute this statement.

2011-01-10-12.52.20.823237-480 I5885478A450 LEVEL: Severe
PID : 409846 TID : 58811 PROC : db2sysc 0
INSTANCE: INSTNM NODE : 000 DB : DBNAME
APPHDL : 0-40438 APPID: *LOCAL.INSTNM.110110223700
AUTHID : INSTNM
EDUID : 58811 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq, probe:10
RETCODE : ZRC=0xFFFFFBEE=-1042

2011-01-10-12.52.20.823673-480 E5885929A3361 LEVEL: Severe
PID : 409846 TID : 58811 PROC : db2sysc 0
INSTANCE: INSTNM NODE : 000 DB : DBNAME
APPHDL : 0-40438 APPID: *LOCAL.INSTNM.110110223700
AUTHID : INSTNM
EDUID : 58811 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq, probe:910
DATA #1 : String, 58 bytes
marking fmp as unstable after new thread creation failure:
DATA #2 : String, 8 bytes
Fmp TID:
DATA #3 : Hexdump, 4 bytes
0x0780000001CDD1D0 : 0000 0000 ....
DATA #4 : String, 8 bytes
Fmp Row:
DATA #5 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 496 bytes
fmpPid: 512292
fmpPoolList Ptr: 0x0000000000000000 fmpForcedList Ptr: 0x0000000000000000
nextFmpCB Ptr: 0x078000000073aa00 prevFmpCB Ptr: 0x0000000000000000
fmpIPCList Ptr: 0x0780000001cdb9a0
stateFlags: 0x00000152 numFmp32Attaches: 0
numActiveThreads: 0 numPoolThreads: 0
fmpCodePage: 0 fmpRowUseCount: 1
active: 0x01 rowLoaderValidate: 0x00
ipcLatch:
0x0780000001CDB784 : 1000 0000 01A5 0000 ........
rowLatch:
0x0780000001CDB78C : 1000 0000 01A6 0000 ........
fmpAgentList:
0x0780000001CDB798 : 0008 0007 3336 0000 0000 0000 0000 0000 ....36..........
0x0780000001CDB7A8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB7B8 : 0000 0000 0000 0000 0000 0000 2A00 0000 ............*...
0x0780000001CDB7C8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB7D8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB7E8 : 0000 0000 2A00 0000 0000 0000 0000 0000 ....*...........
0x0780000001CDB7F8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB808 : 0000 0000 0000 0000 0000 0000 2A00 0000 ............*...
0x0780000001CDB818 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB828 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB838 : 0000 0000 2A00 0000 0000 0000 0000 0000 ....*...........
0x0780000001CDB848 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB858 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB868 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB878 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB888 : 0000 0000 5345 4D41 5048 4F52 4520 5741 ....SEMAPHORE WA
0x0780000001CDB898 : 4954 0000 0000 0000 0000 0000 0000 0000 IT..............
0x0780000001CDB8A8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB8B8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB8C8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB8D8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB8E8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB8F8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB908 : 0780 0000 01CD B780 0780 0000 01CD FF00 ................
0x0780000001CDB918 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0780000001CDB928 : 0000 0001 0000 0000 ........

2011-01-10-12.52.20.824169-480 I5889291A447 LEVEL: Severe
PID : 409846 TID : 58811 PROC : db2sysc 0
INSTANCE: INSTNM NODE : 000 DB : DBNAME
APPHDL : 0-40438 APPID: *LOCAL.INSTNM.110110223700
AUTHID : INSTNM
EDUID : 58811 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerGetFmpThread, probe:20
RETCODE : ZRC=0xFFFFFBEE=-1042

2011-01-10-12.52.20.824459-480 E5889739A614 LEVEL: Error
PID : 409846 TID : 58811 PROC : db2sysc 0
INSTANCE: INSTNM NODE : 000 DB : DBNAME
APPHDL : 0-40438 APPID: *LOCAL.INSTNM.110110223700
AUTHID : INSTNM
EDUID : 58811 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerRemoveAllIPCforRow, probe:10
DATA #1 : String, 32 bytes
Freeing IPC resource explicitly:
DATA #2 : Process ID, 4 bytes
512292
DATA #3 : Hexdump, 4 bytes
0x0780000001CDB9A0 : 0000 0000 ....

2011-01-10-12.52.20.824722-480 E5890354A517 LEVEL: Error
PID : 409846 TID : 58811 PROC : db2sysc 0
INSTANCE: INSTNM NODE : 000 DB : DBNAME
APPHDL : 0-40438 APPID: *LOCAL.INSTNM.110110223700
AUTHID : INSTNM
EDUID : 58811 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerRemoveAllIPCforRow, probe:20
DATA #1 : String, 22 bytes
IPC resources Address:
DATA #2 : Pointer, 8 bytes
0x0780000010030080

2011-01-10-12.52.20.824984-480 E5890872A1351 LEVEL: Error
PID : 409846 TID : 58811 PROC : db2sysc 0
INSTANCE: INSTNM NODE : 000 DB : DBNAME
APPHDL : 0-40438 APPID: *LOCAL.INSTNM.110110223700
AUTHID : INSTNM
EDUID : 58811 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerRemoveAllIPCforRow, probe:30
DATA #1 : String, 29 bytes
Number of IPC resource found:
DATA #2 : signed integer, 4 bytes
1
DATA #3 : String, 29 bytes
Number of IPC resource freed:
DATA #4 : signed integer, 4 bytes
1
CALLSTCK:
[0] 0x0900000006F60D18 pdLog + 0x88
[1] 0x0900000007890DA4 sqlerRemoveAllIPCforRow__FP11sqlerFmpRowb + 0x578
[2] 0x09000000091F30D0 sqlerRemoveFmpFromTable__FP11sqlerFmpRowb + 0x2B0
[3] 0x090000000788F014 @135@sqlerShutdownFMP__FP11sqlerFmpRowP14sqlerFmpH andleP13sqlerFmpTableP8sqeAgentbT5 + 0x65C
[4] 0x09000000078AC4C4 sqlerGetFmpFromPool__FP14sqlerFmpHandleP13sqlerFmp Parms + 0x4C0
[5] 0x09000000078B6A48 sqlerInvokeFencedRoutine__FP13sqlerFmpParms + 0x2720
[6] 0x0900000006B275AC sqlriInvokeInvoker__FP10sqlri_ufob + 0xA3C
[7] 0x0900000008199178 sqlriutf__FP8sqlrr_cb + 0x130
[8] 0x0900000007135240 sqlri_tfopn__FP8sqlrr_cbP9sqlri_tao + 0x1D4
[9] 0x0900000006D5EBD0 sqlriopn__FP8sqlrr_cbP9sqlri_taoPi + 0x75C

Check the following file for each instance:
$DB2HOME/sqllib/ctrl/.fencedID

This will tell you if the fenced ID was used during instance creation. Take a look at this link for additional info:
https://www-304.ibm.com/support/docv...id=swg21166020

If it contains the fenced ID, compare how this ID is setup on working and non-working systems (ie. check if it's part of the same group as the instance owner ID).

Okay, I see the difference between working and non-working servers. Thanks for pointing me to the right place. Here are the details.
On working servers, the Fenced User is the same as instance owner and
on non-working, they different. However, the Fenced User and instance owner are part of same SYSADM group but not the DBCTRL group. Could that be the reason?
Any system level changes (like group changes) takes longer time in our environment, is there a way to come over this issue as instance owner?

Appreciate for your info.

I suspect that the fenced ID doesn't have read access on some directory and/or file owned by the instance owner (only owner had read access). Check if the fenced ID can access all directories in $DB2HOME/sqllib and $DB2HOME/sqllib/db2systm file (this is dbm cfg file). As a test, you can login as the fenced and cd to $DB2HOME/sqllib. If the permissions look ok, one workaround would be to drop and re-create the instance without using the fenced ID (if the fenced ID is not required). You can use db2cfexp/db2cfimp to save and re-import config info.

Or open a PMR to report this problem and they can check if this is some known bug / suggest another workaround.

The permissions seems to be fine and permissions granted to sqllib folder and db2systm file in it to all. I will schedule for outage for the instance and recreate it.
Thank you very much.

Finally I was able to schedule for downtime, had root to drop and recreate instances and everything is back to normal. Below are the steps I did.

1. Export the connectivity configuration information to export profile. (db2cfexp)
2. Stop replication on all environments
3. Have root to perform the instance drop and recreation. (db2idrop <instname>, db2icrt -u <instname> <instname>
4. Import the connectivity configuration files generated in step#1 (db2cfimp)
5. Start replication in all the environments