enviroment: sap ecc6, db2 version 09.01.0004, windows 2003 x64
sympton: the db2 crash suddenly at 2010.07.05 22:29. No clear clue is shown in db2dialog.log. No useful log is shown in os log except message like db2 need to shutdown.
will it be a bug? give me some hints pls.

Your db2diag.log file has numerous dump messages and such. You should open a PMR with IBM support to determine the cause.

Andy

Pantao, there are a couple of clues eventually useful to open a pmr

- the instance has started to crash just after the automatic runstats

- the instance has started the operations of automatic resize apparently lowering the resource requests

hi arwinner and tibullo
for the limited sap/ibm contract, sap/ibm is not charge of db2 software maintenance.so i can't open a pmr with ibm or create message to sap.


hi tibullo
automatic runstats run without any problem before.so maybe the crash is not caused by runstats.

I don't think crash is related to automatic runstats which completed at 21:05.

PID 1992 / TID 4144 trapped:

2010-07-05-22.29.39.312000+480 I299768520F515 LEVEL: Error
PID : 1992 TID : 4144 PROC : db2syscs.exe
INSTANCE: DB2PRD NODE : 000 DB : PRD
APPHDL : 0-1673 APPID: 192.168.2.160.3862.100705051103
AUTHID : SAPPRD
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:1
MESSAGE : Error in agent servicing application with coor_node:
DATA #1 : Hexdump, 2 bytes
0x0000000149938242 : 0000 ..


Application - disp+work.EXE

Statement it was executing at the time of the crash:

0x0000000114ACF020 : 5345 4C45 4354 202A 2046 524F 4D20 2254 SELECT * FROM "T
0x0000000114ACF030 : 4E4F 4445 3031 2220 5748 4552 4520 2250 NODE01" WHERE "P
0x0000000114ACF040 : 4152 454E 545F 4944 2220 3D20 3F20 414E ARENT_ID" = ? AN
0x0000000114ACF050 : 4420 2245 5854 5F4B 4559 2220 3D20 3F20 D "EXT_KEY" = ?
0x0000000114ACF060 : 2057 4954 4820 5552 202D 2D20 4F50 544C WITH UR -- OPTL
0x0000000114ACF070 : 4556 454C 2820 3520 2920 2D2D 2051 5545 EVEL( 5 ) -- QUE
0x0000000114ACF080 : 5259 5F44 4547 5245 4528 2031 2029 202D RY_DEGREE( 1 ) -
0x0000000114ACF090 : 2D20 4C4F 4341 5449 4F4E 2820 5341 504C - LOCATION( SAPL
0x0000000114ACF0A0 : 5348 4932 202C 2037 3639 2029 202D 2D20 SHI2 , 769 ) --
0x0000000114ACF0B0 : 5359 5354 454D 2820 5052 4420 2C20 5341 SYSTEM( PRD , SA
0x0000000114ACF0C0 : 5050 5244 2029 PPRD )


A trap file was supposed to be generated (try looking for DB19924144.TRP in the same dir where you have db2diag.log) and format it using db2xprt:
DB2 Database for Linux, UNIX, and Windows


In the formatted output, look for a "Failing instruction" and this will give you the function where it trapped. Then, you can search for known problems using this function name here:
IBM Support Portal - DB2 for Linux, UNIX and Windows

hi,db2girl
thank u for your analysis method first.
the trap file has been reformated and attached.u can be open with notepad.

which keyword should i use in ibm support portal? i'm a newbee.

i do not understand why so many people, not knowing db2, should handle dba jobs - problem determination..
the first thing todo;; is to read the doc carefully and try to practice db2 and see if you understand something...

Oh, well, sometimes there is some pressure from the ranks and some good willing people tries to perform a duty even from a disadvantaged position...

In a perfect world this shouldn't happen.

You can search for sqlns_qgs, but you won't find a good match. This crash is related to the following APAR:
IBM IZ05531: HIT -901 DUE TO "SECTION IDS DO NOT MATCH" WHEN PERFORM SQL QUERIES. - United States


You will find that the information in the db2diag.log / trap file doesn't match the APAR description, but it's

You're on v9.1 fp4 + SAP_19276, you need to upgrade to fp5+ or get another SAP build that contains this fix (SAP_20061 has this fix if that helps).

hi, db2girl
i'also found sap note 1173256 - DB6: 9.1 FP4SAP database might crash due to a bad pointer.the solution is given as u said. i wonder should i apply fp5 or fp8? the newest fp is fp9.

i'm a sap basis in fact.

It's really up to you... fp5 or higher but might as well upgrade to the latest.

hi, db2girl
i prefer to fp5.for every fp has several bugs, no fp is an exception. i chose the fp with less bugs.
thanks for your kindly help again!

Two things,

1. This is a DB2 BUG. This will bother you when your database uses external stored procedures ( e.g java stored procedures). External stored procedures run in fenced mode. Such procedures can crash DB2 in many situations, because db2 does not have control over the code. that code even ineract with OS, without telling DB2.

2. IBM knows of it. Hopefully fixpack 7 will include the complete solution. They had included the fix in FP6 but we still had issues. ( we obtained a special build). The issue is known as CPU CORE ISSUE with two digit cores.What it means is that on system with over 9 cores DB2 cannot manage FMP's properly. Your cpu is 16 cores. We had this issue on 16 cores on win 2003 in our case.

In your diaglog file I found only one FMP terminated which brought the instance down.

In my situations there were many such entries ( about 6-10 ) before instance goes down.

Before anything else, if your application uses fenced mode processes, make sure it does not use memory directly. In my case it was wasting (leaking) memory not being ablke to return memory to pool.



Regards

Regards

Please be warned that we had moved from 3b to 4 and 4 to 5. IBM had built 2 hotfixes for us, the first one had not helped, we are applying second one this Wednesday.

Regards

Sorry, but I was taking about db2 v9.5. I will try to find equivalent fixpack for 9.1