Results 1 to 14 of 14
  1. #1
    Join Date
    Jul 2013
    Posts
    31

    Unanswered: DB2 is hanged, urgent help is needed.

    Im not able to connect to the database or restart the instance.
    I can list the db directory & I can do nothing else. db2diag shows the following error

    2013-07-09-22.11.27.769055-330 I13076181A3239 LEVEL: Severe
    PID : 618576 TID : 5656 PROC : db2sysc 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 5656 EDUNAME: db2pclnr (BS7) 0
    FUNCTION: DB2 UDB, buffer pool services, sqlbClnrAsyncWriteCompletion, probe:300
    DATA #1 : String, 13 bytes
    *ioRequest =
    DATA #2 : Hexdump, 184 bytes
    0x0000000110900BE8 : 0000 0002 0000 00B2 0000 0000 0000 0000 ................
    0x0000000110900BF8 : 0000 0000 0000 0000 0700 0000 481A C000 ............H...
    0x0000000110900C08 : FFFF FFFF FFFF FFFF 0000 0001 0000 0000 ................
    0x0000000110900C18 : 0000 0000 0000 4000 0200 0000 0000 0000 ......@.........
    0x0000000110900C28 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
    0x0000000110900C38 : 0000 0004 FFFF FFFF 0000 0001 1090 0BF0 ................
    0x0000000110900C48 : 0000 0000 351C 8000 0000 0000 0000 0000 ....5...........
    0x0000000110900C58 : 0900 1000 A7D1 B578 0000 0000 0000 0000 .......x........
    0x0000000110900C68 : 0000 0001 1090 0BC0 0000 0001 1090 0580 ................
    0x0000000110900C78 : 0000 0000 0000 0000 0000 0001 1090 0CF8 ................
    0x0000000110900C88 : 0000 0000 0000 0000 FFFF FFFF FFFF FFFF ................
    0x0000000110900C98 : 0000 0002 860F 0004 ........
    DATA #3 : Buffer page descriptor, PD_TYPE_SQLB_BPD, 152 bytes
    Pagekey: {pool:14;obj:9;type:0} PPNum:162
    objectPageNum: 162
    bucketGroupHashIndex: 28894
    hashnext: 0x0000000000000000
    hashprev: 0x0000000000000000
    osslatch:
    {
    state = 0x0000000000010000
    = {
    held X: 1
    reserved for X: 0
    shared holders: 0
    firstSharIndex: 0x0
    firstExclIndex: 0x0
    }
    starve X mode = true
    xWaitCount = 0
    requestCount = 0
    identity = SQLB_BPD::bpdLatch (43)
    }
    UFlags: 103
    pageStatusFlag: 0
    permanentState: 0
    transientState: 0
    fixcount: 1
    bpp: 0x07000000481ac000
    pinCount:
    0x070000003AAACF50 : 0000 0000 ....
    weight: 01
    dirtyprev: 0x0000000000000000
    dirtynext: 0x070000003aaaf4d8
    blockDesc: 0x0000000000000000
    pminlsn: 00000048EDE38907
    hatenext: 0x0000000000000000
    hateprev: 0x0000000000000000
    onWhichHateList: 65535
    permanentHateList: 5
    newbpdidx: 3221225472
    diskRelativePageID: 217540
    DATA #4 : Page header, PD_TYPE_SQLB_PAGE_HEAD, 48 bytes
    pageHead: {tbspSeed:14;obj:9;type:0} PPNum:217540 OPNum:162
    begoff: 48
    datlen: 16336
    pagebinx: 9637
    revnum: 1280
    pagelsn: 00000048EDE38907 flag: a
    future32: 0
    cbits1to31: 40010501
    cbits32to63: 0

    2013-07-09-22.11.27.896165-330 I13126603A423 LEVEL: Severe
    PID : 618576 TID : 5656 PROC : db2sysc 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 5656 EDUNAME: db2pclnr (BS7) 0
    FUNCTION: DB2 UDB, base sys utilities, sqle_panic, probe:10
    MESSAGE : sqle_panic: Panic/sleep =
    DATA #1 : Hexdump, 4 bytes
    0x07800000002A020C : 0000 0000 ....

    2013-07-09-22.11.27.900057-330 I13127027A423 LEVEL: Severe
    PID : 618576 TID : 5913 PROC : db2sysc 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 5913 EDUNAME: db2pclnr (BS7) 0
    FUNCTION: DB2 UDB, base sys utilities, sqle_panic, probe:10
    MESSAGE : sqle_panic: Panic/sleep =
    DATA #1 : Hexdump, 4 bytes
    0x07800000002A020C : 0000 0000 ....

    2013-07-09-22.11.27.904116-330 I13127451A423 LEVEL: Severe
    PID : 618576 TID : 5399 PROC : db2sysc 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 5399 EDUNAME: db2pclnr (BS7) 0
    FUNCTION: DB2 UDB, base sys utilities, sqle_panic, probe:10
    MESSAGE : sqle_panic: Panic/sleep =
    DATA #1 : Hexdump, 4 bytes
    0x07800000002A020C : 0000 0000 ....

  2. #2
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Quote Originally Posted by db2j View Post
    Im not able to ... restart the instance.
    What do you mean by "unable"? Did you forget the command? Is your computer turned on? If you did something specific, please explain what exactly and what was the result. You may also want to include a larger fragment of db2diag.log, showing the initial error.
    ---
    "It does not work" is not a valid problem statement.

  3. #3
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    In the db2diag.log, look for the error just prior to the first one you posted. And check OS logs for any errors.

  4. #4
    Join Date
    Jul 2013
    Posts
    31
    Whenever I issue connect to dbname command it hangs indefinitely, so is the case with db2stop.

    here is the log

    2013-07-09-22.11.37.181565-330 I13129162A468 LEVEL: Warning
    PID : 614762 TID : 1 PROC : db2vend (PD Vendor Process - 258
    INSTANCE: db2bs7 NODE : 000
    EDUID : 1 EDUNAME: db2vend (PD Vendor Process - 258
    FUNCTION: DB2 UDB, trace services, pdInvokeCalloutScript, probe:110
    MESSAGE : Detected end of execution of call-out script
    DATA #1 : Process ID, 4 bytes
    520402
    DATA #2 : unsigned integer, 4 bytes
    9

    2013-07-09-22.11.37.186047-330 I13129631A399 LEVEL: Event
    PID : 614762 TID : 1 PROC : db2vend (PD Vendor Process - 258
    INSTANCE: db2bs7 NODE : 000
    EDUID : 1 EDUNAME: db2vend (PD Vendor Process - 258
    FUNCTION: DB2 UDB, trace services, pdInvokeCalloutScript, probe:20
    STOP : Completed invoking /db2/db2bs7/sqllib/bin/db2cos_trap

    2013-07-09-22.11.37.193906-330 E13130031A1228 LEVEL: Critical
    PID : 618576 TID : 4885 PROC : db2sysc 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 4885 EDUNAME: db2pclnr (BS7) 0
    FUNCTION: DB2 UDB, oper system services, sqloEDUCodeTrapHandler, probe:999
    MESSAGE : ADM14001C An unexpected and critical error has occurred: "Panic".
    The instance may have been shutdown as a result. "Automatic" FODC
    (First Occurrence Data Capture) has been invoked and diagnostic
    information has been recorded in directory
    "/db2/BS7/db2dump/FODC_Panic_2013-07-09-22.11.27.813371_0000/".
    Please look in this directory for detailed evidence about what
    happened and contact IBM support if necessary to diagnose the
    problem.
    DATA #1 : Signal Number Recieved, 4 bytes
    6
    DATA #2 : Siginfo, 64 bytes
    0x070000000B7FD2F0 : 0000 0006 0000 0000 0000 0009 0000 0000 ................
    0x070000000B7FD300 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
    0x070000000B7FD310 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
    0x070000000B7FD320 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

    2013-07-09-22.11.37.194070-330 I13131260A438 LEVEL: Event
    PID : 614762 TID : 1 PROC : db2vend (PD Vendor Process - 258
    INSTANCE: db2bs7 NODE : 000
    EDUID : 1 EDUNAME: db2vend (PD Vendor Process - 258
    FUNCTION: DB2 UDB, trace services, pdInvokeCalloutScript, probe:10
    START : Invoking /db2/db2bs7/sqllib/bin/db2cos_trap from oper system services sqloEDUCodeTrapHandler

    2013-07-09-22.11.37.560441-330 E13131699A546 LEVEL: Severe
    PID : 487824 TID : 258 PROC : db2wdog 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 258 EDUNAME: db2wdog 0
    FUNCTION: DB2 UDB, base sys utilities, sqleWatchDog, probe:20
    MESSAGE : ADM0503C An unexpected internal processing error has occurred. All
    DB2 processes associated with this instance have been shutdown.
    Diagnostic information has been recorded. Contact IBM Support for
    further assistance.

    2013-07-09-22.11.37.615908-330 E13132246A421 LEVEL: Error
    PID : 487824 TID : 258 PROC : db2wdog 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 258 EDUNAME: db2wdog 0
    FUNCTION: DB2 UDB, base sys utilities, sqleWatchDog, probe:21
    DATA #1 : Process ID, 4 bytes
    618576
    DATA #2 : Hexdump, 8 bytes
    0x07000000003FDF70 : 0000 0101 0000 0006 ........

    2013-07-09-22.11.37.616023-330 I13132668A388 LEVEL: Severe
    PID : 487824 TID : 258 PROC : db2wdog 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 258 EDUNAME: db2wdog 0
    FUNCTION: DB2 UDB, base sys utilities, sqleCleanupResources, probe:5
    DATA #1 : Hexdump, 4 bytes
    0x07000000003FDEF0 : 0000 0101 ....

    2013-07-09-22.11.37.616098-330 I13133057A453 LEVEL: Warning
    PID : 487824 TID : 258 PROC : db2wdog 0
    INSTANCE: db2bs7 NODE : 000
    EDUID : 258 EDUNAME: db2wdog 0
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerKillAllFmps, probe:5
    MESSAGE : Bringing down all db2fmp processes as part of db2stop
    DATA #1 : Hexdump, 4 bytes
    0x07000000003FDC60 : 0000 0000 ....

  5. #5
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    These messages are not helpful.

    If the existing db2diag.log is not too big, attach it here. If it's a large file, use the tail command to copy all messages from today to a file and attach it. Include your OS and DB2 version.

  6. #6
    Join Date
    Jul 2013
    Posts
    31
    Hi, Please find the log attached.
    Attached Files Attached Files

  7. #7
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    Try the following:

    - db2stop + ipclean or db2_kill + ipclean
    - clean-up any leftover processes / IPC resouces owned by db2bs7
    - update instance using db2iupdt command
    - db2start
    - db2 connect ....

  8. #8
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Code:
    2013-07-09-22.11.27.715083-330 E13003588A874      LEVEL: Error (OS)
    PID     : 618576               TID  : 5656        PROC : db2sysc 0
    INSTANCE: db2bs7               NODE : 000
    EDUID   : 5656                 EDUNAME: db2pclnr (BS7) 0
    FUNCTION: DB2 UDB, oper system services, sqloDispatchNBlocks, probe:100
    MESSAGE : ZRC=0x860F0004=-2045837308=SQLO_DGFL "general failure (DD)"
              DIA8402C A disk error has occurred.
    CALLED  : OS, -, lio_listio
    OSERR   : EPERM (1) "Not owner"
    Would db2iupdt fix the tablespace container permissions?
    ---
    "It does not work" is not a valid problem statement.

  9. #9
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    No, it won't. But wanted to make sure no hanging / leftover processes or resources.

    lio_listio:
    Initiates a list of asynchronous I/O requests with a single call.

    OS was not provided, but if this is AIX, check:
    IBM DB2 9.7 for Linux, UNIX and Windows Information Center

    IOCP = Available

  10. #10
    Join Date
    Jul 2013
    Posts
    31
    The OS is AIX 5 im getting the following error for IOCP.

    ERROR MESSAGE

    Press Enter or Cancel to return to the
    application.

    1800-106 An error occurred:

    lsattr: 0514-519 The following device was not found in the customized
    device configuration database:
    iocp0


    I have stopped & updated the instance, but when i issue a start command it shows the database manager is already active.

    I also tried to connect to the database, but it still hangs indefinately.

  11. #11
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Check permissions and ownership on tablespace containers. File permission requirements for the instance and database directories
    ---
    "It does not work" is not a valid problem statement.

  12. #12
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    Ask AIX admin to resolve the following:

    lsattr: 0514-519 The following device was not found in the customized
    device configuration database:
    iocp0


    IOCP should be in Available state. When this is fixed, clean-up all DB2 processes/resouces, start db2 and try connecting.

  13. #13
    Join Date
    Jul 2013
    Posts
    31
    Issue resolved.. got the IOCP error fixed by the AIX admin. cleaned up the instance.. Restarted the server n voila... Its working now..

    Thank you db2girl & n_i.. Much appreciated

  14. #14
    Join Date
    Jul 2013
    Posts
    1
    Hii.. try searching with this error code ADM0503C and ur exact db2 version and ur OS and u will find how to fix it

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •