Results 1 to 10 of 10

Thread: Adm10500e

  1. #1
    Join Date
    Jul 2013
    Posts
    31

    Unanswered: Adm10500e

    One of my instance crashed yesterday, when i checked the logs it was due to disk full error. But there is more than sufficient space in the filesystem.
    Can anyone help?

    log:

    PID : 31192 TID : 1082308928 PROC : db2acd 0
    INSTANCE: db2inst1 NODE : 000
    FUNCTION: DB2 UDB, Health Monitor, HealthIndicator::update, probe:500
    MESSAGE : ADM10500E Health indicator "Database Automatic Storage Utilization"
    ("db.auto_storage_util") breached the "upper" alarm threshold of "90
    %" with value "91 %" on "database" "db2inst1.FDMSPROD". Calculation:
    "((db.auto_storage_used/db.auto_storage_total)*100);" =
    "((289731960832 / 317107200000) * 100)" = "91 %". History (Timestamp,
    Value, Formula): " (01/17/2015 21:37:09.787988, 89, ((283377537024 /
    317107200000) * 100)), (01/17/2015 21:27:08.415029, 88,
    ((278100533248 / 317107200000) * 100)), (01/17/2015 21:17:10.544324,
    86, ((272287518720 / 317107200000) * 100)), (01/17/2015
    21:07:10.565664, 84, ((267176316928 / 317107200000) * 100)),
    (01/17/2015 20:57:08.056273, 82, ((261378781184 / 317107200000) *
    100)), (01/17/2015 20:47:11.460515, 81, ((257104896000 /
    317107200000) * 100)), (01/17/2015 20:37:08.988340, 79,
    ((251965321216 / 317107200000) * 100)), (01/17/2015 20:27:08.521780,
    78, ((247224365056 / 317107200000) * 100)), (01/17/2015
    20:17:11.692383, 76, ((241007144960 / 317107200000) * 100))"

    2015-01-17-22.00.39.924798+330 E115130628E487 LEVEL: Info
    PID : 31170 TID : 47906258151744PROC : db2sysc 0
    INSTANCE: db2inst1 NODE : 000 DB : FDMSPROD
    APPHDL : 0-24202 APPID: *LOCAL.db2inst1.150213150702
    AUTHID : DB2INST1
    EDUID : 53242 EDUNAME: db2agent (FDMSPROD) 0
    FUNCTION: DB2 UDB, database utilities, sqluxGetDegreeParallelism, probe:762
    DATA #1 : <preformatted>
    Autonomic BAR - using parallelism = 1.

    2015-01-17-22.18.47.936710+330 I115131116E527 LEVEL: Info
    PID : 31170 TID : 47829519165760PROC : db2sysc 0
    INSTANCE: db2inst1 NODE : 000 DB : FDMSPROD
    APPHDL : 0-8 APPID: *LOCAL.DB2.150114061904
    AUTHID : DB2INST1
    EDUID : 58 EDUNAME: db2stmm (FDMSPROD) 0
    FUNCTION: DB2 UDB, buffer pool services, sqlbAlterBufferPoolAct, probe:90
    MESSAGE : Altering bufferpool "IBMDEFAULTBP" From: "17793021" <automatic> To:
    "17777341" <automatic>

    2015-01-17-22.35.11.365391+330 E115131644E1279 LEVEL: Error (OS)
    PID : 31170 TID : 47906258151744PROC : db2sysc 0
    INSTANCE: db2inst1 NODE : 000 DB : FDMSPROD
    APPHDL : 0-24202 APPID: *LOCAL.db2inst1.150213150702
    AUTHID : DB2INST1
    EDUID : 53242 EDUNAME: db2agent (FDMSPROD) 0
    FUNCTION: DB2 UDB, oper system services, sqloseekwrite64, probe:40
    MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
    DIA8312C Disk was full.
    CALLED : OS, -, pwrite OSERR: ENOSPC (28)
    DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
    File Handle = 22
    File System Block Size = 4096 bytes
    File System Type = ext3
    File Handle Flags :
    Require Sector Align = No
    DIO/CIO Mode = No
    Raw Block Device = No
    Reserved Handle = No
    Flush On Close = No
    Thread-Level Lock = No
    Write-through Mode = No
    File Not Tracked = Yes
    DATA #2 : unsigned integer, 8 bytes
    4096
    DATA #3 : signed integer, 8 bytes
    43374739456
    DATA #4 : signed integer, 8 bytes
    -1


    -1
    DATA #5 : String, 105 bytes
    Search for ossError*Analysis probe point after this log entry for further
    self-diagnosis of this problem.

    2015-01-17-22.35.11.407108+330 E115132924E1309 LEVEL: Error (OS)
    PID : 31170 TID : 47920220989760PROC : db2sysc 0
    INSTANCE: db2inst1 NODE : 000 DB : FDMSPROD
    APPHDL : 0-39726 APPID: 0000:0000:0000:0000:0000:FFFF:ac10:29c.40940.42149 5940455
    UTHID : DMSFORCE
    EDUID : 43972 EDUNAME: db2agent (FDMSPROD) 0
    FUNCTION: DB2 UDB, oper system services, sqloseekwrite64, probe:40
    MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
    DIA8312C Disk was full.
    CALLED : OS, -, pwrite OSERR: ENOSPC (28)
    DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
    File Handle = 104
    File System Block Size = 4096 bytes
    File System Type = ext3
    File Handle Flags :
    Require Sector Align = No
    DIO/CIO Mode = No
    Raw Block Device = No
    Reserved Handle = No
    Flush On Close = No
    Thread-Level Lock = No
    Write-through Mode = No
    File Not Tracked = Yes
    DATA #2 : unsigned integer, 8 bytes
    4096
    DATA #3 : signed integer, 8 bytes
    20272713728
    DATA #4 : signed integer, 8 bytes
    -1
    DATA #5 : String, 105 bytes
    Search for ossError*Analysis probe point after this log entry for further
    self-diagnosis of this problem.

  2. #2
    Join Date
    Apr 2012
    Posts
    1,034
    Provided Answers: 18
    If the db2diag shows "Disk full" (DIA8312C) and OS error 28 (ENOSPC) then I would trust that more than your statement to the contrary.

    Post your *evidence* that no file system of type ext3 was full at timestamp 2015-01-17-22.35.11

    You should also post your db2 version+fixpack and operatating system , full disclosure of relevant facts.
    Last edited by db2mor; 01-19-15 at 10:13.

  3. #3
    Join Date
    Jul 2013
    Posts
    31
    My filesystem has around 250GB of storage & only 31GB is in use. So I'm pretty sure it was not full. Even after the crash it wasn't full.
    I'm using DB2 9.7 FP 4 on Linux

  4. #4
    Join Date
    Apr 2012
    Posts
    1,034
    Provided Answers: 18
    Evidence? Output of df -h ? What help are you expecting if you post no evidence?

  5. #5
    Join Date
    Jul 2013
    Posts
    31
    [db2inst1@host4 ~]$ df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/md1 39G 7.7G 30G 21% /
    /dev/md6 15G 193M 14G 2% /tmp
    /dev/md5 25G 2.4G 21G 11% /opt
    /dev/md4 25G 18G 5.8G 75% /var
    /dev/md3 25G 4.8G 19G 21% /usr
    /dev/md0 494M 22M 447M 5% /boot
    tmpfs 44G 4.0K 44G 1% /dev/shm
    /dev/mapper/mpath2 591G 2.0G 559G 1% /dbdata
    /dev/mapper/mpath3 296G 191M 281G 1% /dbext1
    /dev/mapper/mpath4 296G 3.6G 277G 2% /dbext2
    /dev/mapper/mpath6 296G 520M 280G 1% /dblog
    /dev/mapper/mpath5 296G 31G 250G 11% /dbsys

  6. #6
    Join Date
    Apr 2012
    Posts
    1,034
    Provided Answers: 18
    Check system message log (for the timestamp of the symptom) to see what messages got logged.

    Also check db2 notification log (unless you configured smtp notifications) to see if it gives the path for which linux reported the error.

    You should also check with your storage/admin folks (for whatever device provides /dev/mapper/mpath*) to see if there were any blips around the time of the symptom.

    This is most likely not a DB2 issue - if Linux reports OSERR ENOSPC(28) to DB2 then DB2 will react...the question is what is causing Linux to report ENOSPC...

  7. #7
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    You'll notice that the health monitor reported 91% utilization of the database storage at some point, and then an automatic backup started some minutes before the database ran out of space. One explanation for "disappearing" disk utilization might be the use of temporary tablespaces during statement execution -- temporary space will be released when the transaction terminates (by the instance crash, for example). You need real time monitoring of space utilization to determine the actual cause.
    ---
    "It does not work" is not a valid problem statement.

  8. #8
    Join Date
    Jul 2013
    Posts
    31
    You're right n_i,
    One of my temp tablespaces got filled. leading to the instance crash. Now the question is how to rectify this?? is there a way I can restrict the growth? what parameters should I tweak? Any insights would be much appreciated

  9. #9
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    You can either tune the offending SQL statement to avoid using that much temporary space or provide sufficient filesystem space (possibly on a filesystem separate from the database home, logs, and other tablespaces -- to avoid the risk of crashing the instance again if you still run out of space).
    ---
    "It does not work" is not a valid problem statement.

  10. #10
    Join Date
    Jul 2013
    Location
    Moscow, Russia
    Posts
    666
    Provided Answers: 55
    Hi,

    Try to set DB2_OPT_MAX_TEMP_SIZE variable or recreate your system temporary tablespace as DMS with some MAXSIZE.
    Regards,
    Mark.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •