Page 1 of 2 12 LastLast
Results 1 to 15 of 18
  1. #1
    Join Date
    Oct 2004
    Posts
    268

    Unanswered: HADR Database Data Import, Backup Error

    DB2 9.5 FP8 - RHEL 6.2

    We have a HADR setup and I am loading the primary database with data from a file. The exact command is:

    db2 "IMPORT FROM /home/db2inst1/production/output/daily.values OF DEL MODIFIED BY noeofchar coldel| chardel~~ COMMITCOUNT 1000 MESSAGES db2import.log INSERT INTO Express.datastagingtbl" >> /home/db2inst1/production/DAILYPD1_LOAD.out;

    I am not sure this command considered to be a load, import or Insert. Do I need to use LOAD COPY YES option for this load to be replicated to the standby server ?

    I am doing an online backup before the load and nightly backup @11:00 PM. The nightly backup fails with below message. Do you think it is related to this load ?

    2012-09-12-23.16.43.678570-240 E178431E954 LEVEL: Severe
    PID : 11780 TID : 140062226376448PROC : db2sysc
    INSTANCE: db2inst1 NODE : 000 DB : DAILYPD1
    APPHDL : 0-57128 APPID: *LOCAL.db2inst1.120913031622
    AUTHID : DB2INST1
    EDUID : 13154 EDUNAME: db2agent (DAILYPD1)
    FUNCTION: DB2 UDB, database utilities, sqlubMWResponse, probe:933
    MESSAGE : SQL2062N An error occurred while accessing media "". Reason code:
    "".
    DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
    sqlcaid : SQLCA sqlcabc: 136 sqlcode: -2062 sqlerrml: 35
    sqlerrmc: /db2/backup/FULL/DAILYPD1/ 0x860f0003
    sqlerrp : sqlubMWR
    sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
    (4) 0x00000000 (5) 0x00000000 (6) 0x00000000
    sqlwarn : (1) (2) (3) (4) (5) (6)
    (7) (8) (9) (10) (11)
    sqlstate:

    2012-09-12-23.16.43.679018-240 E179386E532 LEVEL: Error
    PID : 11780 TID : 140062226376448PROC : db2sysc
    INSTANCE: db2inst1 NODE : 000 DB : DAILYPD1
    APPHDL : 0-57128 APPID: *LOCAL.db2inst1.120913031622
    AUTHID : DB2INST1
    EDUID : 13154 EDUNAME: db2agent (DAILYPD1)
    FUNCTION: DB2 UDB, database utilities, sqlubMWResponse, probe:937
    MESSAGE : SQL2062N An error occurred while accessing media
    "/db2/backup/FULL/DAILYPD1/". Reason code: "0x860f0003".

    2012-09-12-23.16.43.895621-240 E179919E428 LEVEL: Severe
    PID : 11780 TID : 140062226376448PROC : db2sysc
    INSTANCE: db2inst1 NODE : 000 DB : DAILYPD1
    APPHDL : 0-57128 APPID: *LOCAL.db2inst1.120913031622
    AUTHID : DB2INST1
    EDUID : 13154 EDUNAME: db2agent (DAILYPD1)
    FUNCTION: DB2 UDB, database utilities, sqlubcka, probe:865
    MESSAGE : Backup terminated.


    Thanks.

  2. #2
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    The IMPORT command is considered to be import, most probably. Since it is a logged operation, it should not affect HADR.

    The backup error is not related to a load; as can be seen from the db2diag.log messages, it is related to the disk error on /db2/backup/FULL/DAILYPD1/
    ---
    "It does not work" is not a valid problem statement.

  3. #3
    Join Date
    Oct 2004
    Posts
    268
    Thank you for the quick response. According to our network admin, it is not a network connection/LUN issue. He says he setup a heart beat around the time backup was going on, there was no disconnect with SAMBA share.

    I have scheduled the backup via task manager. Should I use cron job to backup the database ? Any other areas I should look at ?

    Thanks again.

  4. #4
    Join Date
    Apr 2012
    Posts
    1,034
    Provided Answers: 18
    Is there some problem with the file-system that holds /db2/backup/FULL/DAILYPD1 ?
    Is it local or SAN?
    Some mounting problem?
    Some access problem? Running out of space?
    Look in the system message log /var/log/messages (at the timestamp of the message in db2diag.log) to see if some fault is happening with the file-system that holds /db2/backup/FULL/DAILYPD1

  5. #5
    Join Date
    Oct 2004
    Posts
    268
    Quote Originally Posted by db2mor View Post
    Is there some problem with the file-system that holds /db2/backup/FULL/DAILYPD1 ?
    Is it local or SAN?
    Some mounting problem?
    Some access problem? Running out of space?
    Look in the system message log /var/log/messages (at the timestamp of the message in db2diag.log) to see if some fault is happening with the file-system that holds /db2/backup/FULL/DAILYPD1

    No mounting problem according to Network people. No access problem because the second database backup is successful to the same LUN. No space issues and no apperant error/warning message in messages file. I have already posted the error message during the time of the backup.

    There a 2 LUNs, one where the database files resides and the other where the backup files. Failure does not happen every night but like a night or two a week.

    Thanks.

  6. #6
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Quote Originally Posted by mdx34 View Post
    No mounting problem according to Network people. No access problem because the second database backup is successful to the same LUN. No space issues...
    So far the only proof you have is of the opposite. You are free to continue chasing ghosts, of course.
    ---
    "It does not work" is not a valid problem statement.

  7. #7
    Join Date
    Apr 2012
    Posts
    1,034
    Provided Answers: 18
    If DB2 gets an error writing to /db2/backup/FULL/DAILYPD1 on one or two nights a week, you ought to be able to find more details from *some* log file.

    It's best to focus on getting the sysadmin to investigate further, perhaps by trying some diagnostics during the backup.

    What is the file system format type? (ext3, ext4 , other?) that contains /db2/backup/FULL/DAILYPD1

    Is there a pattern (approx timestamp of the error, day of week, size of the backup file(s) at the time of failure, etc , etc )?

    Is the db2diag.log entry identical each time the backup fails? (same -2062 code & rc )

    You might get some more detail of the context if you set DIAGLEVEL to 4 temporarily during backup (returning to diaglevel 3 at end of backup)

    Can you post the full command-line being used to backup the database?

    Is there any possibility that some other tool (not DB2) is performing a file-system based activity (e.g filesystem backup) at the same moment that db2 backup is running ?


    Is the hostname (for RHEL 6.x) physical or virtual?
    Last edited by db2mor; 09-13-12 at 15:17.

  8. #8
    Join Date
    Oct 2004
    Posts
    268
    Quote Originally Posted by n_i View Post
    So far the only proof you have is of the opposite. You are free to continue chasing ghosts, of course.
    I don't have time for your sarcasm. You can keep your knowledge to yourself.

  9. #9
    Join Date
    Oct 2004
    Posts
    268
    Quote Originally Posted by db2mor View Post
    If DB2 gets an error writing to /db2/backup/FULL/DAILYPD1 on one or two nights a week, you ought to be able to find more details from *some* log file.

    It's best to focus on getting the sysadmin to investigate further, perhaps by trying some diagnostics during the backup.

    What is the file system format type? (ext3, ext4 , other?) that contains /db2/backup/FULL/DAILYPD1

    Is there a pattern (approx timestamp of the error, day of week, size of the backup file(s) at the time of failure, etc , etc )?

    Is the db2diag.log entry identical each time the backup fails? (same -2062 code & rc )

    You might get some more detail of the context if you set DIAGLEVEL to 4 temporarily during backup (returning to diaglevel 3 at end of backup)

    Can you post the full command-line being used to backup the database?

    Is there any possibility that some other tool (not DB2) is performing a file-system based activity (e.g filesystem backup) at the same moment that db2 backup is running ?


    Is the hostname (for RHEL 6.x) physical or virtual?

    Thank you.

    The error message is always the same. The server are physical servers. The DIAGLEVEL is 3 and I may need to change that to 4. Database size is only a half of Gig. There is no other activity on the server while backup is running.

    Backup command is:

    BACKUP DATABASE DAILYPD1 ONLINE TO "/db2/backup/FULL/DAILYPD1/" WITH 2 BUFFERS BUFFER 1024 PARALLELISM 1 COMPRESS UTIL_IMPACT_PRIORITY 30 WITHOUT PROMPTING;

    db2diag.log file attached.

    Thanks again.
    Last edited by mdx34; 09-13-12 at 15:43.

  10. #10
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Quote Originally Posted by mdx34 View Post
    I don't have time for your sarcasm. You can keep your knowledge to yourself.
    I think you're being irrational. You trust unsubstantiated claims by third parties ("network people") that there is no problem on _your_ server, yet you do not believe the fact that DB2 is unable to write to the backup destination path (which is evident from the DB2 log).

    You will have to find why DB2 cannot write to that path, until then no amount of knowledge, shared or not, can help you.
    ---
    "It does not work" is not a valid problem statement.

  11. #11
    Join Date
    Apr 2012
    Posts
    1,034
    Provided Answers: 18
    This may not be a DB2 problem (unless you find relevant APARs fixed in fixpacks 9 and 10)

    Concentrate on getting diagnostics from the filesystem during backup - involve the sysadmin for this

  12. #12
    Join Date
    Oct 2004
    Posts
    268
    Quote Originally Posted by n_i View Post
    I think you're being irrational. You trust unsubstantiated claims by third parties ("network people") that there is no problem on _your_ server, yet you do not believe the fact that DB2 is unable to write to the backup destination path (which is evident from the DB2 log).

    You will have to find why DB2 cannot write to that path, until then no amount of knowledge, shared or not, can help you.

    I don't trust anyone. They showed me the trace they ran for the heart beat and nothing in the messages file. I also think that the problem is on their side but I have to rule out the DB2 portion. I am using the cluster server they setup for different purpose as my Quorum device and that may be having issues.

    Also, why one database backup succedes and the other not puzzles me. They are pointing to the same directory until /db2/backups/FULL/

    the rest is /db2/backups/FULL/DATABASE1
    /db2/backups/FULL/DATABASE2

  13. #13
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    I don't know what "heartbeat" means in your case. You can add your own "heartbeat" by trying to write to the directory in question immediately before and immediately after you invoke the backup (touch would work just fine).

    You haven't said it explicitly, but since "network people" are somehow involved, I'm assuming the backup destination path is a network mount (NFS?). In that case you have at least four layers of complexity, and each can contribute to the problem:
    - TCP
    - NFS server and client
    - remote file system
    - authentication on the remote server

    Care to post the output of "mount | grep backups"?
    ---
    "It does not work" is not a valid problem statement.

  14. #14
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    $ db2diag -rc 0x860f0003

    Input ZRC string '0x860f0003' parsed as 0x860F0003 (-2045837309).

    ZRC value to map: 0x860F0003 (-2045837309)
    V7 Equivalent ZRC value: 0xFFFFE603 (-6653)

    ZRC class :
    Critical Media Error (Class Index: 6)
    Component:
    SQLO ; oper system services (Component Index: 15)
    Reason Code:
    3 (0x0003)

    Identifer:
    SQLO_DERR
    Identifer (without component):
    SQLZ_RC_DERR

    Description:
    disk error occurred (DOS)

    Associated information:
    Sqlcode -980
    SQL0980C A disk error occurred. Subsequent SQL statements cannot be
    processed.

    Number of sqlca tokens : 0
    Diaglog message number: 8402




    Are there any errors logged in the db2diag.log prior to the 1st one you posted? I would expect db2 to log an error it received from the OS (ie. EIO - disk error) and OS call made by db2.

  15. #15
    Join Date
    Oct 2004
    Posts
    268
    Quote Originally Posted by db2girl View Post
    $ db2diag -rc 0x860f0003

    Input ZRC string '0x860f0003' parsed as 0x860F0003 (-2045837309).

    ZRC value to map: 0x860F0003 (-2045837309)
    V7 Equivalent ZRC value: 0xFFFFE603 (-6653)

    ZRC class :
    Critical Media Error (Class Index: 6)
    Component:
    SQLO ; oper system services (Component Index: 15)
    Reason Code:
    3 (0x0003)

    Identifer:
    SQLO_DERR
    Identifer (without component):
    SQLZ_RC_DERR

    Description:
    disk error occurred (DOS)

    Associated information:
    Sqlcode -980
    SQL0980C A disk error occurred. Subsequent SQL statements cannot be
    processed.

    Number of sqlca tokens : 0
    Diaglog message number: 8402




    Are there any errors logged in the db2diag.log prior to the 1st one you posted? I would expect db2 to log an error it received from the OS (ie. EIO - disk error) and OS call made by db2.
    I tought I posted a file but in case. this is what I have form beginning of the error till the end.
    Attached Files Attached Files

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •