Environment:

DPF with 17 database partitions spanned across 3 physical nodes. 1 partition on Admin node, 8 partitions each on two data nodes.

Operating system: Linux 2.6.18-164.2.1.el5

Issue:

When trying to run an online database backup, the backup is failing. The backup is actually getting interrupted as it got failed on one of the partition with the below error. Highlighted in red.

Looks like it is something related to maxfilop parameter. Can some one please suggest how to adjust this parameter and how to verify how many files are actually being open and what is the operating system limit for that ??

0000 SQL2001N The utility was interrupted. The output data may be incomplete.
0001 SQL2001N The utility was interrupted. The output data may be incomplete.
0002 SQL2001N The utility was interrupted. The output data may be incomplete.
0003 SQL2001N The utility was interrupted. The output data may be incomplete.
0004 SQL2001N The utility was interrupted. The output data may be incomplete.
0005 SQL2001N The utility was interrupted. The output data may be incomplete.
0006 SQL0931C Operating system file table overflow occurred. Subsequent SQL statements cannot be processed.0007 SQL2001N The utility was interrupted. The output data may be incomplete.
0008 SQL2001N The utility was interrupted. The output data may be incomplete.
0009 SQL2001N The utility was interrupted. The output data may be incomplete.
0010 SQL2001N The utility was interrupted. The output data may be incomplete.
0011 SQL2001N The utility was interrupted. The output data may be incomplete.
0012 SQL2001N The utility was interrupted. The output data may be incomplete.
0013 SQL2001N The utility was interrupted. The output data may be incomplete.
0014 SQL2001N The utility was interrupted. The output data may be incomplete.
0015 SQL2001N The utility was interrupted. The output data may be incomplete.
0016 SQL2001N The utility was interrupted. The output data may be incomplete.

SQL2429N The database backup failed. The following database partitions
returned errors: "6".

Hope this explains

Well, that is not I was looking for. I would get that if I see what that error message is. Byt my question is how do we determine what is the linux operating system limit for file descriptors and currently how many file descriptors are being used ? If I get to know how to look at them. I can monitor the number of file descriptors being used while running my online backup and make sure they are not crossing the operating system limit.

On Linux, check file-max / ulimit -n. To check the number of file desrcriptors being used, I use lsof on AIX (example: lsof -p <PID of db2sysc> | wc -l). I believe the same command works on Linux.

Yes, Bella you are right. I checked the file-max in /etc/sysctl.conf. And I was also issuing cat /proc/sys/fs/file-nr to determine the usage. At the same time I also monitored lsof | wc -l and found that it was crossing the threshold . This issue started happening becuase we migrated our database to 9.7 from 9.1. As we now 9.7 is a thread based and maxfilop would be changed from 64 to 61440 by default. Now we are planning to increase file-max to avoid this issue.

Thanks for your help Bella.