Stats:
"DB2 v8.1.1.128", "s061108", "U810098", and FixPak "14"
"DB2 Enterprise Server Edition"
AIX 5300-11-03-1013

Problem:
During normal processing, applications began failing as the db2diag.log reported errors that the disk is full. See below for the exact error messages.

All tablespaces are SMS and there is 9.73 GB or 46% free space on the directory, so there should be no issue there.

It's odd to note that the SQL00003.DAT file in tablespace directory is .9999999 GB (strange that it seems to fail at the 1GB threshold)


Additional problem with similar issue:


When performing a backup, the backup must be spread over 3 files. It seems that once a backup file becomes great than 1GB it must be broken up.

$ db2 backup db cmdlilsp online to '/db2backup/CMDLILSP/daily' compress include logs without prompting
SQL2419N The target disk "/db2backup/CMDLILSP/daily" has become full.

but this comand works fine....
db2 backup db cmdlilsp online to '/db2backup/CMDLILSP/daily','/db2backup/CMDLILSP/daily','/db2backup/CMDLILSP/daily' compress include logs without prompting;


Things I've checked:
I've check the db2set and env paramaters as well as the dbm and db cfg's, nothing that I've found would make me think there is a 1GB limitation.

I've had the AIX admin check to make sure there is no limitation for the db2 instance owner or any other user.


Table space growth errors:


2011-08-18-08.44.45.010664-240 I26681339C760 LEVEL: Error
PID : 36808 TID : 1 PROC : db2agent (CMDLILSP) 0
INSTANCE: cmililsp NODE : 000 DB : CMDLILSP
APPHDL : 0-1201 APPID: GA651871.IB06.103FC8105522
FUNCTION: DB2 UDB, oper system services, sqloseekwrite64, probe:100
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
0x2FF19108 : 0000 0009 0000 0000 ........
DATA #2 : unsigned integer, 4 bytes
32768
DATA #3 : signed integer, 8 bytes
1073709056
DATA #4 : unsigned integer, 4 bytes
0
DATA #5 : Hex integer, 4 bytes
0x00000000
DATA #6 : signed integer, 4 bytes
32256

2011-08-18-08.44.45.205610-240 E26682100C777 LEVEL: Error
PID : 36808 TID : 1 PROC : db2agent (CMDLILSP) 0
INSTANCE: cmililsp NODE : 000 DB : CMDLILSP
APPHDL : 0-1201 APPID: GA651871.IB06.103FC8105522
FUNCTION: DB2 UDB, buffer pool services, sqlbWritePageToDisk, probe:20
MESSAGE : ADM6017E The table space "ICMLNF32" (ID "4") is full. Detected on
container "/cmdata/cmililsp/NODE0000/SQL00001/ICMLNF32" (ID "0").
The underlying file system is full or the maximum allowed space usage
for the file system has been reached. It is also possible that there
are user limits in place with respect to maximum file size and these
limits have been reached.e



Backup errors:


2011-08-22-15.46.16.005807-240 I34457679C385 LEVEL: Warning
PID : 30446 TID : 1 PROC : db2agent (idle) 0
INSTANCE: cmililsp NODE : 000 DB : CMDLILSP
APPHDL : 0-1106 APPID: *LOCAL.cmililsp.110822194615
FUNCTION: DB2 UDB, database utilities, sqlubSetupJobControl, probe:2028
MESSAGE : Starting an online db backup.

2011-08-22-15.48.16.113401-240 I34458065C573 LEVEL: Error
PID : 41002 TID : 1 PROC : db2med.30446.0 0
INSTANCE: cmililsp NODE : 000
FUNCTION: DB2 UDB, oper system services, sqlowrite, probe:200
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
0x2FF21098 : 0000 0003 0000 0000 ........
DATA #2 : unsigned integer, 4 bytes
1044480
DATA #3 : signed integer, 4 bytes
11776
DATA #4 : signed integer, 4 bytes
0

2011-08-22-15.48.16.157595-240 I34458639C330 LEVEL: Warning
PID : 41002 TID : 1 PROC : db2med.30446.0 0
INSTANCE: cmililsp NODE : 000
FUNCTION: DB2 UDB, database utilities, sqluMCWriteToDevice, probe:60
MESSAGE : Media controller -- Disk full encountered on /db2backup/CMDLILSP/dail

Check user limits for your instance owner id ( ulimit -a ). Look for:

file(blocks)

$ whoami
cmililsp
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 245760
stack(kbytes) 16384
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 2000


I'm not an AIX wizard, so I don't know if any of these look too low.

As the user cmililsp, try to create a file larger than 1 GB in size in one of the directories where DB2 encounters an error. I would expect it to fail; if that's the case then there IS a limit imposed by the operating system.

Try also running a "quota" command (without any parameters) - it will show if there's a disk quota set for the login ID.

I successfully copied a > 1 GB file from another server to the backup directory while signed on as the db2 intance owner.

$ quota
Disk quotas for user cmililsp (uid 207): none


I'm thinking the OS is innocent, could DB2 be responsible in any way for the limitaiton? I'm a still a new DBA, so I'm not sure I've checked everywhere I should have.

With this version of DB2 being out of support, I'm not even able to open a pmr with IBM for assistance. Any other thoughts? I really appreciate the help so far. If nothing else, I've learned some new unix commands

Check my last update here:
sql2310n when performing runstats on a large table


Use the information in the technote to check the ulimits in effect (pstat command). I suspect it will show 1GB. Or if you can, restart the instance using your instance owner id and then retry the backup command (with one target dir).

db2girl, you rock! Thank you so much for pointing me to this post. It was the source of my problem and now I have a fix. THANK YOU!