Hello, All

I have this intermittent problem in my web application. An exception is thrown by JDBC:
COM.ibm.db2.jdbc.DB2Exception: [IBM][CLI Driver] SQL1224N A database agent could not be started to service a request, or was terminated as a result of a database system shutdown or a force command. SQLSTATEU032
at COM.ibm.db2.jdbc.app.SQLExceptionGenerator.throw_S QLException(Unknown Source)
at COM.ibm.db2.jdbc.app.SQLExceptionGenerator.throw_S QLException(Unknown Source)
at COM.ibm.db2.jdbc.app.SQLExceptionGenerator.check_r eturn_code(Unknown Source)
at COM.ibm.db2.jdbc.app.DB2Statement.execute2(Unknown Source)
at COM.ibm.db2.jdbc.app.DB2Statement.executeQuery(Unk nown Source)
at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.ex ecuteQuery(DelegatingStatement.java:205)

My environment is:

OS: Windows 2003 server
Web server: Apache 2.0
App server: Tomcat 5.5
Web application: J2EE
DB server: DB2 Express 8.1 with fixpack
JDBC driver: db2java.jar

Connection is made using DataSource with connection pooling.
The problem occurs after a full backup is run (both offline and online) and is cured by restarting the application server (Tomcat).
db2diag.log does not show anything suspicious (at least to me):
A full off-line backup was done on 2007-12-09-01.00.03
and an incremental online backup on 2007-12-10-01.00.03
2007-12-09-01.00.03.917000-300 E1856161H390

LEVEL: Error
PID: 4688
TID : 2836

PROC : db2syscs.exe
INSTANCE: DB2
NODE : 000
DB: SYMAGI
APPHDL : 0-291

APPID: *LOCAL.DB2.071209060021
FUNCTION: DB2 UDB, base sys utilities, sqleDatabaseQuiesce, probe:1
MESSAGE : ADM7506W Database quiesce has been requested.
2007-12-09-01.00.03.948000-300 E1856553H406

LEVEL: Error
PID: 4688

TID : 2836

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-291

APPID: *LOCAL.DB2.071209060021
FUNCTION: DB2 UDB, base sys utilities, sqleDatabaseQuiesce, probe:2
MESSAGE : ADM7507W Database quiesce request has completed successfully.
2007-12-09-01.00.04.870000-300 I1856961H334

LEVEL: Warning
PID: 4688

TID : 5648

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpghck, probe:1390
MESSAGE : ExtNum 900, state 401, baselsn 00000000DCF28000 nextlsn

00000000DCF3470F
2007-12-09-01.00.04.964000-300 I1857297H417

LEVEL: Warning
PID: 4688

TID : 5352

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgArchiveLogFile, probe:3180
MESSAGE : Successfully archived log file S0000900.LOG to

D:\DB2ARCHLOGS\DB2\SYMAGI\NODE0000\C0000000\ from

D:\DB2\NODE0000\SQL00001\SQLOGDIR\.
2007-12-09-01.00.05.230000-300 I1857716H320

LEVEL: Warning
PID: 4688

TID : 5744

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgLoggrInitDelOldLog, probe:1440
MESSAGE : Cleaning up logs from RenameArchNum 900 to delLimit 901
2007-12-09-01.00.05.292000-300 I1858038H379

LEVEL: Warning
PID: 4688

TID : 5660

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-296

APPID: *LOCAL.DB2.071209060023
FUNCTION: DB2 UDB, database utilities, sqlubSetupJobControl, probe:2025
MESSAGE : Starting an offline db backup.
2007-12-09-01.00.40.542000-300 I1858419H353

LEVEL: Warning
PID: 4688

TID : 5660

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-296

APPID: *LOCAL.DB2.071209060023
FUNCTION: DB2 UDB, database utilities, sqlubcka, probe:130
MESSAGE : Backup Complete.
2007-12-09-01.00.40.855000-300 E1858774H395

LEVEL: Error
PID: 4688

TID : 4728

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-297

APPID: *LOCAL.DB2.071209060040
FUNCTION: DB2 UDB, base sys utilities, sqleDatabaseUnquiesce, probe:1
MESSAGE : ADM7510W Database unquiesce has been requested.
2007-12-09-01.00.41.026000-300 E1859171H410

LEVEL: Error
PID: 4688

TID : 4728

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-297

APPID: *LOCAL.DB2.071209060040
FUNCTION: DB2 UDB, base sys utilities, sqleDatabaseUnquiesce, probe:2
MESSAGE : ADM7509W Database unquiesce request has completed successfully.
2007-12-09-01.00.43.808000-300 I1859583H334

LEVEL: Warning
PID: 4688

TID : 6044

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpghck, probe:1390
MESSAGE : ExtNum 901, state 401, baselsn 00000000DD310000 nextlsn

00000000DD311513
2007-12-09-01.00.43.917000-300 I1859919H417

LEVEL: Warning
PID: 4688

TID : 2644

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgArchiveLogFile, probe:3180
MESSAGE : Successfully archived log file S0000901.LOG to

D:\DB2ARCHLOGS\DB2\SYMAGI\NODE0000\C0000000\ from

D:\DB2\NODE0000\SQL00001\SQLOGDIR\.
2007-12-09-01.00.44.120000-300 I1860338H320

LEVEL: Warning
PID: 4688

TID : 2208

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgLoggrInitDelOldLog, probe:1440
MESSAGE : Cleaning up logs from RenameArchNum 901 to delLimit 902
2007-12-09-01.00.45.542000-300 I1860660H334

LEVEL: Warning
PID: 4688

TID : 2820

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpghck, probe:1390
MESSAGE : ExtNum 902, state 401, baselsn 00000000DD6F8000 nextlsn

00000000DD6F8544
2007-12-09-01.00.45.605000-300 I1860996H417

LEVEL: Warning
PID: 4688

TID : 2700

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgArchiveLogFile, probe:3180
MESSAGE : Successfully archived log file S0000902.LOG to

D:\DB2ARCHLOGS\DB2\SYMAGI\NODE0000\C0000000\ from

D:\DB2\NODE0000\SQL00001\SQLOGDIR\.
2007-12-09-20.44.33.234000-300 I1861415H320

LEVEL: Warning
PID: 4688

TID : 2552

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgLoggrInitDelOldLog, probe:1440
MESSAGE : Cleaning up logs from RenameArchNum 902 to delLimit 903
2007-12-10-01.00.04.362000-300 I1861737H390

LEVEL: Warning
PID: 4688

TID : 3536

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-68

APPID: *LOCAL.DB2.071210060021
FUNCTION: DB2 UDB, database utilities, sqlubSetupJobControl, probe:2025
MESSAGE : Starting an online incremental db backup.
2007-12-10-01.00.14.797000-300 I1862129H417

LEVEL: Warning
PID: 4688

TID : 3564

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000
FUNCTION: DB2 UDB, data protection, sqlpgArchiveLogFile, probe:3180
MESSAGE : Successfully archived log file S0000903.LOG to

D:\DB2ARCHLOGS\DB2\SYMAGI\NODE0000\C0000000\ from

D:\DB2\NODE0000\SQL00001\SQLOGDIR\.
2007-12-10-01.00.24.918000-300 I1862548H353

LEVEL: Warning
PID: 4688

TID : 3536

PROC : db2syscs.exe
INSTANCE: DB2

NODE : 000

DB: SYMAGI
APPHDL : 0-68

APPID: *LOCAL.DB2.071210060021
FUNCTION: DB2 UDB, database utilities, sqlubcka, probe:130
MESSAGE : Backup Complete.



Any help would be much appreciated

the db was doing an offline backup and in quiesce state
no connection are allowed
if not the purpose - take online backup

Thanks for your comment, Przytula.
This was my 1st attempt - to try online backup.
It helped in my test environment - the exception did not happen upon login to web.app after the backup.
However on the production system it did not help.
This is what I did on prod. system:
1) run full online backup
2) after several minutes run incremental backup (to have set of full + incr. + logs for restore - I actually tried to restore to another DB later to make sure it's functional)
3) tried to login to web.app - and got the exception.
4) tried to login to another web.app working on another DB on the same system on wich the backup was NOT done - and got the exception.
5) restarted the app.server (Tomcat)
6) didn't do anything with the DB or DB manager
7) tried to login to web.app - no exception, everything works

So it looks like the problem lies not in the DB or the DB manager but in the communication between the DB2 JDBC driver and appropriate DB manager's agents.

Couple of ideas:

1. A broken (e.g. forced) JDBC connection does not report an error until an attempt is made to use it. In other words, the time when an exception appears in the Tomcat log does not necessarily correspond to the actual time of the connection being closed. If you quiesce (and unquiesce) the database tonight and use the application tomorrow, errors will be reported tomorrow. Make sure you conduct a clean experiment.

2. This error can also appear if your AS connection pool size exceeds MAXAPPLS or MAXCONNECTIONS settings in DB2.

The problem seems to be solved. I replaced the offline full backup with online. I waited the full 1 week cycle (full online backup Sunday night plus incremental online every weekday night). The wep.app. continued working throughout the whole time and no exceptions occured.

Still unclear why a new connection can't be started by the JDBC driver but those IBM drivers have enough other quirks to expect them to be perfect.

Thanks to everybody who replied.