I keep getting asked to 'go and find something' within the logs whenever our applications hangs/spikes/etc on the client side.
I know to check the db2diag.log and capture current snapshots, but upper mgmt still wants to know what happened on the transaction level. I tell them that you can't read the transaction logs, but they still persist on trying to figure out what happens.
I understand that I can set up event monitors, but they don't want to do that due to performance concerns. What else can I look at/do to provide the level of information they are looking for?
DIAGLEVEL = 3
NOTIFYLEVEL = 3
The diag log isn't showing the -911 errors that are showing up in the application logs along with other 'db2' errors.
Your question is too general. Tools that you use depend on the problem that you are trying to investigate.
SQL0911N could indicate either deadlocks or lock timeouts. For the former you use the deadlock event monitor, which has very little overhead and is created and enabled by default - all you need to do is to read the files. For lock timeouts there is additional monitoring that was introduced in DB2 9.5 - you can read about it here: Lock timeout reporting
You should explain to your bosses that there's no such thing as free lunch. If they want to monitor the system all the time, they will pay the price of performance impact all the time.