I am getting following error when trying to run db2 connect to <dbname> command.

SQL1084C Shared memory segment cannot be allocated. SQLSTATE = 57019

1. I have increased the size of the kernel.shmmax value.
2. I have set the intance_memory database configuration setting to AUTOMATIC.

but no success. Please help.

Thank you.

V9.5? Try this:
This will limit your RAM consumption to 1,000,000 pages (4G).
If the database starts normally, you can set Instance_Memory to automatic as well. You must stop/start the instance
To reset your bufferpools, you can After start/stop all the bufferpools come up with 4096 pages. STMM will pick them up and bring them to the ideal values in a day or 2.