Sorry, don't know what happened to my window...
Technote:
IBM - Why are the instance owner ulimits ignored when the instance is started with the DB2 fault monitor?
The ulimits are inherited from the user ID that starts the instance. If the instance is started from the non-instance owner ID (ie. some admin or root), then the ulimits for that ID are in effect. Root's ulimit is applied when the instance is autostarted on reboot. The pstat command mentioned in the technote can be used to check the ulimits in effect (look at the "max data" value).
The technote mentions EFBIG (in case of "file" not being set to unlimited), You'll see ENOMEM if db2 is unable to allocate private memory (for "data" parameter). If pstat confirms that ulimit (data) is not unlimited, change root's ulimit to match the instance owner ID.