Hi,

I was running a job last night, and this morning Informix 10 was hung; a systems guy tried to bounce it, but it failed. The result of the oninit cmd follows. Below that are lines from the .log file...

some nerve-wracking messages:
08:56:58 I/O error, Primary Chunk '/xxx/rootdbs' -- Offline (sanity)
Cannot open chunk '/logs/engine/rootdbs'. errno = 2

Also, I noticed:
08:56:57 Results: Chunk 1 is being taken OFFLINE.
08:56:57 Action: Restore chunk from archive.
---? Does this mean simply restore the rootdbs OS file from tape (which we may or may not have, I'll chk w/ the systems guys)?

Caveats:
The rootdbs file DOES exist right where onconfig says it does.
The server has a lot of databases, and I don't want to have to reinstall them.
I read the other messages here on the topic but they didn't relate / help.
Thanks,
MichaelG



[informix@itest ~$
[informix@itest ~$ oninit -vy
Checking group membership to determine server run mode...succeeded
Reading configuration file '/opt/informix/etc/onconfig'...succeeded
Creating /INFORMIXTMP/.infxdirs...succeeded
Creating infos file "/opt/informix/etc/.infos.itest"...succeeded
Linking conf file "/opt/informix/etc/.conf.itest"...succeeded
Writing to infos file...succeeded
Checking config parameters...succeeded
Allocating and attaching to shared memory...succeeded
Creating resident pool 14560 kbytes...succeeded
Allocating 10016 kbytes for buffer pool of 2K page size...succeeded
Initializing rhead structure...succeeded
Initializing ASF...succeeded
Initializing Dictionary Cache and SPL Routine Cache...succeeded
Bringing up ADM VP...succeeded
Creating VP classes...succeeded
Onlining 1 additional cpu vps...succeeded
Onlining 12 IO vps...succeeded
Initialization of Encryption...succeeded
Forking main_loop thread...succeeded
Initializing DR structures...succeeded
Forking 1 'soctcp' listener threads...succeeded
Forking 1 'ipcshm' listener threads...succeeded
Starting tracing...succeeded
Initializing 40 flushers...succeeded
sh: /usr/informix/etc/alarmprogram.sh: No such file or directory
oninit: Cannot open chunk '/logs/engine/rootdbs'. errno = 2
oninit: Fatal error in shared memory initialization
[informix@itest ~$



THE LOG FILE from last night and this morning:
...
...

16:58:31 Maximum server connections 3
17:00:52 Checkpoint Completed: duration was 1 seconds.
17:00:52 Checkpoint loguniq 135, logpos 0x71af018, timestamp: 0x6caec6e9

17:00:52 Maximum server connections 3
17:03:00 Checkpoint Completed: duration was 0 seconds.
17:03:00 Checkpoint loguniq 135, logpos 0x8db3018, timestamp: 0x6ccfaf30

17:03:00 Maximum server connections 3
17:03:02 Checkpoint Completed: duration was 0 seconds.
17:03:02 Checkpoint loguniq 135, logpos 0x8db6050, timestamp: 0x6ccfaf66

17:03:02 Maximum server connections 3
17:03:02 'taom' - New logging mode: UNBUFFERED
17:04:02 Fuzzy Checkpoint Completed: duration was 2 seconds, 7 buffers not flushed.
17:04:02 Checkpoint loguniq 135, logpos 0x95010e8, timestamp: 0x6ce12285

17:04:02 Maximum server connections 3

Thu Feb 8 07:49:57 2007

07:49:57 stack trace for pid 19375 written to /logs/af.61293894
07:49:57 Assert Failed: No Exception Handler
07:49:57 IBM Informix Dynamic Server Version 10.00.F
07:49:57 Who: Session(210, develope@serv8.nmcourts.com, -1, 0x45e0dba8)
Thread(23873, sqlexec, 45de0a18, 1)

File: mtex.c Line: 472
07:49:57 Results: Exception Caught. Type: MT_EX_OS, Context: mem
07:49:57 Action: Please notify IBM Informix Technical Support.
07:49:57 stack trace for pid 19375 written to /logs/af.61293894
07:49:57 See Also: /logs/af.61293894, shmem.61293894.0
07:50:00 mtex.c, line 472, thread 23873, proc id 19375, No Exception Handler.
07:50:00 invoke_alarm(): /bin/sh -c '/usr/informix/etc/alarmprogram.sh 5 6 "Internal Subsystem failure: 'MT'" "mtex.c, line 472, thread 23873
, proc id 19375, No Exception Handler." '
07:50:00 invoke_alarm(): mt_exec failed, status 32512, errno 0
07:50:00 The Master Daemon Died
07:50:00 invoke_alarm(): /bin/sh -c '/usr/informix/etc/alarmprogram.sh 5 6 "Internal Subsystem failure: 'MT'" "The Master Daemon Died" '
07:50:00 invoke_alarm(): mt_exec failed, status 32512, errno 0
07:50:00 PANIC: Attempting to bring system down
08:11:44 IBM Informix Dynamic Server Started.

Thu Feb 8 08:11:45 2007

08:11:45 Warning: ONCONFIG dump directory (DUMPDIR) '/logs' has insecure permissions
08:11:45 Event alarms enabled. ALARMPROG = '/usr/informix/etc/alarmprogram.sh'
08:11:45 Dynamically allocated new virtual shared memory segment (size 8316KB)
08:11:45 Booting Language <c> from module <>
08:11:45 Loading Module <CNULL>
08:11:45 Booting Language <builtin> from module <>
08:11:45 Loading Module <BUILTINNULL>
08:11:49 Dynamically allocated new virtual shared memory segment (size 8192KB)
08:11:51 DR: DRAUTO is 0 (Off)

08:11:51 Dynamically allocated new message shared memory segment (size 124KB)
08:11:52 IBM Informix Dynamic Server Version 10.00.FC5 Software Serial Number AAA#B000000
08:11:53 IBM Informix Dynamic Server Stopped.

08:11:53 mt_shm_remove: WARNING: may not have removed all/correct segments
08:54:01 IBM Informix Dynamic Server Started.

Thu Feb 8 08:54:01 2007

08:54:01 Warning: ONCONFIG dump directory (DUMPDIR) '/logs' has insecure permissions
08:54:01 Event alarms enabled. ALARMPROG = '/usr/informix/etc/alarmprogram.sh'
08:54:01 Dynamically allocated new virtual shared memory segment (size 8316KB)
08:54:01 Booting Language <c> from module <>
08:54:01 Loading Module <CNULL>
08:54:01 Booting Language <builtin> from module <>
08:54:01 Loading Module <BUILTINNULL>
08:54:06 Dynamically allocated new virtual shared memory segment (size 8192KB)
08:54:07 DR: DRAUTO is 0 (Off)
08:54:08 Dynamically allocated new message shared memory segment (size 124KB)
08:54:08 IBM Informix Dynamic Server Version 10.00.FC5 Software Serial Number AAA#B000000
08:54:08 Assert Warning: chunk failed sanity check

08:54:08 IBM Informix Dynamic Server Version 10.00.F
08:54:08 Who: Session(1, informix@itest.nmcourts.com, 0, 0x45e01028)
Thread(17, main_loop(), 45dc6028, 1)
File: rspartn.c Line: 8673
08:54:08 Results: Chunk 1 is being taken OFFLINE.
08:54:08 Action: Restore chunk from archive.
08:54:08 stack trace for pid 21487 written to /logs/af.3f9479f
08:54:08 See Also: /logs/af.3f9479f

08:54:08 chunk failed sanity check

08:54:08 invoke_alarm(): /bin/sh -c '/usr/informix/etc/alarmprogram.sh 5 6 "Internal Subsystem failure: 'MT'" "chunk failed sanity check
" '
08:54:08 invoke_alarm(): mt_exec failed, status 32512, errno 0
08:54:08 Process exited with return code 127: /bin/sh /bin/sh -c /usr/informix/etc/alarmprogram.sh 3 4 "Chunk is off-line, mirror is active:
17894197." "chunk failed s
08:54:08 I/O error, Primary Chunk '/logs/engine/rootdbs' -- Offline (sanity)
08:54:08 IBM Informix Dynamic Server Stopped.

08:54:08 Process exited with return code 127: /bin/sh /bin/sh -c /usr/informix/etc/alarmprogram.sh 3 4 "Chunk is off-line, mirror is active:
17893887." "I/O error, Pri
08:54:08 mt_shm_remove: WARNING: may not have removed all/correct segments
08:56:50 IBM Informix Dynamic Server Started.

Thu Feb 8 08:56:51 2007

08:56:51 Warning: ONCONFIG dump directory (DUMPDIR) '/logs' has insecure permissions
08:56:51 Event alarms enabled. ALARMPROG = '/usr/informix/etc/alarmprogram.sh'
08:56:51 Dynamically allocated new virtual shared memory segment (size 8316KB)
08:56:51 Booting Language <c> from module <>
08:56:51 Loading Module <CNULL>
08:56:51 Booting Language <builtin> from module <>
08:56:51 Loading Module <BUILTINNULL>
08:56:55 Dynamically allocated new virtual shared memory segment (size 8192KB)
08:56:57 DR: DRAUTO is 0 (Off)
08:56:57 Dynamically allocated new message shared memory segment (size 124KB)
08:56:57 IBM Informix Dynamic Server Version 10.00.FC5 Software Serial Number AAA#B000000
08:56:57 Assert Warning: chunk failed sanity check
08:56:57 IBM Informix Dynamic Server Version 10.00.F
08:56:57 Who: Session(1, informix@itest.nmcourts.com, 0, 0x45e01028)
Thread(17, main_loop(), 45dc6028, 1)
File: rspartn.c Line: 8673
08:56:57 Results: Chunk 1 is being taken OFFLINE.
08:56:57 Action: Restore chunk from archive.
08:56:57 stack trace for pid 21598 written to /logs/af.3f94849
08:56:57 See Also: /logs/af.3f94849
08:56:57 chunk failed sanity check

08:56:57 invoke_alarm(): /bin/sh -c '/usr/informix/etc/alarmprogram.sh 5 6 "Internal Subsystem failure: 'MT'" "chunk failed sanity check
" '
08:56:57 invoke_alarm(): mt_exec failed, status 32512, errno 0
08:56:57 Process exited with return code 127: /bin/sh /bin/sh -c /usr/informix/etc/alarmprogram.sh 3 4 "Chunk is off-line, mirror is active:
17894197." "chunk failed s
08:56:58 I/O error, Primary Chunk '/logs/engine/rootdbs' -- Offline (sanity)
08:56:58 IBM Informix Dynamic Server Stopped.

08:56:58 Process exited with return code 127: /bin/sh /bin/sh -c /usr/informix/etc/alarmprogram.sh 3 4 "Chunk is off-line, mirror is active:
17893887." "I/O error, Pri
08:56:58 mt_shm_remove: WARNING: may not have removed all/correct segments