IDS Forum

IDS 11.50 Crashing

Posted By: VIKAS HIVARKAR
Date: Thursday, 2 July 2009, at 4:40 a.m.

Hi,

IDS 11.50.FC4 on Solaris 10

We migrated out test instance from IDS 11.10.FC3 to 11.50.FC4.
After changing the softlinks and copying the onconfig and sqlhosts the new instance was started simply with "oninit -v".

I could not see any error while the conversion process and the end lines were
"Conversion Completed Successfully"
"On-Line Mode"

I did a level 0 backup, rebooted the instance two time and every thing was looking good untill aftersome time, I found that the instance went down.

The online.log shows

12:27:35 stack trace for pid 10196 written to /opt/informix/lt3/dump/af.3f53a17
12:27:35 Assert Failed: No Exception Handler
12:27:35 IBM Informix Dynamic Server Version 11.50.FC4
12:27:35 Who: Session(1, informix@cseloukdudb05, 0, 1689fe028)

Thread(13, main_loop(), 1689bb028, 1)

File: mtex.c Line: 491
12:27:35 Results: Exception Caught. Type: MT_EX_OS, Context: mem
12:27:35 Action: Please notify IBM Informix Technical Support.
12:27:35 See Also: /opt/informix/lt3/dump/af.3f53a17
12:27:39 mtex.c, line 491, thread 13, proc id 10196, No Exception Handler.
12:27:39 Assert Failed: Page Check Error in rsread:bad data page
12:27:39 IBM Informix Dynamic Server Version 11.50.FC4
12:27:39 Who: Session(24, informix@, 0, 168a059e0)

Thread(79, dbWorker1, 1689cec08, 3)

File: rsdebug.c Line: 1080
12:27:39 Results: Possible inconsistencies in 'sysadmin:"informix".sysprocbody'
12:27:39 Action: Run 'oncheck -cD 1048770'
12:27:39 stack trace for pid 10198 written to /opt/informix/lt3/dump/af.4373a1b
12:27:39 The Master Daemon Died
12:27:39 See Also: /opt/informix/lt3/dump/af.4373a1b
12:27:39 SCHAPI: Error -397 System catalog (sysprocbody) corrupted.
12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL.
12:27:39 SCHAPI: Error -105 ISAM error: bad isam file format.
12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL.
12:27:39 SCHAPI: Error -7530 Missing parenthesis in the signature specified for user-defined routine lookup
12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL.
12:27:39 SCHAPI: Error -105 ISAM error: bad isam file format.
12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL.
12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Error Message: Unable to find function[ procedure ph_dbs_alert ] .
12:27:39 PANIC: Attempting to bring system down
14:44:41 IBM Informix Dynamic Server Started.
14:44:41 Requested shared memory segment size rounded from 1517568KB to 1540096KB
14:44:41 Shared memory segment will use large pages with intimate shared memory (ISM) if available
14:44:42 Segment locked: addr=10a000000, size=1577058304

The effords to take instance up (oninit) failed with error:

14:57:12 IBM Informix Dynamic Server Version 11.50.FC4 Software Serial Number AAA#B000000
14:57:12 Logical Log File not found.
14:57:12 oninit: Fatal error in shared memory initialization

14:57:12 IBM Informix Dynamic Server Stopped.

14:57:12 mt_shm_remove: WARNING: may not have removed all/correct segments

I checked for "ipcs |grep informix" and did "onmode -ky" 2 times but of no use.
The logfile and other file permissions were intact and nothing seems to be changed externally.

I restored the instance with the level 0 backup and took it online and runned UPDATE STATISTICS on every database, again every thing looked good untill today morning when i started another leve 0 backup which failed and instance crashed.

The online.log now shows

09:56:09 Level 0 Archive started on rootdbs, physdbs, logdbs, datadbs2, datadbs8
09:56:10 Assert Warning: Archive detects that page 1:13 is corrupt.
09:56:10 IBM Informix Dynamic Server Version 11.50.FC4
09:56:10 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8)

Thread(236, arcbackup1, 1689d04f8, 3)

File: rsarcbu.c Line: 3819
09:56:10 stack trace for pid 15153 written to /opt/informix/lt3/dump/af.4d4681a
09:56:10 See Also: /opt/informix/lt3/dump/af.4d4681a
09:56:11 Archive detects that page 1:13 is corrupt.
09:56:11 Assert Warning: Archive detects that page 1:14 is corrupt.
09:56:11 IBM Informix Dynamic Server Version 11.50.FC4
09:56:11 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8)

Thread(236, arcbackup1, 1689d04f8, 3)

File: rsarcbu.c Line: 3819
09:56:11 stack trace for pid 15153 written to /opt/informix/lt3/dump/af.4d4681a
09:56:11 See Also: /opt/informix/lt3/dump/af.4d4681a
09:56:11 Archive detects that page 1:14 is corrupt.
09:56:11 Assert Warning: Archive detects that page 1:15 is corrupt.
09:56:11 IBM Informix Dynamic Server Version 11.50.FC4
09:56:11 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8)

Thread(236, arcbackup1, 1689d04f8, 3)

File: rsarcbu.c Line: 3819
09:56:11 stack trace for pid 15153 written to /opt/informix/lt3/dump/af.4d4681a
09:56:11 See Also: /opt/informix/lt3/dump/af.4d4681a
09:56:11 Archive detects that page 1:15 is corrupt.
09:56:12 Assert Warning: Archive detects that page 1:16 is corrupt.
09:56:12 IBM Informix Dynamic Server Version 11.50.FC4
09:56:12 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8)

Thread(236, arcbackup1, 1689d04f8, 3)

File: rsarcbu.c Line: 3819
.
.
.

I checked the kernel parameters from machine notes can could not see any change.

Any Help on this is appriciated

Regards
Vikas

Messages In This Thread

• IDS 11.50 Crashing
  VIKAS HIVARKAR -- Thursday, 2 July 2009, at 4:40 a.m.
  • Re: IDS 11.50 Crashing
    Art Kagel -- Sunday, 5 July 2009, at 11:57 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.