I have two separate servers with two similar but separate issues. Both involve Secondary servers in a DR setup. Both are due to a power outage that caused these servers to power down without Informix being shut down.
For the first server, here is the error log:
16:19:21 Event alarms enabled. ALARMPROG = '/ifmx/ifmx00175/etc/log_full.sh'
16:19:22 Booting Language <c> from module <>
16:19:22 Loading Module <CNULL>
16:19:22 Booting Language <builtin> from module <>
16:19:22 Loading Module <BUILTINNULL>
16:19:22 Could not disable priority aging: errno = 1
16:19:22 Could not disable priority aging: errno = 1
16:19:22 Could not disable priority aging: errno = 1
16:19:22 Could not disable priority aging: errno = 1
16:19:22 Could not disable priority aging: errno = 1
16:19:22 Could not disable priority aging: errno = 1
16:19:28 DR: DRAUTO is 0 (Off)
16:19:28 Requested shared memory segment size rounded from 544KB to 1024KB
16:19:29 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial Number AAA#B000000
16:19:29 WARNING: Next backup of DBspace llogdbs must be level-0 backup.
16:19:29 WARNING: Next backup of DBspace plogdbs must be level-0 backup.
16:19:29 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.
16:19:29 Physical Recovery Started at Page (3:29800).
16:19:29 Physical Recovery Complete: 18 Pages Examined, 18 Pages Restored.
16:19:29 DR: Trying to connect to primary server = ifmx00174_repl
16:19:30 Dataskip is now OFF for all dbspaces
16:19:30 Restartable Restore has been ENABLED
16:19:30 Recovery Mode
16:19:33 DR: Secondary server connected
16:19:34 DR: Configuration parameter values of the paired servers do not match:
Parameter: TAPESIZE
Current server's value: 20480000
Paired server's value: 30720000
WARNING: The parameter values must match if ontape is used as
backup/recovery tool.
Please adjust this parameter value in onconfig file to correctly
reflect the setting of the primary server.
16:19:34 DR: Secondary server needs failure recovery
16:19:34 DR: Failure recovery from disk in progress ...
16:19:34 Logical Recovery Started.
16:19:34 10 recovery worker threads will be started.
16:19:34 Start Logical Recovery - Start Log 54282, End Log ?
16:19:34 Starting Log Position - 54282 0x10018
16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap
16:19:34 IBM Informix Dynamic Server Version 10.00.UC6
16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0)
Thread(129, xchg_1.8, 3fc1d548, 3)
File: rsdebug.c Line: 1081
16:19:34 Results: Possible inconsistencies in an index of 'campaign:"dbaparts".camptss_ses'
"/ifmx/ifmx00175/online.log" [Read only] 92 lines, 5329 characters
16:19:30 Dataskip is now OFF for all dbspaces
16:19:30 Restartable Restore has been ENABLED
16:19:30 Recovery Mode
16:19:33 DR: Secondary server connected
16:19:34 DR: Configuration parameter values of the paired servers do not match:
Parameter: TAPESIZE
Current server's value: 20480000
Paired server's value: 30720000
WARNING: The parameter values must match if ontape is used as
backup/recovery tool.
Please adjust this parameter value in onconfig file to correctly
reflect the setting of the primary server.
16:19:34 DR: Secondary server needs failure recovery
16:19:34 DR: Failure recovery from disk in progress ...
16:19:34 Logical Recovery Started.
16:19:34 10 recovery worker threads will be started.
16:19:34 Start Logical Recovery - Start Log 54282, End Log ?
16:19:34 Starting Log Position - 54282 0x10018
16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap
16:19:34 IBM Informix Dynamic Server Version 10.00.UC6
16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0)
Thread(129, xchg_1.8, 3fc1d548, 3)
File: rsdebug.c Line: 1081
16:19:34 Results: Possible inconsistencies in an index of 'campaign:"dbaparts".camptss_ses'
16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses'
16:19:34 stack trace for pid 5543 written to /dbengines/informix/tmp/af.46904f6
16:19:34 See Also: /dbengines/informix/tmp/af.46904f6, shmem.46904f6.0
16:19:34 Assert Warning: Error during recovery left index inconsistent.
16:19:34 IBM Informix Dynamic Server Version 10.00.UC6
16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520)
Thread(130, xchg_1.9, 3fc1da74, 5)
File: rskey.c Line: 1516
16:19:34 Results: Index 'campaign:"dbaparts".camptss_ses# 120_163' is now unusable
16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses# 120_163'
16:19:34 stack trace for pid 5559 written to /dbengines/informix/tmp/af.46a04f6
16:19:34 See Also: /dbengines/informix/tmp/af.46a04f6
16:19:35 Rollforward of log record failed. iserrno = 126
16:19:35 Log Record: log = 54282, pos = 0x1906c, type = OLDRSAM:HINSERT(40), trans = 91
16:19:35 Error during recovery left index inconsistent.
16:19:35 Rollforward of log record failed. iserrno = 105
16:19:35 Log Record: log = 54282, pos = 0x1917c, type = OLDRSAM:ADDITEM(28), trans = 91
16:19:35 Rollforward of log record failed. iserrno = 105
16:19:35 Log Record: log = 54282, pos = 0x1917c, type = OLDRSAM:ADDITEM(28), trans = 91
16:19:44 Assert Warning: Chunk 8 is being taken OFFLINE.
16:19:44 IBM Informix Dynamic Server Version 10.00.UC6
16:19:44 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520)
Thread(126, xchg_1.5, 3fc1c5c4, 1)
File: rsmirror.c Line: 1846
16:19:44 Results: Dynamic Server will block at next checkpoint
16:19:44 Action: Shutdown (onmode -k) or override (onmode -O)
16:19:44 stack trace for pid 5462 written to /dbengines/informix/tmp/af.46604f6
16:19:44 See Also: /dbengines/informix/tmp/af.46604f6
16:19:45 Chunk 8 is being taken OFFLINE.
16:19:45 Rollforward of log record failed. iserrno = 126
16:19:45 Log Record: log = 54282, pos = 0x1906c, type = OLDRSAM:HINSERT(40), trans = 91
16:19:45 Rollforward of log record failed. iserrno = 101
16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17), trans = 93
16:19:45 Rollforward of log record failed. iserrno = 101
16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17), trans = 93
16:21:38 Page Check Error in page_reorg: slots overlap
16:21:38 Rollforward of log record failed. iserrno = 105
16:21:38 Log Record: log = 54282, pos = 0x1b140, type = OLDRSAM:ADDITEM(28), trans = 93
16:21:38 Rollforward of log record failed. iserrno = 105
16:21:38 Log Record: log = 54282, pos = 0x1b140, type = OLDRSAM:ADDITEM(28), trans = 93
16:21:38 Checkpoint blocked by down space, waiting for override or shutdown
I tried to run 'oncheck -cI campaign:"dbaparts".camptss_ses,' but it tells me that I can't run it because I don't own that table. The chunk that is down includes all of the indexes for this instance. Is it possible to drop the chunk, recreate the chunk, and then recreate the indexes, or will we need to restore the data from a backup?
17:13:34 Event alarms enabled. ALARMPROG = '/ifmx/ifmx00804/etc/event.pl'
17:13:34 Booting Language <c> from module <>
17:13:34 Loading Module <CNULL>
17:13:34 Booting Language <builtin> from module <>
17:13:34 Loading Module <BUILTINNULL>
17:13:34 Could not disable priority aging: errno = 1
17:13:34 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:35 Could not disable priority aging: errno = 1
17:13:41 DR: DRAUTO is 0 (Off)
17:13:41 Requested shared memory segment size rounded from 544KB to 1024KB
17:13:43 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial Number AAA#B000000
17:13:44 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.
17:13:44 Physical Recovery Started at Page (3:206904).
17:13:44 Physical Recovery Complete: 36 Pages Examined, 36 Pages Restored.
17:13:44 DR: Trying to connect to primary server = ifmxrepla
17:13:44 Dataskip is now OFF for all dbspaces
17:13:44 Restartable Restore has been ENABLED
17:13:44 Recovery Mode
17:13:49 DR: Secondary server connected
17:13:50 DR: Secondary server needs failure recovery
17:13:50 DR: Failure recovery from disk in progress ...
17:13:51 Logical Recovery Started.
17:13:51 10 recovery worker threads will be started.
17:13:51 Start Logical Recovery - Start Log 128356, End Log ?
17:13:51 Starting Log Position - 128356 0x1fb018
17:13:52 Rollforward of log record failed. iserrno = 126
17:13:52 Log Record: log = 128356, pos = 0x215044, type = OLDRSAM:HINSERT(40), trans = 843
17:14:01 Assert Warning: Chunk 5 is being taken OFFLINE.
17:14:01 IBM Informix Dynamic Server Version 10.00.UC6
17:14:01 Who: Session(44, informix@t5240-02-z00, 0, 3fc4cb38)
Thread(142, xchg_1.4, 3fc34b6c, 8)
File: rsmirror.c Line: 1846
17:14:01 Results: Dynamic Server will block at next checkpoint
17:14:01 Action: Shutdown (onmode -k) or override (onmode -O)
17:14:01 stack trace for pid 9979 written to /dbengines/informix/tmp/af.47611b0
17:14:01 See Also: /dbengines/informix/tmp/af.47611b0
17:14:02 Chunk 5 is being taken OFFLINE.
17:14:03 Rollforward of log record failed. iserrno = 126
17:14:03 Log Record: log = 128356, pos = 0x215044, type = OLDRSAM:HINSERT(40), trans = 843
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x226098, type = OLDRSAM:ADDITEM(28), trans = 645
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x226098, type = OLDRSAM:ADDITEM(28), trans = 645
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x22b098, type = OLDRSAM:ADDITEM(28), trans = 152
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x22b098, type = OLDRSAM:ADDITEM(28), trans = 152
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x22f098, type = OLDRSAM:ADDITEM(28), trans = 986
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x22f098, type = OLDRSAM:ADDITEM(28), trans = 986
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x22d0a4, type = OLDRSAM:DELITEM(29), trans = 986
17:14:03 Rollforward of log record failed. iserrno = 101
17:14:03 Log Record: log = 128356, pos = 0x22d0a4, type = OLDRSAM:DELITEM(29), trans = 986
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x233098, type = OLDRSAM:ADDITEM(28), trans = 149
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x233098, type = OLDRSAM:ADDITEM(28), trans = 149
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x22e0a4, type = OLDRSAM:ADDITEM(28), trans = 986
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x22e0a4, type = OLDRSAM:ADDITEM(28), trans = 986
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x2310a4, type = OLDRSAM:DELITEM(29), trans = 149
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x2310a4, type = OLDRSAM:DELITEM(29), trans = 149
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x2320a4, type = OLDRSAM:ADDITEM(28), trans = 149
17:14:04 Rollforward of log record failed. iserrno = 101
17:14:04 Log Record: log = 128356, pos = 0x2320a4, type = OLDRSAM:ADDITEM(28), trans = 149
17:14:04 Checkpoint blocked by down space, waiting for override or shutdown
The chunk that is down does contain data. Is there any way of avoiding restoring the entire database to this server, or can we follow some other steps?
Both are running Informix 10.00.UC6 on Sun Solaris 10.
Keith Schleicher
IT Database Administrator
Cell: 224-210-8358
Blackberry: 2242108358@messaging.sprintpcs.com<mailto:2242108358@messaging.sprintpcs.com>
Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com>
For more information, use our DBA Wiki page link below:
http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management#DatabaseManagement<http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management>
This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.