|
IDS Forum
Re: Replication issues
Posted By: Keith Simmons Date: Saturday, 26 January 2013, at 5:38 p.m.
In Response To: Replication issues (Schleicher, Keith)
Keith
Don't know if any else has responded direct, or if you have resolved this
now ?
I would suggest in both cases (after you have checked the physical disks are
OK) is to switch both Primaries to Standard Mode then re-initialise HDR by
taking archives (make sure you resolve the tape size issue0 and restoring to
the secondaries.
You very limited about what actions can be undertaken on Secondaries and I
would look very carefully at your physical disk before doing anything as it
looks
suspiciously like you have some bad areas (head crash/corruption from the
power
issue).
Keith (Simmons)
On 26 January 2013 01:30, Schleicher, Keith <Keith.Schleicher@searshc.com>wrote:
> I have two separate servers with two similar but separate issues. Both
> involve
> Secondary servers in a DR setup. Both are due to a power outage that caused
> these servers to power down without Informix being shut down.
>
> For the first server, here is the error log:
>
> 16:19:21 Event alarms enabled. ALARMPROG =
> '/ifmx/ifmx00175/etc/log_full.sh'
> 16:19:22 Booting Language <c> from module <>
> 16:19:22 Loading Module <CNULL>
> 16:19:22 Booting Language <builtin> from module <>
> 16:19:22 Loading Module <BUILTINNULL>
> 16:19:22 Could not disable priority aging: errno = 1
> 16:19:22 Could not disable priority aging: errno = 1
> 16:19:22 Could not disable priority aging: errno = 1
> 16:19:22 Could not disable priority aging: errno = 1
> 16:19:22 Could not disable priority aging: errno = 1
> 16:19:22 Could not disable priority aging: errno = 1
> 16:19:28 DR: DRAUTO is 0 (Off)
> 16:19:28 Requested shared memory segment size rounded from 544KB to 1024KB
> 16:19:29 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial
> Number
> AAA#B000000
> 16:19:29 WARNING: Next backup of DBspace llogdbs must be level-0 backup.
> 16:19:29 WARNING: Next backup of DBspace plogdbs must be level-0 backup.
> 16:19:29 IBM Informix Dynamic Server Initialized -- Shared Memory
> Initialized.
>
> 16:19:29 Physical Recovery Started at Page (3:29800).
> 16:19:29 Physical Recovery Complete: 18 Pages Examined, 18 Pages Restored.
> 16:19:29 DR: Trying to connect to primary server = ifmx00174_repl
> 16:19:30 Dataskip is now OFF for all dbspaces
> 16:19:30 Restartable Restore has been ENABLED
> 16:19:30 Recovery Mode
> 16:19:33 DR: Secondary server connected
> 16:19:34 DR: Configuration parameter values of the paired servers do not
> match:
>
> Parameter: TAPESIZE
>
> Current server's value: 20480000
>
> Paired server's value: 30720000
>
> WARNING: The parameter values must match if ontape is used as
>
> backup/recovery tool.
>
> Please adjust this parameter value in onconfig file to correctly
>
> reflect the setting of the primary server.
> 16:19:34 DR: Secondary server needs failure recovery
>
> 16:19:34 DR: Failure recovery from disk in progress ...
> 16:19:34 Logical Recovery Started.
> 16:19:34 10 recovery worker threads will be started.
> 16:19:34 Start Logical Recovery - Start Log 54282, End Log ?
> 16:19:34 Starting Log Position - 54282 0x10018
> 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap
> 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6
> 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0)
>
> Thread(129, xchg_1.8, 3fc1d548, 3)
>
> File: rsdebug.c Line: 1081
> 16:19:34 Results: Possible inconsistencies in an index of
> 'campaign:"dbaparts".camptss_ses'
> "/ifmx/ifmx00175/online.log" [Read only] 92 lines, 5329 characters
> 16:19:30 Dataskip is now OFF for all dbspaces
> 16:19:30 Restartable Restore has been ENABLED
> 16:19:30 Recovery Mode
> 16:19:33 DR: Secondary server connected
> 16:19:34 DR: Configuration parameter values of the paired servers do not
> match:
>
> Parameter: TAPESIZE
>
> Current server's value: 20480000
>
> Paired server's value: 30720000
>
> WARNING: The parameter values must match if ontape is used as
>
> backup/recovery tool.
>
> Please adjust this parameter value in onconfig file to correctly
>
> reflect the setting of the primary server.
> 16:19:34 DR: Secondary server needs failure recovery
>
> 16:19:34 DR: Failure recovery from disk in progress ...
> 16:19:34 Logical Recovery Started.
> 16:19:34 10 recovery worker threads will be started.
> 16:19:34 Start Logical Recovery - Start Log 54282, End Log ?
> 16:19:34 Starting Log Position - 54282 0x10018
> 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap
> 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6
> 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0)
>
> Thread(129, xchg_1.8, 3fc1d548, 3)
>
> File: rsdebug.c Line: 1081
> 16:19:34 Results: Possible inconsistencies in an index of
> 'campaign:"dbaparts".camptss_ses'
> 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses'
> 16:19:34 stack trace for pid 5543 written to
> /dbengines/informix/tmp/af.46904f6
> 16:19:34 See Also: /dbengines/informix/tmp/af.46904f6, shmem.46904f6.0
> 16:19:34 Assert Warning: Error during recovery left index inconsistent.
> 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6
> 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520)
>
> Thread(130, xchg_1.9, 3fc1da74, 5)
>
> File: rskey.c Line: 1516
> 16:19:34 Results: Index 'campaign:"dbaparts".camptss_ses# 120_163' is now
> unusable
> 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses# 120_163'
> 16:19:34 stack trace for pid 5559 written to
> /dbengines/informix/tmp/af.46a04f6
> 16:19:34 See Also: /dbengines/informix/tmp/af.46a04f6
> 16:19:35 Rollforward of log record failed. iserrno = 126
> 16:19:35 Log Record: log = 54282, pos = 0x1906c, type =
> OLDRSAM:HINSERT(40),
> trans = 91
> 16:19:35 Error during recovery left index inconsistent.
> 16:19:35 Rollforward of log record failed. iserrno = 105
> 16:19:35 Log Record: log = 54282, pos = 0x1917c, type =
> OLDRSAM:ADDITEM(28),
> trans = 91
> 16:19:35 Rollforward of log record failed. iserrno = 105
> 16:19:35 Log Record: log = 54282, pos = 0x1917c, type =
> OLDRSAM:ADDITEM(28),
> trans = 91
> 16:19:44 Assert Warning: Chunk 8 is being taken OFFLINE.
> 16:19:44 IBM Informix Dynamic Server Version 10.00.UC6
> 16:19:44 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520)
>
> Thread(126, xchg_1.5, 3fc1c5c4, 1)
>
> File: rsmirror.c Line: 1846
> 16:19:44 Results: Dynamic Server will block at next checkpoint
> 16:19:44 Action: Shutdown (onmode -k) or override (onmode -O)
> 16:19:44 stack trace for pid 5462 written to
> /dbengines/informix/tmp/af.46604f6
> 16:19:44 See Also: /dbengines/informix/tmp/af.46604f6
> 16:19:45 Chunk 8 is being taken OFFLINE.
> 16:19:45 Rollforward of log record failed. iserrno = 126
> 16:19:45 Log Record: log = 54282, pos = 0x1906c, type =
> OLDRSAM:HINSERT(40),
> trans = 91
> 16:19:45 Rollforward of log record failed. iserrno = 101
> 16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17),
> trans = 93
> 16:19:45 Rollforward of log record failed. iserrno = 101
> 16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17),
> trans = 93
> 16:21:38 Page Check Error in page_reorg: slots overlap
> 16:21:38 Rollforward of log record failed. iserrno = 105
> 16:21:38 Log Record: log = 54282, pos = 0x1b140, type =
> OLDRSAM:ADDITEM(28),
> trans = 93
> 16:21:38 Rollforward of log record failed. iserrno = 105
> 16:21:38 Log Record: log = 54282, pos = 0x1b140, type =
> OLDRSAM:ADDITEM(28),
> trans = 93
> 16:21:38 Checkpoint blocked by down space, waiting for override or shutdown
>
> I tried to run 'oncheck -cI campaign:"dbaparts".camptss_ses,' but it tells
> me
> that I can't run it because I don't own that table. The chunk that is down
> includes all of the indexes for this instance. Is it possible to drop the
> chunk, recreate the chunk, and then recreate the indexes, or will we need
> to
> restore the data from a backup?
>
> 17:13:34 Event alarms enabled. ALARMPROG = '/ifmx/ifmx00804/etc/event.pl'
> 17:13:34 Booting Language <c> from module <>
> 17:13:34 Loading Module <CNULL>
> 17:13:34 Booting Language <builtin> from module <>
> 17:13:34 Loading Module <BUILTINNULL>
> 17:13:34 Could not disable priority aging: errno = 1
> 17:13:34 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:35 Could not disable priority aging: errno = 1
> 17:13:41 DR: DRAUTO is 0 (Off)
> 17:13:41 Requested shared memory segment size rounded from 544KB to 1024KB
> 17:13:43 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial
> Number
> AAA#B000000
> 17:13:44 IBM Informix Dynamic Server Initialized -- Shared Memory
> Initialized.
>
> 17:13:44 Physical Recovery Started at Page (3:206904).
> 17:13:44 Physical Recovery Complete: 36 Pages Examined, 36 Pages Restored.
> 17:13:44 DR: Trying to connect to primary server = ifmxrepla
> 17:13:44 Dataskip is now OFF for all dbspaces
> 17:13:44 Restartable Restore has been ENABLED
> 17:13:44 Recovery Mode
> 17:13:49 DR: Secondary server connected
> 17:13:50 DR: Secondary server needs failure recovery
>
> 17:13:50 DR: Failure recovery from disk in progress ...
> 17:13:51 Logical Recovery Started.
> 17:13:51 10 recovery worker threads will be started.
> 17:13:51 Start Logical Recovery - Start Log 128356, End Log ?
> 17:13:51 Starting Log Position - 128356 0x1fb018
> 17:13:52 Rollforward of log record failed. iserrno = 126
> 17:13:52 Log Record: log = 128356, pos = 0x215044, type =
> OLDRSAM:HINSERT(40),
> trans = 843
> 17:14:01 Assert Warning: Chunk 5 is being taken OFFLINE.
> 17:14:01 IBM Informix Dynamic Server Version 10.00.UC6
> 17:14:01 Who: Session(44, informix@t5240-02-z00, 0, 3fc4cb38)
>
> Thread(142, xchg_1.4, 3fc34b6c, 8)
>
> File: rsmirror.c Line: 1846
> 17:14:01 Results: Dynamic Server will block at next checkpoint
> 17:14:01 Action: Shutdown (onmode -k) or override (onmode -O)
> 17:14:01 stack trace for pid 9979 written to
> /dbengines/informix/tmp/af.47611b0
> 17:14:01 See Also: /dbengines/informix/tmp/af.47611b0
> 17:14:02 Chunk 5 is being taken OFFLINE.
> 17:14:03 Rollforward of log record failed. iserrno = 126
> 17:14:03 Log Record: log = 128356, pos = 0x215044, type =
> OLDRSAM:HINSERT(40),
> trans = 843
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x226098, type =
> OLDRSAM:ADDITEM(28),
> trans = 645
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x226098, type =
> OLDRSAM:ADDITEM(28),
> trans = 645
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x22b098, type =
> OLDRSAM:ADDITEM(28),
> trans = 152
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x22b098, type =
> OLDRSAM:ADDITEM(28),
> trans = 152
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x22f098, type =
> OLDRSAM:ADDITEM(28),
> trans = 986
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x22f098, type =
> OLDRSAM:ADDITEM(28),
> trans = 986
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x22d0a4, type =
> OLDRSAM:DELITEM(29),
> trans = 986
> 17:14:03 Rollforward of log record failed. iserrno = 101
> 17:14:03 Log Record: log = 128356, pos = 0x22d0a4, type =
> OLDRSAM:DELITEM(29),
> trans = 986
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x233098, type =
> OLDRSAM:ADDITEM(28),
> trans = 149
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x233098, type =
> OLDRSAM:ADDITEM(28),
> trans = 149
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x22e0a4, type =
> OLDRSAM:ADDITEM(28),
> trans = 986
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x22e0a4, type =
> OLDRSAM:ADDITEM(28),
> trans = 986
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x2310a4, type =
> OLDRSAM:DELITEM(29),
> trans = 149
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x2310a4, type =
> OLDRSAM:DELITEM(29),
> trans = 149
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x2320a4, type =
> OLDRSAM:ADDITEM(28),
> trans = 149
> 17:14:04 Rollforward of log record failed. iserrno = 101
> 17:14:04 Log Record: log = 128356, pos = 0x2320a4, type =
> OLDRSAM:ADDITEM(28),
> trans = 149
> 17:14:04 Checkpoint blocked by down space, waiting for override or shutdown
>
> The chunk that is down does contain data. Is there any way of avoiding
> restoring the entire database to this server, or can we follow some other
> steps?
>
> Both are running Informix 10.00.UC6 on Sun Solaris 10.
>
> Keith Schleicher
> IT Database Administrator
> Cell: 224-210-8358
> Blackberry:
> 2242108358@messaging.sprintpcs.com<mailto:
> 2242108358@messaging.sprintpcs.com>
> Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com>
>
> For more information, use our DBA Wiki page link below:
>
>
> http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management#DatabaseManagement
> <
> http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management
> >
>
> This message, including any attachments, is the property of Sears Holdings
> Corporation and/or one of its subsidiaries. It is confidential and may
> contain
> proprietary or legally privileged information. If you are not the intended
> recipient, please delete it without reading the contents. Thank you.
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--20cf307ac899fde03204d436cf14
Messages In This Thread
- Replication issues
Schleicher, Keith -- Friday, 25 January 2013, at 8:30 p.m.
- Re: Replication issues
Keith Simmons -- Saturday, 26 January 2013, at 5:38 p.m.
- Re: Replication issues
Art Kagel -- Saturday, 26 January 2013, at 7:23 p.m.
IDS Forum is maintained by Administrator with WebBBS 5.12.
|
|