Thanks Art,
This info will be useful either this time around or in
the future for checking chunks validity.
Jim
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
Kagel
Sent: Monday, June 20, 2011 1:24 PM
To: ids@iiug.org
Subject: Re: is there a way to find out the physical de.... [24093]
Sanity check failures mean that the pages in the chunks do not "look
right". The 20 byte at the beginning of each page doesn't look like it
contains the correct header information or the checksum at the bottom of the
page does not match the calculated checksum for the page. Also the first
page of the first chunk must contain copyright, platform, and version
details.
Just for information sake. Good luck.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.
On Mon, Jun 20, 2011 at 1:31 PM, jim-cramer@uiowa.edu
<jim-cramer@uiowa.edu>wrote:
> Hi Art,
>
> Thanks for your reply from 6/16 with some suggested things to check.
>
> The permissions and ownership of the chunk device files, in /dev/mapper
> directory, had been correct.
>
> Your suggestion about changing the ownership of the links might
> have helped. Before I changed the ownership, I had errno 2 on rootdbs
> and this logged into online.log:
>
> 15:30:23 Assert Warning: I/O read chunk 1, pagenum 6641, pagecnt 1 -->
> errno = 2
> 15:30:23 IBM Informix Dynamic Server Version 11.70.FC1GE
> 15:30:23 Who: Session(9, informix@s-l008, 0, 0x50291c20)
>
> Thread(29, xchg_1.2, 502519a8, 3)
>
> File: rsbuff.c Line: 6302
> 15:30:23 Action: Please notify IBM Informix Technical Support.
> 15:30:23 stack trace for pid 12822 written to /opt/db/ids/tmp/af.40567de
> 15:30:23 See Also: /opt/db/ids/tmp/af.40567de
> 15:30:23 I/O read chunk 1, pagenum 6641, pagecnt 1 --> errno = 2
> 15:30:23 Rollforward of log record failed. iserrno = 126
> 15:30:23 Log Record: log = 27, pos = 0x3ef8518, type =
OLDRSAM:HINSERT(40),
> trans = 26
> 15:30:23 Assert Failed: INFORMIX-OnLine Must ABORT
>
> Critical media failure.
>
> (By the way, I also noticed just now that one of my attempts to restart
the
> instance failed on Chunk 6, which is the first chunk of my Logical Logs,
> instead of on Chunk 1.)
>
> NOW:
>
> The ownership of my symbolic link in /dev was 777 BEFORE (when the crash
> shown above
> happened) but the ownership was root. So, I changed the ownership of the
> link to
> informix:informix and quite a bit of the above error message went away.
>
> However, I still had an assertion failure, but of a different nature.
> Instead of
> an errno 2 error on my 1st chunk, some sort of sanity check fails at a
> different
> routine in the Thread on I now get this in online.log:
>
> 10:52:59 IBM Informix Dynamic Server Version 11.70.FC1GE Software Serial
> Number AAA#B000000
> 10:52:59 Warning: stat() failed for chunk file /dev/rootdbs
> 10:53:00 Assert Warning: chunk failed sanity check
>
> 10:53:00 IBM Informix Dynamic Server Version 11.70.FC1GE
> 10:53:00 Who: Session(1, informix@s-l008, 0, 0x5028e028)
>
> Thread(8, main_loop(), 50249028, 3)
>
> File: rspartn.c Line: 10531
> 10:53:00 Results: Chunk 1 is being taken OFFLINE.
> 10:53:00 Action: For all spaces other than temporary dbspaces, restore the
> space
>
> containing the chunk from the archive. If this chunk belongs to a
>
> temporary dbspace and if it is not the only chunk in that
> dbspace,
>
> then drop and re-create the chunk. Otherwise drop and re-create
>
> the temporary dbspace.
> 10:53:00 stack trace for pid 8923 written to /opt/db/ids/tmp/af.3f06cdc
> 10:53:00 See Also: /opt/db/ids/tmp/af.3f06cdc
> 10:53:00 chunk failed sanity check
>
> 10:53:01 I/O error, Primary Chunk '/dev/rootdbs' -- Offline (sanity)
> 10:53:01 oninit: Fatal error in shared memory initialization
>
> 10:53:01 IBM Informix Dynamic Server Stopped.
>
> 10:53:01 mt_shm_remove: WARNING: may not have removed all/correct segments
>
> So, I am going to go ahead and change the ownership of all of the symbolic
> links that I created for chunks to informix:informix and see what happens.
>
> I will let you know.
>
> Thanks,
>
> Jim
>
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
> Kagel
> Sent: Thursday, June 16, 2011 5:59 PM
> To: ids@iiug.org
> Subject: Re: is there a way to find out the physical de.... [24068]
>
> Errno 2 is permissions (EPERM). Make sure that all of the low level chunk
> devices are informix:informix with 0660 privileges and make sure that the
> links themselves are 777 (owner of the links doesn't really matter, but I
> usually make them informix:informix also).
>
> Art
>
> Art S. Kagel
> Advanced DataTools (www.advancedatatools.com)
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and
>
> do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
> organization with which I am associated either explicitly, implicitly, or
> by
>
> inference. Neither do those opinions reflect those of other individuals
> affiliated with any entity with which I am affiliated nor those of the
> entities themselves.
>
> On Thu, Jun 16, 2011 at 5:17 PM, jim-cramer@uiowa.edu
> <jim-cramer@uiowa.edu>wrote:
>
> > Hi Art,
> >
> > Well, no doubt about it, you are THE MAN! I *almost* have the server
> > back up (see below). Good news and bad news.
> >
> > Thanks very much for the clever way of recreating lost symbolic links
> > used originally to create chunks and pointing these links at the actual
> > chunk device file paths/names. This method is based upon your expert
> > knowledge
> > of chunk storage internals.
> >
> > I used your method to get the chunk number for each chunk/block device
> > file (using RAW chunks), looked for that number in the oncfg file's
> > "Chunks" section, got the name of the symbolic link used to create
> > the chunk from that line in the file, and created a script to recreate
> > the symbolic links/pathnames that had been used in the onspaces commands
> > that created the chunks.
> >
> > The server begins to start with the "oninit" command, but it ABORTS.
> > According to the online.log file, it gets quite a ways through startup
> > but finds something wrong, Errno 2 (missing file or directory?)
> > with chunk 1, my rootdbs. I have included the tail end of the log file,
> > starting just before the warning about chunk 1, continuing to the end.
> >
> > Any ideas? It seems like perhaps I made a mistake with the creation of
> > the symbolic link for rootdbs, but it looks OK I think. The "ls -l" of
it
> > produces:
> >
> > lrwxrwxrwx 1 root root 19 Jun 16 15:18 /dev/rootdbs ->
> > /dev/VolGrp01/vol01
> >
> > and the rootdbs link points to another link, and ls -l on that link
> > produces:
> >
> > lrwxrwxrwx 1 root root 26 Jun 15 15:49 /dev/VolGrp01/vol01 ->
> > /dev/mapper/VolGrp01-vol01
> >
> > and ls -l on the target of that link, the block device file for the LV
> that
> > I used to create Chunk 1 with, produces:
> >
> > brw-rw---- 1 informix informix 253, 5 Jun 15 10:49
> > /dev/mapper/VolGrp01-vol01
> >
> > The "ls -lL /dev/rootdbs" command produces:
> >
> > brw-rw---- 1 informix informix 253, 5 Jun 15 10:49 /dev/rootdbs
> >
> > I am new to Linux (SUSE 11 SLES) and trying to migrate IDS from HPUX to
> it.
> > I
> > do not really understand the "device-mapper" Linux functionality that is
> > creating
> > symbolic links instead of Logical Volume device files in the traditional
> > directory
> > created for the Volume Group that the LVs belong to, and pointing those
> > links in
> > the VG dir to /dev/mapper, where the device files for the LV actually
> > reside.
> >
> > So, due to my lack of knowledge of mapper, the following might be the
> > problem and
> > I would appreciate your opinion on it.
> >
> > Originally I created the sym links in /dev to point straight at the
> device
>
> > file
> > in /dev/mapper. For example, the rootdbs was created with:
> >
> > ln -s /dev/mapper/VolGrp01-vol01b /dev/rootdbs
> >
> > However, due to some discussions with our Linux Sys Admins here, I have
> > recreated
> > the links in /dev to point to the sym links in the /dev/vgxx dir, which
> > then
> > point
> > to the actual device files that I have set the ownershiip and
permissions
> > on
> >
> > correctly.
> >
> > The rootdbs was created with the path /dev/rootdbs, and the "ls -lL" on
> > that
> > sym link/path
> > shows the target of this double-linking to be:
> >
> > brw-rw---- 1 informix informix 253, 5 Jun 15 10:49 /dev/rootdbs
> >
> > which looks OK permissions and ownership-wise, but perhaps the double
> > linking is the problem?
> >
> > What do you think? Sorry this is so long. Thanks for bearing with me.
> >
> > Jim
> >
> > 15:30:20 Event notification facility epoll enabled.
> > 15:30:20 IBM Informix Dynamic Server Version 11.70.FC1GE Software Serial
> > Number AAA#B000000
> > 15:30:20 Warning: stat() failed for chunk file /dev/rootdbs
> > 15:30:22 IBM Informix Dynamic Server Initialized -- Shared Memory
> > Initialized.
> >
> > 15:30:22 Started 1 B-tree scanners.
> > 15:30:22 B-tree scanner threshold set at 5000.
> > 15:30:22 B-tree scanner range scan size set to -1.
> > 15:30:22 B-tree scanner ALICE mode set to 6.
> > 15:30:22 B-tree scanner index compression level set to med.
> > 15:30:22 Physical Recovery Started at Page (2:131643).
> > 15:30:23 Physical Recovery Complete: 8 Pages Examined, 8 Pages Restored.
> > 15:30:23 Logical Recovery Started.
> > 15:30:23 Dynamically allocated new virtual shared memory segment (size
> > 8192KB)
> > 15:30:23 Memory sizes:resident:192512 KB, virtual:253840 KB,
> > SHMTOTAL:16777216 KB
> > 15:30:23 10 recovery worker threads will be started.
> > 15:30:23 Assert Warning: I/O read chunk 1, pagenum 6641, pagecnt 1 -->
> > errno = 2
> > 15:30:23 IBM Informix Dynamic Server Version 11.70.FC1GE
> > 15:30:23 Who: Session(9, informix@s-l008, 0, 0x50291c20)
> >
> > Thread(29, xchg_1.2, 502519a8, 3)
> >
> > File: rsbuff.c Line: 6302
> > 15:30:23 Action: Please notify IBM Informix Technical Support.
> > 15:30:23 stack trace for pid 12822 written to /opt/db/ids/tmp/af.40567de
> > 15:30:23 See Also: /opt/db/ids/tmp/af.40567de
> > 15:30:23 I/O read chunk 1, pagenum 6641, pagecnt 1 --> errno = 2
> > 15:30:23 Rollforward of log record failed. iserrno = 126
> > 15:30:23 Log Record: log = 27, pos = 0x3ef8518, type =
> OLDRSAM:HINSERT(40),
> > trans = 26
> > 15:30:23 Assert Failed: INFORMIX-OnLine Must ABORT
> >
> > Critical media failure.
> > 15:30:23 IBM Informix Dynamic Server Version 11.70.FC1GE
> > 15:30:23 Who: Session(9, informix@s-l008, 0, 0x50291c20)
> >
> > Thread(29, xchg_1.2, 502519a8, 3)
> >
> > File: rsmirror.c Line: 1734
> > 15:30:23 stack trace for pid 12822 written to /opt/db/ids/tmp/af.40567de
> > 15:30:23 See Also: /opt/db/ids/tmp/af.40567de, shmem.40567de.0
> > 15:30:28 Starting crash time check of:
> > 15:30:28 1. memory block headers
> > 15:30:28 2. stacks
> > 15:30:28 Crash time checking found no problems
> > 15:30:28 rsmirror.c, line 1734, thread 29, proc id 12822,
INFORMIX-OnLine
> > Must ABORT
> >
> > Critical media failure..
> > 15:30:29 Fatal error in ADM VP at mt.c:14250
> > 15:30:29 Unexpected virtual processor termination, pid = 12822, exit =
> > 0x100
> >
> > 15:30:29 PANIC: Attempting to bring system down
> >
> > -----Original Message-----
> > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> Art
> > Kagel
> > Sent: Thursday, June 16, 2011 1:34 PM
> > To: ids@iiug.org
> > Subject: Re: is there a way to find out the physical de.... [24060]
> >
> > Unfortunately, no. Informix only stored the links you provide, it does
> not
>
> > follow the links to their core names at all, it depends on the OS to
> > maintain and promote the links.
> >
> > HOWEVER, there is one way. Use dd piped to od -x to read the first few
> > pages (so 2K each if you used the default pagesize or whatever the
> pagesize
> > of the dbspaces were) of each chunk's physical device. The first six
> bytes
>
> > of each page have the chunk number encoded. So on my server for chunk
#1:
> >
> > $ dd if=/opt/informix/infmx.11.70.FC1B7/dbspaces/rootdb.1.chunk bs=2k
> > count=2|od -x
> > 2+0 records in
> > 2+0 records out
> > 4096 bytes (4.1 kB) copied, 0.00441886 s, 927 kB/s
> > 0000000 *0000 0000 0001* 9600 0003 1800 0130 06c0
> > 0000020 0000 0000 0000 0000 4249 204d 6e49 6f66
> >
> > For chunk #6:
> >
> > $ dd if=/opt/informix/infmx.11.70.FC1B7/dbspaces/tempdbs1.1.chunk bs=2k
> > count=1|od -x
> > 1+0 records in
> > 1+0 records out
> > 2048 bytes (2.0 kB) copied, 7.7661e-05 s, 26.4 MB/s
> > 0000000 *0000 0000 0006* 492e 0000 1800 0018 07e4
> > 0000020 0000 0000 0000 0000 0000 0000 0000 0000
> >
> > Armed with the chunk numbers you should be able to recreate the links.
> >
> > Art S. Kagel
> > Advanced DataTools (www.advancedatatools.com)
> > Blog: http://informix-myview.blogspot.com/
> >
> > Disclaimer: Please keep in mind that my own opinions are my own opinions
> > and
> >
> > do not reflect on my employer, Advanced DataTools, the IIUG, nor any
> other
>
> > organization with which I am associated either explicitly, implicitly,
or
> > by
> >
> > inference. Neither do those opinions reflect those of other individuals
> > affiliated with any entity with which I am affiliated nor those of the
> > entities themselves.
> >
> > On Thu, Jun 16, 2011 at 2:23 PM, jim-cramer@uiowa.edu
> > <jim-cramer@uiowa.edu>wrote:
> >
> > > Help!
> > >
> > > 11.70.FC1 on Linux SUSE 11 SLES
> > >
> > > After setting up a 11.7 IDS instance and creating a bunch
> > > of Logical Volumes, creating Symbolic Links in /dev that
> > > pointed to the block device files that the LV Manager created,
> > > changing the ownership and permission on these device files
> > > appropriately, creating dbspaces from the chunks *using the
> > > Symbolic Link pathnames* to the chunk device files,
> > > I found out that when the Linux system was rebooted all
> > > of the Symbolic Links for my Informix Chunks were deleted
> > > from /dev (and also that the ownership and permissions for the
> > > chunk devices files had reverted to my pre-configured state).
> > >
> > > I need to re-create the symbolic links for the chunks, in
> > > some non-system directory I assume.
> > >
> > > The names of the symbolic links that I created indicated
> > > what dbspace the each chunk belonged to. I have those link names in
> > > my $INFORMIXDIR/etc/oncfg_servername.0 file as the pathname
> > > to each Chunks.
> > >
> > > UNFORTUNATELY, the names that I gave the Logical
> > > Volumes, and hence the names of the devices files created
> > > for the LVs/Chunks, did not indicate which Chunk/DBSpace
> > > each was created for (i.e.: I used a systematic pattern vol01,
> > > vol02, vol02,...)
> > >
> > > So, I need a way to find out the physical device file pathname
> > > that each of the symbolic links had pointed to, so that I
> > > can recreate the links correctly.
> > >
> > > OR
> > >
> > > since those links have been blown away by the OS and the instance
> > > is down, is there a file somewhere where I would find this info?
> > >
> > > Or
> > >
> > > is there an Informix Utility that can be run against a down instance
> > > that would list out the chunks that had been created for that instance
> > > along with the physical device file pathname for each?
> > >
> > > Any information or ideas would be appreciated.
> > >
> > > Thanks,
> > >
> > > Jim Cramer
> > > Univ of Iowa
> > >
> > >
> > >
> > >
> >
> >
>
>
****************************************************************************
>
> > ***
> > > Forum Note: Use "Reply" to post a response in the discussion forum.
> > >
> > >
> >
> > --bcaec5016349fe9e0304a5d882f3
> >
> >
> >
>
>
****************************************************************************
>
> > ***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
> >
> >
>
>
****************************************************************************
> ***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --bcaec51a7e3ed80ad604a5dc379a
>
>
>
****************************************************************************
> ***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
>
****************************************************************************
***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--20cf3071c696d3e43d04a628d547
****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.
