I attached more info from online log.
But I think I got the smoking gun, The chunk 25 's permission is wrong,
should be Informix/Informix
Raw Linux Lun,
[informix@ivan tmp]$ ll /dev/dm-35
brw-rw---- 1 root disk 253, 35 Jan 28 19:39 /dev/dm-35
Agree?
Thank you !
Frank
00:55:22 Maximum server connections 0
00:55:22 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked
0, Plog used 8, Llog used 2
01:00:06 Logical Log 7910 Complete, timestamp: 0xa17f35c9.
01:00:06 I/O read chunk 25, pagenum 4999066, pagecnt 1 --> errno = 2
01:00:06 I/O read chunk 25, pagenum 4998977, pagecnt 1 --> errno = 9
01:00:06 SCHAPI: [Auto Update Statistics Evaluation 66-17] Error -211
Cannot read system catalog (sysdistrib).
01:00:06 SCHAPI: [Auto Update Statistics Evaluation 66-17] Error -112 ISAM
error: there is no current record.
01:00:23 Checkpoint Completed: duration was 0 seconds.
01:00:23 Sat Jan 31 - loguniq 7911, logpos 0x301018, timestamp: 0xa17f4aa4
Interval: 283432
....................................
9:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:43 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:43 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:43 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:43 Action: Please notify IBM Informix Techical Support.
19:12:43 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:43 See Also: /tmp/af.4306f1b
19:12:43 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:44 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:44 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:44 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:44 Action: Please notify IBM Informix Techical Support.
19:12:44 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:44 See Also: /tmp/af.4306f1b
19:12:44 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:44 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:44 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:44 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:44 Action: Please notify IBM Informix Techical Support.
19:12:44 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:44 See Also: /tmp/af.4306f1b
19:12:44 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:45 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:45 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:45 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:45 Action: Please notify IBM Informix Techical Support.
19:12:45 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:45 See Also: /tmp/af.4306f1b
19:12:45 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:45 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:45 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:45 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:45 Action: Please notify IBM Informix Techical Support.
19:12:45 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:45 See Also: /tmp/af.4306f1b
19:12:45 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:45 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:45 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:45 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:45 Action: Please notify IBM Informix Techical Support.
19:12:45 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:46 See Also: /tmp/af.4306f1b
19:12:46 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:46 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:46 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:46 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
File: rsbuff.c Line: 5724
19:12:46 Action: Please notify IBM Informix Techical Support.
19:12:46 stack trace for pid 9641 written to /tmp/af.4306f1b
19:12:47 See Also: /tmp/af.4306f1b
19:12:47 I/O read chunk 25, pagenum 3617603, pagecnt 1 --> errno = 9
19:12:47 Assert Warning: I/O read chunk 25, pagenum 3617603, pagecnt 1 -->
errno = 9
19:12:47 IBM Informix Dynamic Server Version 12.10.FC4W1X1
19:12:47 Who: Session(35, informix@, 0, 0x4c4926d8)
Thread(72, dbWorker1, 4c454528, 10)
On Wed, Feb 4, 2015 at 4:22 PM, Fernando Nunes <domusonline@gmail.com>
wrote:
> This is the sort of issue that should be analized by tech support... but at
> first glance I see errno 2 and 9. 2 is file not found and 9 is bad file
> number or bad file descriptor....
>
> Do you still "have" your chunk files? Can you read from them?
>
> I don't think it's "corruption". Most likely a disk access issue. Or an
> informix bug if we tried an access with a wrong file descriptor (not likely
> I would say....)
>
> Regards
> On Feb 4, 2015 10:03 PM, "FRANK" <yunyaoqu@gmail.com> wrote:
>
> > HI, Folks,
> >
> > Is this a disk chunk corruption warning or something else?
> >
> > [informix@ivan tmp]$ more /tmp/af.68a726c
> > 19:26:37 mt_aio_wait(0x4cfe0028)
> > 19:26:37 chunkio(28, 0x9f91f800, 1365319, 1, 0x4d5d6028, 33, 1) errno 9
> > AIOREQ *a
> > 000000005047fc80: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fc90: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fca0: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fcb0: 2003fe4c 00000000 08fcfd4c 00000000 ..L.... ...L....
> > 000000005047fcc0: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fcd0: 98110000 41494f72 65710000 00000000 ....AIOr eq......
> > 000000005047fce0: 00000000 00000000 d1070000 00000000 ........ ........
> > 000000005047fcf0: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fd00: 04000000 01050000 0b000000 02000000 ........ ........
> > 000000005047fd10: 01000000 00000000 01000000 00000000 ........ ........
> > 000000005047fd20: 0801fe4c 00000000 00000000 00000000 ...L.... ........
> > 000000005047fd30: 01000000 09000000 01000000 00000000 ........ ........
> > 000000005047fd40: 30076c8e 00000000 00000000 00000000 0.l..... ........
> > 000000005047fd50: 00000000 00000000 00000000 00000000 ........ ........
> > 19:26:37
> > 19:26:37 IBM Informix Dynamic Server Version 12.10.FC4W1X1 Software
> Serial
> > Number AAA#B000000
> > 19:26:37 Assert Warning: I/O read chunk 25, pagenum 1365319, pagecnt 1
> -->
> > errno = 2
> > 19:26:37 Who: Session(149, informix@, 0, 0x4c490fa8)
> >
> > Thread(674, StorageMGR, 4c452228, 10)
> >
> > File: rsbuff.c Line: 5724
> > 19:26:37 Action: Please notify IBM Informix Techical Support.
> > 19:26:37 SHM Globals and Master Pool/Master Block Adresses:
> > 19:26:37 shmcb = 0x000000004414d768
> > 19:26:37 rhead = 0x00000000441b8800
> > 19:26:37 pool list = 0x000000004414d840
> > 19:26:37 block pool list = 0x00000000441b3048
> > 19:26:37 TRANSP = 0x000000004c490fa8
> > 19:26:37 PARTP = 0x000000004e3bc7f0
> > 19:26:37 PARTNP = 0x000000004e3bc900
> > 19:26:37 OPENP = 0x00000000504348b0
> > 19:26:37 FILEP = 0x0000000050434ce8
> > 19:26:37 Raw hex dump of stack located in /tmp/af.68a726c.rawstk
> > 19:26:37 Stack for thread: 674 StorageMGR
> >
> > ..................
> >
> > 19:26:37 IBM Informix Dynamic Server Version 12.10.FC4W1X1 Software
> Serial
> > Number AAA#B000000
> > 19:26:37 Assert Warning: I/O read chunk 25, pagenum 2834541, pagecnt 1
> -->
> > errno = 9
> > 19:26:37 Who: Session(149, informix@, 0, 0x4c490fa8)
> >
> > Thread(674, StorageMGR, 4c452228, 10)
> >
> > File: rsbuff.c Line: 5724
> > 19:26:37 Action: Please notify IBM Informix Techical Support.
> > 19:26:37 SHM Globals and Master Pool/Master Block Adresses:
> > 19:26:37 shmcb = 0x000000004414d768
> > 19:26:37 rhead = 0x00000000441b8800
> > 19:26:37 pool list = 0x000000004414d840
> > 19:26:37 block pool list = 0x00000000441b3048
> > 19:26:37 TRANSP = 0x000000004c490fa8
> > 19:26:37 PARTP = 0x000000004e3bc7f0
> > 19:26:37 PARTNP = 0x000000004e3bc900
> > 19:26:37 OPENP = 0x00000000504348b0
> > 19:26:37 FILEP = 0x0000000050434ce8
> > 19:26:37 Raw hex dump of stack located in /tmp/af.68a726c.rawstk
> > 19:26:37 Stack for thread: 674 StorageMGR
> > base: 0x0000000050461000
> > len: 135168
> >
> > pc: 0x00000000013d6d26
> > tos: 0x000000005047f080
> > state: running
> >
> > vp: 10
> > 0x00000000013d6d26 (oninit) afstack
> > 0x00000000013d9e0d (oninit) afhandler
> > 0x00000000013da14d (oninit) afwarn_interface
> > 0x0000000000d770b3 (oninit) physio
> > 0x0000000000d7e6d0 (oninit) buffget
> > 0x0000000000e48e10 (oninit) rssample_index_for_index_compression
> > 0x00000000014a748c (oninit) fmreorg_idx
> > 0x0000000000ac37a5 (oninit) docompress_idx_op
> > 0x00000000013d3e70 (oninit) th_init_initgls
> > 0x000000000139d268 (oninit) startup
> >
> > siginfo: <NULL>
> > 19:26:38 See Also: /tmp/af.68a726c
> > ---------------------------------
> > Begin System Alarm Program Output
> > ---------------------------------
> > Assertion Failure Type: Warning
> > Host Name: ivan
> > Database Server Name: nsofintdwref_tcp
> > Time of failure: Wed Feb 4 19:26:38 UTC 2015
> > AF file: /tmp/af.68a726c
> > Shared memory file: None
> > System Blocking: OFF
> >
> > -------------------------------
> > End System Alarm Program Output
> > -------------------------------
> > 19:26:38 sh /usr/informix/etc/evidence.sh 1 0 /tmp/af.68a726c 149
> > 0x4c452228 674 0x4ef7a028 1025 0 0 0 0
> > 19:26:38
> > ------------------ End of assertion failure 2 -----------------
> > 19:26:38 mt_aio_wait(0x4cfe0028)
> > 19:26:38 chunkio(28, 0x9f9ed800, 2849591, 1, 0x4d5d6028, 33, 1) errno 9
> > AIOREQ *a
> > 000000005047fc80: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fc90: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fca0: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fcb0: 2003fe4c 00000000 08fcfd4c 00000000 ..L.... ...L....
> > 000000005047fcc0: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fcd0: 98110000 41494f72 65710000 00000000 ....AIOr eq......
> > 000000005047fce0: 00000000 00000000 d1070000 00000000 ........ ........
> > 000000005047fcf0: 00000000 00000000 00000000 00000000 ........ ........
> > 000000005047fd00: 04000000 01050000 0b000000 02000000 ........ ........
> > 000000005047fd10: 01000000 00000000 01000000 00000000 ........ ........
> > 000000005047fd20: 0801fe4c 00000000 00000000 00000000 ...L.... ........
> > 000000005047fd30: 00000000 09000000 01000000 00000000 ........ ........
> > 000000005047fd40: b0086d8e 00000000 00000000 00000000 ..m..... ........
> > 000000005047fd50: 00000000 00000000 00000000 00000000 ........ ........
> > 19:26:38
> > 19:26:38 IBM Informix Dynamic Server Version 12.10.FC4W1X1 Software
> Serial
> > Number AAA#B000000
> > 19:26:38 Assert Warning: I/O read chunk 25, pagenum 2849591, pagecnt 1
> -->
> > errno = 9
> > 19:26:38 Who: Session(149, informix@, 0, 0x4c490fa8)
> >
> > Thread(674, StorageMGR, 4c452228, 10)
> >
> > File: rsbuff.c Line: 5724
> > 19:26:38 Action: Please notify IBM Informix Techical Support.
> > 19:26:38 SHM Globals and Master Pool/Master Block Adresses:
> > 19:26:38 shmcb = 0x000000004414d768
> > 19:26:38 rhead = 0x00000000441b8800
> > 19:26:38 pool list = 0x000000004414d840
> > 19:26:38 block pool list = 0x00000000441b3048
> > 19:26:38 TRANSP = 0x000000004c490fa8
> > 19:26:38 PARTP = 0x000000004e3bc7f0
> > 19:26:38 PARTNP = 0x000000004e3bc900
> > 19:26:38 OPENP = 0x00000000504348b0
> > 19:26:38 FILEP = 0x0000000050434ce8
> > 19:26:38 Raw hex dump of stack located in /tmp/af.68a726c.rawstk
> > 19:26:38 Stack for thread: 674 StorageMGR
> >
> > But, chunk 25 looks fine from output of onstat -d ,
> >
> > Chunks
> > address chunk/dbs offset size free bpages
> > flags pathname
> >
> > 4d5d6028 25 8 0 5000000 385
> > PO-B-- /usr/informix/dev-links/nsofopsdw/dwdata00a_ops
> >
> > puzzling ....
> >
> > Thanks
> > Frank
> >
> > --001a11351944bf29bb050e4981dd
> >
> >
> >
> >
>
> *******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --047d7b33d4f26b9dfe050e49c8b4
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a11c13e268a079a050e4a3c4d