I've noticed this a couple of times in onstat, for no apparent reason cached reads go from 95% or so to 0.00% and then slowly build back up to 95% over the next 24hours or so. There's a cronjob that runs at 8, noon, 5 and midnight that records a few interesting statistics on the machine. We're running Informix 7.31.UD6 upgrading to a 9.4 on HPUX 11i. The hardware is a little old and we have had problems with disk failures in the recent past, any clues?
Chris S.
Noon stats run:
Uptime
12:05pm up 18 days, 5 mins, 57 users, load average: 0.46, 0.41, 0.41
Free Disk Space
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 88747 51249 63% /
/dev/vg00/lvol1 159509 40391 103167 28% /stand
/dev/vg00/lvol8 2048000 1541798 477349 76% /var
/dev/vg00/lvol7 2048000 1894310 144902 93% /usr
/dev/vg00/lvol4 524288 127645 377515 25% /tmp
/dev/vg00/lvol6 2457600 1888515 533739 78% /opt
/dev/vg00/lvol12 155648 1141 144858 1% /opt/perl5
/dev/vg00/lvol19 1024000 889055 126530 88% /opt/informix
/dev/vg00/lvol11 307200 10529 278156 4% /opt/gnu
/dev/vg00/lvol10 1024000 379940 603844 39% /opt/carstrain
/dev/vg00/lvol9 3072000 1589200 1390347 53% /opt/carsi
/dev/vg00/lvol20 3072000 1229736 1729229 42% /opt/carsi/smo
/dev/vg00/lvol13 2048000 334130 1606763 17% /opt/apache
/dev/vg00/lvol5 20480 1114 18161 6% /home
/dev/vg00/lvol14 2457600 1594848 810532 66% /home/carsids
Database Free Space
Name KB Size KB Free KB Used Percent
------------ ------- ------- ------- -------
root 1024000 668062 355938 34.8%
dbs0 12288000 4632328 7655672 62.3%
temp1 512000 511434 566 0.1%
temp0 512000 511498 502 0.1%
------------ ------- ------- ------- -------
Total 14336000 6323322 8012678
Logical Logs 300000 243820 56180 18.7%
who
prices pts/tb Jun 15 17:54
stuberj pts/tc Jun 16 07:05
berryj pts/td Jun 16 07:33
watkinst pts/te Jun 16 07:37
skaggsk pts/tf Jun 16 07:39
engelkek pts/tg Jun 16 07:39
bishopa pts/th Jun 16 07:40
dennisg pts/ti Jun 15 11:52
hammerm pts/tj Jun 16 08:04
moughond pts/tl Jun 16 07:41
rueggt pts/tm Jun 15 08:13
westbros pts/tn Jun 16 07:41
hoffpowj pts/tp Jun 16 07:45
hellmutk pts/tq Jun 16 07:54
clarka pts/tr Jun 16 08:09
hammd pts/ts Jun 15 08:29
maloned pts/tt Jun 15 08:53
barnesc pts/tu Jun 16 07:56
wessonc pts/tw Jun 16 07:59
daviss pts/tx Jun 16 08:12
deyoungj pts/ty Jun 16 08:08
jonesa pts/tz Jun 16 08:14
sarverl pts/tA Jun 16 08:09
murleyt pts/tB Jun 16 08:09
mcgheeb pts/tC Jun 16 09:06
hammerm pts/tD Jun 16 08:16
hoodm pts/tE Jun 16 08:19
ackerk pts/tF Jun 16 09:24
keilersv pts/tG Jun 16 08:21
legera pts/tH Jun 16 08:36
hellmutk pts/tI Jun 16 08:23
moughond pts/tJ Jun 16 08:24
smithc pts/tL Jun 16 08:35
weatherb pts/tM Jun 16 10:00
bullardk pts/tN Jun 16 08:44
westbros pts/tO Jun 16 08:42
ardend pts/tP Jun 16 11:16
tilleryl pts/tQ Jun 16 09:13
childree pts/tR Jun 16 08:47
mccalluj pts/tS Jun 15 10:17
dickersd pts/tT Jun 15 10:21
westbros pts/tU Jun 16 12:01
royek pts/tV Jun 16 09:22
edwardje pts/tW Jun 16 10:26
dickersd pts/tX Jun 15 10:25
burrowsr pts/tY Jun 16 09:44
harperk pts/tZ Jun 16 09:28
reedd pts/t2 Jun 16 11:28
bradfork pts/t3 Jun 16 10:01
bakerl pts/t4 Jun 16 10:52
rueggt pts/t5 Jun 15 11:30
waltonj pts/t7 Jun 16 11:40
holleyj pts/t8 Jun 16 10:46
prices pts/t9 Jun 16 10:44
holleyj pts/tab Jun 16 11:31
erpeldic pts/tbb Jun 16 11:32
powellc pts/tdb Jun 16 11:47
Database Status
Informix Dynamic Server Version 7.31.UD6 -- On-Line -- Up 18 days 00:03:58 -- 415112 Kbytes
Userthreads
address flags sessid user tty wait tout locks nreads nwrites
d11cc018 ---P--D 1 root - 0 0 0 3496 24626
d11cc520 ---P--F 0 root - 0 0 0 0 508061
d11cca28 ---P--F 0 root - 0 0 0 0 386810
d11ccf30 ---P--F 0 root - 0 0 0 0 354661
d11cd438 ---P--F 0 root - 0 0 0 0 337935
d11cd940 ---P--F 0 root - 0 0 0 0 388000
d11cde48 ---P--F 0 root - 0 0 0 0 388613
d11ce350 ---P--F 0 root - 0 0 0 0 281473
d11ce858 ---P--F 0 root - 0 0 0 0 161051
d11ced60 ---P--- 14 root - 0 0 0 0 816
d11cf268 ---P--B 15 root - 0 0 0 1941 3472
d11cf770 Y--P--- 255079 carsu tg d68d3fb8 0 1 555 0
d11cfc78 Y--P--- 253225 prices tb d40d6c78 0 1 20 0
d11d0180 ---P--D 21 root - 0 0 0 0 0
<SNIP>
d5be60e0 Y--P--- 256093 ackerk tF d488eac0 0 1 0 0
278 active, 384 total, 380 maximum concurrent
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
61252555 63656511 38272855 0.00 4388404 8766681 84453462 94.80
isamtot open start read write rewrite delete commit rollbk
3701808677 65244515 301778893 2584721159 65058168 1805574 587069 560468 1814
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
1065 0 0 325269.04 13615.07 1298 3452
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
9670592 339 420679365 0 0 183 114309 8420667
ixda-RA idx-RA da-RA RA-pgsused lchwaits
15789770 1108760 29257518 46141844 30166786
Database Locks
--------- LISTLOCK: Lock Listing for cars database ---------
Table Level Type User Ses ID
---------------------- ------------- ------------------------ ---------- ------
cars.prog_enr_rec row:0x13f005 SR carsu 246346
cars.prog_enr_rec table intent-shared carsu 246346
cars.stufa_rec row:0xe4d01 update carsu 255076
cars.stufa_rec table intent-exclusive carsu 257326
cars.stufa_rec table intent-exclusive carsu 255062
cars.stufa_rec row:0x106f05 update carsu 255078
cars.stufa_rec row:0x120103 update carsu 254986
cars.stufa_rec table intent-exclusive carsu 254986
cars.stufa_rec row:0x116d03 update carsu 260351
cars.stufa_rec table intent-exclusive carsu 255076
cars.stufa_rec table intent-exclusive carsu 255144
cars.stufa_rec table intent-exclusive carsu 260351
cars.stufa_rec table intent-exclusive carsu 255078
cars.stufa_rec row:0x111f01 update carsu 257326
cars.stufa_rec row:0xfe801 update carsu 255144
cars.stufa_rec row:0x12ee07 update carsu 255062
cars.sysdistrib page:0xc5400 intent-shared carsu 255062
cars.sysdistrib page:0xc5400 intent-shared carsu 255076
cars.sysdistrib page:0x1300 intent-shared ackerk 258640
cars.sysdistrib table intent-shared carsu 255078
cars.sysdistrib table intent-shared keilersv 257987
cars.sysdistrib page:0xc5400 intent-shared keilersv 257987
cars.sysdistrib page:0x1300 intent-shared carsu 255144
cars.sysdistrib table intent-shared burrowsr 258393
cars.sysdistrib table intent-shared edwardje 259145
cars.sysdistrib table intent-shared ackerk 258640
cars.sysdistrib page:0xc5400 intent-shared edwardje 258099
cars.sysdistrib table intent-shared carsu 255062
cars.sysdistrib page:0x1300 intent-shared carsu 255076
cars.sysdistrib table intent-shared edwardje 258099
cars.sysdistrib table intent-shared carsu 260351
cars.sysdistrib page:0x1300 intent-shared burrowsr 258393
cars.sysdistrib table intent-shared carsu 257326
cars.sysdistrib page:0xc5400 intent-shared moughond 255082
cars.sysdistrib table intent-shared burrowsr 256779
cars.sysdistrib table intent-shared carsu 255076
cars.sysdistrib page:0x1300 intent-shared edwardje 258099
cars.sysdistrib page:0xc5400 intent-shared burrowsr 258393
cars.sysdistrib page:0xc5400 intent-shared carsu 254986
cars.sysdistrib table intent-shared carsu 255144
cars.sysdistrib page:0xc5400 intent-shared jonesa 256408
cars.sysdistrib table intent-shared carsu 257326
cars.sysdistrib table intent-shared carsu 254986
cars.sysdistrib page:0x1300 intent-shared smithc 256856
cars.sysdistrib page:0x1300 intent-shared keilersv 257987
cars.sysdistrib page:0x1300 intent-shared carsu 255078
cars.sysdistrib page:0xc5400 intent-shared ackerk 258640
cars.sysdistrib page:0xc5400 intent-shared carsu 257326
cars.sysdistrib page:0xc5400 intent-shared carsu 255078
cars.sysdistrib page:0xc5400 intent-shared carsu 260351
cars.sysdistrib page:0x1300 intent-shared carsu 254986
cars.sysdistrib page:0x1300 intent-shared jonesa 256408
cars.sysdistrib table intent-shared jonesa 256408
cars.sysdistrib page:0x1300 intent-shared edwardje 259145
cars.sysdistrib page:0xc5400 intent-shared carsu 255144
cars.sysdistrib table intent-shared smithc 256856
cars.sysdistrib page:0x1300 intent-shared burrowsr 256779
cars.sysdistrib page:0xc5400 intent-shared smithc 256856
cars.sysdistrib page:0xc5400 intent-shared burrowsr 256779
cars.sysdistrib page:0x1300 intent-shared jonesa 255744
cars.sysdistrib page:0xc5400 intent-shared edwardje 259145
cars.sysdistrib page:0x1300 intent-shared carsu 260351
cars.sysdistrib page:0x1300 intent-shared carsu 255062
cars.sysdistrib page:0xc5400 intent-shared jonesa 255744
cars.sysdistrib table intent-shared jonesa 255744
cars.sysdistrib table intent-shared moughond 255082
cars.sysdistrib page:0x1300 intent-shared moughond 255082
---------------------- ------------- ------------------------ ---------- ------
67 Locks
Done.
And the 5 o'clock run for the same day:
Uptime
5:05pm up 18 days, 5:05, 35 users, load average: 0.33, 0.37, 0.37
Free Disk Space
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 88747 51249 63% /
/dev/vg00/lvol1 159509 40391 103167 28% /stand
/dev/vg00/lvol8 2048000 1542663 476560 76% /var
/dev/vg00/lvol7 2048000 1894323 144890 93% /usr
/dev/vg00/lvol4 524288 112082 392125 22% /tmp
/dev/vg00/lvol6 2457600 1888515 533739 78% /opt
/dev/vg00/lvol12 155648 1141 144858 1% /opt/perl5
/dev/vg00/lvol19 1024000 889523 126091 88% /opt/informix
/dev/vg00/lvol11 307200 10529 278156 4% /opt/gnu
/dev/vg00/lvol10 1024000 379955 603828 39% /opt/carstrain
/dev/vg00/lvol9 3072000 1589210 1390329 53% /opt/carsi
/dev/vg00/lvol20 3072000 1229736 1729229 42% /opt/carsi/smo
/dev/vg00/lvol13 2048000 336009 1605001 17% /opt/apache
/dev/vg00/lvol5 20480 1114 18161 6% /home
/dev/vg00/lvol14 2457600 1594981 810392 66% /home/carsids
Database Free Space
Name KB Size KB Free KB Used Percent
------------ ------- ------- ------- -------
root 1024000 668062 355938 34.8%
dbs0 12288000 4632246 7655754 62.3%
temp1 512000 511514 486 0.1%
temp0 512000 511498 502 0.1%
------------ ------- ------- ------- -------
Total 14336000 6323320 8012680
Logical Logs 300000 207832 92168 30.7%
who
prices pts/tb Jun 15 17:54
bullardk pts/td Jun 16 16:54
engelkek pts/tg Jun 16 07:39
dennisg pts/ti Jun 15 11:52
ackerk pts/tj Jun 16 16:46
rueggt pts/tm Jun 15 08:13
westbros pts/tn Jun 16 07:41
barnesc pts/to Jun 16 16:26
perkinsk pts/tq Jun 16 15:43
clarka pts/tr Jun 16 15:13
hammd pts/ts Jun 15 08:29
maloned pts/tt Jun 15 08:53
lanec pts/tv Jun 16 12:19
wessonc pts/tw Jun 16 07:59
daviss pts/tx Jun 16 16:57
sarverl pts/tA Jun 16 08:09
murleyt pts/tB Jun 16 08:09
mcgheeb pts/tC Jun 16 09:06
hoodm pts/tE Jun 16 08:19
keilersv pts/tG Jun 16 08:21
weatherb pts/tM Jun 16 10:00
swinneyl pts/tQ Jun 16 13:25
childree pts/tR Jun 16 08:47
mccalluj pts/tS Jun 15 10:17
edwardje pts/tW Jun 16 10:26
burrowsr pts/tY Jun 16 09:44
reedd pts/tZ Jun 16 13:44
andersor pts/t2 Jun 16 13:27
bradfork pts/t3 Jun 16 10:01
bakerl pts/t4 Jun 16 10:52
rueggt pts/t5 Jun 15 11:30
holleyj pts/t8 Jun 16 10:46
bassb pts/tcb Jun 16 13:41
gullettj pts/teb Jun 16 13:41
salchc pts/tfb Jun 16 14:18
Database Status
Informix Dynamic Server Version 7.31.UD6 -- On-Line -- Up 18 days 05:03:58 -- 415112 Kbytes
Userthreads
address flags sessid user tty wait tout locks nreads nwrites
d11cc018 ---P--D 1 root - 0 0 0 3553 25038
d11cc520 ---P--F 0 root - 0 0 0 0 514907
d11cca28 ---P--F 0 root - 0 0 0 0 389054
d11ccf30 ---P--F 0 root - 0 0 0 0 355801
d11cd438 ---P--F 0 root - 0 0 0 0 340360
d11cd940 ---P--F 0 root - 0 0 0 0 390411
d11cde48 ---P--F 0 root - 0 0 0 0 393684
d11ce350 ---P--F 0 root - 0 0 0 0 282223
d11ce858 ---P--F 0 root - 0 0 0 0 161051
d11ced60 ---P--- 14 root - 0 0 0 0 826
d11cf268 ---P--B 15 root - 0 0 0 1957 3608
d11cf770 Y--P--- 255079 carsu tg d68d3fb8 0 1 555 0
d11cfc78 Y--P--- 253225 prices tb d40d6c78 0 1 20 0
d11d0180 ---P--D 21 root - 0 0 0 0 0
d11d0688 Y--P--- 238661 carsu tm d3977460 0 1 217 0
<SNIP>
d5bd14d0 Y--P--- 242545 rueggt t5 d488f568 0 1 65 0
d5bd28f0 Y--P--- 256019 bakerl LINDABAK d636e7a8 0 1 1 12
d5bd3300 Y--P--- 268208 lanec tv d68d3cf8 0 1 1 0
d5bd3808 Y--P--- 263009 salchc - d3f124e8 0 1 3 0
d5bd5638 Y--P--- 257274 carsu t3 d309c7a8 0 1 715 0
d5bd6048 Y--P--- 237976 rueggt tm d3785bf0 0 1 483 16
d5bd7970 Y--P--- 263008 salchc - d40d7cf8 0 1 9 0
d5bd8888 Y--P--- 248626 smeetond - d2db01d0 0 1 87 0
d5bd97a0 Y--P--- 257618 solheim - d37853b0 0 1 531 196
d5be60e0 Y--P--- 268760 lanec CINDYCHI d5637568 0 1 46 0
189 active, 384 total, 380 maximum concurrent
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
62287276 64342324 177617272 64.93 4442472 8815359 86347487 94.86
isamtot open start read write rewrite delete commit rollbk
3757653969 65990726 307135134 2624681776 65889472 1811023 592554 576233 1865
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
1065 0 0 330189.94 14022.90 1317 3490
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
9859505 347 708350353 0 0 186 115633 8447945
ixda-RA idx-RA da-RA RA-pgsused lchwaits
16179567 1122539 29709044 46996663 30630949
Database Locks
--------- LISTLOCK: Lock Listing for cars database ---------
Table Level Type User Ses ID
---------------------- ------------- ------------------------ ---------- ------
cars.prog_enr_rec row:0x13f005 SR carsu 246346
cars.prog_enr_rec table intent-shared carsu 246346
cars.stufa_rec table intent-exclusive carsu 255078
cars.stufa_rec row:0x109107 update carsu 255078
cars.stufa_rec table intent-exclusive carsu 255076
cars.stufa_rec row:0xe4d01 update carsu 255076
cars.sysdistrib page:0x1300 intent-shared gullettj 267384
cars.sysdistrib page:0x1300 intent-shared edwardje 258099
cars.sysdistrib table intent-shared edwardje 258099
cars.sysdistrib table intent-shared carsu 255078
cars.sysdistrib page:0xc5400 intent-shared carsu 255078
cars.sysdistrib page:0xc5400 intent-shared carsu 255076
cars.sysdistrib table intent-shared keilersv 257987
cars.sysdistrib table intent-shared murleyt 255462
cars.sysdistrib table intent-shared barnesc 267796
cars.sysdistrib page:0xc5400 intent-shared burrowsr 267613
cars.sysdistrib page:0xc5400 intent-shared barnesc 267796
cars.sysdistrib table intent-shared gullettj 267384
cars.sysdistrib page:0x1300 intent-shared carsu 255076
cars.sysdistrib page:0x1300 intent-shared carsu 255078
cars.sysdistrib page:0x1300 intent-shared barnesc 267796
cars.sysdistrib page:0x1300 intent-shared burrowsr 267613
cars.sysdistrib table intent-shared burrowsr 267613
cars.sysdistrib page:0xc5400 intent-shared edwardje 258099
cars.sysdistrib table intent-shared murleyt 255462
cars.sysdistrib table intent-shared carsu 255076
cars.sysdistrib page:0xc5400 intent-shared keilersv 257987
cars.sysdistrib page:0xc5400 intent-shared murleyt 255462
cars.sysdistrib page:0x1300 intent-shared keilersv 257987
cars.sysdistrib table intent-shared burrowsr 256779
cars.sysdistrib page:0xc5400 intent-shared burrowsr 256779
cars.sysdistrib page:0xc5400 intent-shared gullettj 267384
cars.sysdistrib page:0x1300 intent-shared burrowsr 256779
---------------------- ------------- ------------------------ ---------- ------
33 Locks
Done.
