IDS Forum

Re: onstat -g rea almot 100 queue there !!!

Posted By: Art Kagel
Date: Thursday, 10 July 2008, at 12:47 a.m.

In Response To: Re: onstat -g rea almot 100 queue there !!! (MARS CHEN)

Mars,

See below:

On Wed, Jul 9, 2008 at 11:23 PM, MARS CHEN <mars@jsun.com> wrote:

> Thanks Art Kagel ...
>
> The following is what I get in the server :
>
> >- How many LRUs do you have configured?
> LRUS 8
> LRU_MAX_DIRTY 2
> LRU_MIN_DIRTY 1

OK, looks low, lets see what the metrics below say, read on...

>
>
> >- How much cache is configured?
> Sorry..I don't know what this means ... Buffers ?!

Yes, BUFFERS or bufferpool in the ONCONFIG file or 'Max # of Buffers in the
onmonitor utility.
So you have 100,000 buffers. Also sounds low. Continue...

>
> Physical Log Buffer Size [ 128] K
> Logical Log Buffer Size [ 128] K
> Max # of Logical Logs [ 200]
> Max # of Locks [ 2000000]
> Max # of Buffers [ 100000]
>
> Resident Shared Memory size [ 317542] Kbytes
>
> >- What is the current BR (Bufwaits Ratio)?
> >- What is the BTR (Buffer Turnover Rate)?
> >- What is the RAU (Read Ahead Utilization)?

These three metrics are derived from the onstat -p output see the formulae
and results below:

BR = (bufwaits / (pagreads + bufwrits)) * 100.00
values below 7 are good, 7-10 indicate some LRU or buffer contention and
means your server is slowing down, >10 is death mode, all of your users are
likely complaining about performance at this point.
BR = (104026 / (43972 + 11912950)) * 100.00 = 0.870 -- Looks good

BTR = ((pagreads + bufwrits) / BUFFERS) / (fractional hours since the stats
were last zero'd - or startup)
This is an estimate the number of times per hour that you are turning over,
or replacing, your entire buffer cache.
Values < 10 are OK, > 10 means you probably need more buffers. Keep in
mind, if your BTR is 10 you are replacing the entire contents of your buffer
cache every 6 minutes or you are seriously thrashing a subset of the cache
far mroe frequently than that.
You did not post your onstat output heading lines, so I don't know when your
server was bounced last or if you've run onstat -z to reset the statistics
since startup, so I cannot calculate this one for you. You'll have to do
that yourself. But here's what I do have:
BTR = ((43972 + 11912950) / 100000) / ??? = 119.569 / ??? -- So, you've
turned this cache over almost 120 times since the stats were last zero'd out
(maybe at startup?). If that was fewer than 11.9 hours prior to running
this onstat -p report, you have too few buffers.

RAU = (RA-pgsused / (ixda-RA + idx-RA + da-RA)) * 100.0
This indicates the percentage of read ahead that you are actually using. It
should be as close as possible to 100.00, even 99.5 is suspect if the
absolute number of RA pages represented by that 1/2% represents a
significant portion of your buffer cache.
RAU = (29045 / (1327 + 2753 + 25041)) * 100.0 = 99.73 -- Not too bad for
the default settings and the absolute numbers are low.

>
> post onstat -p :
>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 36850 43972 3714504778 100.00 103209 346874 11912950 99.13
>
> usercpu syscpu numckpts flushes
> 47787.69 5838.60 427 856
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> 104026 4253 981169431 0 0 1301 563 6533000
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 1327 2753 25041 29045 16257541
>
> And ... I didnot set these 2 fields about Read Ahead ~~
>
> Num of Read Ahead Pages [ ]
> Read Ahead Threshold [ ]

So these are defaulting to 8 & 4 respectively. Probably no ideal, but not
excessive.

>
>
> >- How many IO operations per second are being thrown at your disk
> > subsystems per chunk (onstat -g iof)?
> >- Can the 'disk' and channel handle that level of IO?
> >- Are you beating hell out of one structure while other disk spindles are
> > quiescent much of the time?
> >- Are you using RAW disk or at least DIRECT_IO?
> >- Have you avoided RAID5 like the plague that it is?
>
> onstat -g iof showes :
> AIO global files:
> gfd pathname totalops dskread dskwrite io/s
> 3 raw1 8629 38 8591 0.1
> 4 raw2 49352 12930 36422 0.8
> 5 raw3 1 1 0 0.0
> 6 raw4 1 1 0 0.0
> 7 raw5 1 1 0 0.0
>
> onstat -g iov showes :
> class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors
> msc 0 i 0.1 9225 0 0 0 9226 1.0 0
> aio 0 i 6.5 417654 164344 156802 0 413630 1.0 0
> aio 1 i 0.1 9496 2880 5630 0 6529 1.5 0
> aio 2 i 0.1 4553 575 3886 0 1364 3.3 0
> aio 3 i 0.1 4106 266 3767 0 989 4.2 0
> aio 4 i 0.1 3909 174 3668 0 913 4.3 0
> aio 5 i 0.1 3807 132 3626 0 834 4.6 0
> aio 6 i 0.1 3760 129 3577 0 808 4.7 0
> aio 7 i 0.1 3660 96 3509 0 799 4.6 0
> aio 8 i 0.1 3668 105 3514 0 788 4.7 0
> aio 9 i 0.1 3710 81 3576 0 777 4.8 0
> pio 0 i 0.0 538 0 538 0 539 1.0 0
> lio 0 i 0.1 5782 0 5782 0 5783 1.0 0

It looks like either you have KAIO turned off, or all of your chunks are
COOKED files or Block devices. That means that your AIO VPs are doing all
of the IO work and there are not enough of them. All of you AIO VPs show
from 1.0 to 4.8 IOs per wake up (io/wup). You should have at least one of
these with an io/wup that shows less than 1.0. I'm estimating that you need
to increase from 10 to 20 AIO VPs or to move to RAW chunks an KAIO. Right
now, on average, 4 out of 5 IO requests is waiting at least some time before
being serviced.

>
>
> onstat -g ioq showes :
> AIO I/O queues:
> q name/id len maxlen totalops dskread dskwrite dskcopy
> adt 0 0 0 0 0 0 0
> msc 0 0 1 9291 0 0 0
> aio 0 0 8 101442 2994 2 0
> pio 0 0 1 541 0 541 0
> lio 0 0 1 5817 0 5817 0
> gfd 3 0 3 3190 38 3152 0
> gfd 4 0 150 136343 36844 99499 0

This device, gfd4, has a max wait queue of 150 requests. For whatever
reason (aio vps are busy, the disk or controller is slow, there is
contention for the disk or controller with other applications sharing the
resource) this device is slow.

>
> gfd 5 0 1 1 1 0 0
> gfd 6 0 1 1 1 0 0
> gfd 7 0 1 1 1 0 0
> gfd 9 0 1 16 8 8 0
> gfd 11 0 1 10 2 8 0
>
> And I use raw device as my chunk :

Just because you named them 'raw1' etc. doesn't make them RAW. Are these
Character devices or Block devices? Only Character devices are RAW,
filesystem files and block devices are all COOKED. If you are running on
Linux or a later version of Solaris you can turn on O_DIRECT IO which can
use KAIO mode IO which is faster than using the AIO VPs to simulate Kernel
IO. Right now you are using AIO VPs which can ONLY happen if the chunks are
COOKED or if you set KAIOOFF=1 in the server's environment (or if you are
running under HPUX you did not set KAIOON=1 in the environment - HPUX is the
ONLY platform that defaults to KAOOFF). Sounds like you are running on
Linux. You can turn on O_DIRECT IO by adding the parameter DIRECT_IO 1 into
your ONCONFIG file.

>
> Chunks
> address chk/dbs offset size free bpages flags pathname
> 5768e210 1 1 0 900000 666899 PO- /opt/informix/raw1
> 5768ea90 2 2 0 900000 529275 PO- /opt/informix/raw2
> 5768eb90 3 2 0 900000 899997 PO- /opt/informix/raw3
> 5768ec90 4 2 0 900000 899997 PO- /opt/informix/raw4
> 5768ed90 5 2 0 900000 899997 PO- /opt/informix/raw5
>
> I noticed that chuck 2 contained all the tables since chunk 3,4,5 are
> empty,
> so I just create those busy tables to chunk 3,4,5 will make I/o more
> smooth,
> I think ~~~

That will certainly help reduce the IO bottleneck I reported above and
reduce the wait queue on chunk 2.

>
>
> My cpu information : /proc/cpuinfo
> I list processor 0 , but it is 0 ~ 7 ... to save the time , list processor
> 0 :
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 15
> model : 4
> model name : Intel(R) Xeon(TM) MP CPU 3.66GHz
> stepping : 1
> cpu MHz : 3657.666
> cache size : 1024 KB
> physical id : 0
> siblings : 2
> runqueue : 0
> fdiv_bug : no
> hlt_bug : no
> f00f_bug : no
> coma_bug : no
> fpu : yes
> fpu_exception : yes
> cpuid level : 5
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
> pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm nx lm
> bogomips : 7287.60
>
> And for RAID ? ... I try to post /dmesg , I don't know whether it is raid
> 5?!

<SNIP>

> Vendor: ESG-SHV Model: SCA HSBP M35 Rev: 1.04
> Type: Processor ANSI SCSI revision: 02
> blk: queue f7230a18, I/O limit 4095Mb (mask 0xffffffff)
> Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
> SCSI device sda: 143372288 512-byte hdwr sectors (73407 MB)
> Partition check:
> sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10 sda11 sda12 sda13
> sda14 sda15 >

<SNIP>
Looks like your startup found only a single spindle, no RAID setup at all
that I can see, and, as I predicted, the database server is sharing that one
spindle with the Operating System and all of your other processes.

Art

--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)

Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do those
opinions reflect those of other individuals affiliated with any entity with
which I am affiliated nor those of the entities themselves.

Messages In This Thread

• onstat -g rea almot 100 queue there !!!
  MARS CHEN -- Wednesday, 9 July 2008, at 9:10 p.m.
  • Re: onstat -g rea almot 100 queue there !!!
    Art Kagel -- Wednesday, 9 July 2008, at 9:56 p.m.
    • Re: onstat -g rea almot 100 queue there !!!
      MARS CHEN -- Wednesday, 9 July 2008, at 11:23 p.m.
      • Re: onstat -g rea almot 100 queue there !!!
        MARS CHEN -- Thursday, 10 July 2008, at 12:00 a.m.
      • Re: onstat -g rea almot 100 queue there !!!
        Art Kagel -- Thursday, 10 July 2008, at 12:47 a.m.
        • Re: onstat -g rea almot 100 queue there !!!
          MARS CHEN -- Thursday, 10 July 2008, at 2:28 a.m.
          • Re: onstat -g rea almot 100 queue there !!!
            MARS CHEN -- Thursday, 10 July 2008, at 3:33 a.m.
            • Re: onstat -g rea almot 100 queue there !!!
              Art Kagel -- Thursday, 10 July 2008, at 3:45 a.m.
              • Re: onstat -g rea almot 100 queue there !!!
                MARS CHEN -- Thursday, 10 July 2008, at 3:58 a.m.
                • Re: onstat -g rea almot 100 queue there !!!
                  Art Kagel -- Thursday, 10 July 2008, at 10:47 a.m.
                  • Re: onstat -g rea almot 100 queue there !!!
                    MARS CHEN -- Thursday, 10 July 2008, at 9:35 p.m.
          • Re: onstat -g rea almot 100 queue there !!!
            Art Kagel -- Thursday, 10 July 2008, at 3:44 a.m.
            • Re: onstat -g rea almot 100 queue there !!!
              MARS CHEN -- Thursday, 10 July 2008, at 4:07 a.m.
              • Re: onstat -g rea almot 100 queue there !!!
                Art Kagel -- Thursday, 10 July 2008, at 11:01 a.m.
                • Re: onstat -g rea almot 100 queue there !!!
                  jack.parker4@verizon.net -- Thursday, 10 July 2008, at 11:12 a.m.
                  • Re: onstat -g rea almot 100 queue there !!!
                    MARS CHEN -- Thursday, 10 July 2008, at 9:04 p.m.
                • RE: onstat -g rea almot 100 queue there !!!
                  Penza Kenneth at MITTS -- Friday, 11 July 2008, at 7:36 a.m.
                  • RE: onstat -g rea almot 100 queue there !!!
                    John Whittenberger -- Friday, 11 July 2008, at 11:32 a.m.
                    • Re: RE: onstat -g rea almot 100 queue there !!!
                      MARS CHEN -- Sunday, 13 July 2008, at 10:39 p.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.