IDS Forum

Re: Performance Impact after storage migration

Posted By: Art Kagel
Date: Wednesday, 16 October 2013, at 7:05 a.m.

In Response To: Re: Performance Impact after storage migration (VIKAS HIVARKAR)

VIkas:

First, here's some of my onstat -g iof output:
> onstat -g iof

IBM Informix Dynamic Server Version 11.70.FC7 -- On-Line (Prim) -- Up 4
days 07:52:09 -- 6781312 Kbytes

AIO global files:
gfd pathname bytes read page reads bytes write page writes
io/s
3 rootdbs.1 169037824 82538 74778624 36513 915.3

op type count avg. time

seeks 0 N/A

reads 0 N/A

writes 7 0.0011

kaio_reads 16308 0.0012

kaio_writes 21607 0.0010

4 physlog.1 34816 17 1404880896 685977 513.4

op type count avg. time

seeks 0 N/A

reads 0 N/A

writes 0 N/A

kaio_reads 15 0.0016

kaio_writes 11901 0.0019

5 logicallog.1 4621144064 2256418 3662395392 1788279
1129.7

op type count avg. time

seeks 0 N/A

reads 0 N/A

writes 0 N/A

kaio_reads 432663 0.0005

kaio_writes 220925 0.0016

9 datadbs1.1 24896223232 12156387 1337100288 652880
478.4

op type count avg. time

seeks 0 N/A

reads 0 N/A

writes 1893 0.0149

kaio_reads 1071662 0.0008

kaio_writes 331983 0.0061

10 datadbs2.1 18248091648 8910256 53866496 26302
1179.4

op type count avg. time

seeks 0 N/A

reads 0 N/A

writes 0 N/A

kaio_reads 380858 0.0008

kaio_writes 7589 0.0013

This server uses RAW devices for chunks so KAIO is enabled. If you are
using cooked devices or filesystem files for chunks and have not enabled
DIRECT_IO then the data will appear in the seeks, reads, & writes lines
instead, but the interpretation is the same. The "avg. time" column
represents the service time for the device/file or how long it took, on
average, for a read or write (or seek - KAIO eliminates the seek step) to
return. This should be less than 20ms or 0.0200. If it gets higher than
that, then this device is being a bottleneck - however, ignore chunks that
do not have a significant number of IOs since their service times will be
skewed.

Other responses below:

I can continue to answer specific questions for you here, but you have a
complex environment so I don't know how much help I can be without getting
hands on with your system and monitor it over time to locate the
bottlenecks. I can do that if you need me to, but that would be a paid
consulting gig. If your company wants that kind of help, contact me
privately.

Art

Art S. Kagel, Principal Consultant

Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.

On Wed, Oct 16, 2013 at 3:26 AM, VIKAS HIVARKAR <vikas.hivarkar@gmail.com>wrote:

> Hello Art,
>
> We are still facing the timely issue of high response and not sure
> It is because of LRU flusing as comapred to Chunk writes OR
> It is because of the new disk setup we have implemented.
>

OK, so chunks writes versus lru writes is a thorny issue. Chunks writes
are more efficient for the system as a whole and for the SAN in particular
because they allow the OS's IO drivers and the SAN to coalesce multiple
contiguously placed writes into fewer larger IO operations and to do what
are known as elevator sorts to reorder IO operations to avoid random seeks
and minimize disk head location seeking. However, large chunk writes
during synchronous checkpoints (yes, the engine still does blocking
checkpoints under certain conditions) will cause serious performance
problems in an OLTP environment. So, LRU writes are not necessarily a bad
thing and if you are experiencing checkpoint blocking issues then you want
more LRU writes not fewer.

>
> HP team is working on the issue and as always the HP team says that you
> need to
> check the database bottleneck and the IBM AVL support says you need to
> check with HP :(
>
> I took up some of the you queries with the HP team and the response is as
>
> Why 10k rmp and not 15k disk was selected?
> They say the new disk has some kind of 2 GB SSD which will enhance the
> performance
> of disk and it is called as Adaptive optimization configuration, BTW they
> have
> not yet enabled this feature as we are having performance issues now.
>

OK! I just LOVE SAN geeks, they think that their technology is the end-all
and be-all of performance solutions. So, you have a hybrid SSD/spindle
system that stages high throughput files in the SSD drives for speed and
persists the data on the spindles for safety. Note that these HP SANs will
also magically relocate your data from volume group to volume group and
from RAID10 to RAID5 when it decides it needs to. I have another client
who bought into this line of BS and their IO performance on the new
super-SAN is not meeting their expectations. Just a warning. I've said it
elsewhere, SANs and big disks were the absolute worst thing to happen to
database systems. I can build a faster database server using many
singleton disks and many controllers than any SAN.

>
> Raid 1 Vs 5
>
> Yes, 6 DATABASE vgs are on RAID5 but rest all are on RAID 1
> I have also asked them to move these 6 to RAID 1 and not have anything
> releated to database on RAID 5
>

OK, good. Note again, that if this is the HP SAN system I'm thinking of,
it can reconfigure your RAID1/RAID10 drives as RAID5 if free space gets too
low!

>
> They do not understand my question on one single spindle being virtual
> redundancy and if it fails then everything fails
>

Is it RAID1 or RAID10? RAID1 has redundancy as does RAID10, but RAID10 has
better performance as it incorporates the ability of RAID0 to spread the
IOs over multiple RAID1 pairs.

>
> Until they break their heads I want to quicky check few things on informix
> side
> I have even rebooted the servers primary and sds just to check if it helps
>
> For sure chunck writes are better than LRU rights but after reboot I see
> more
> LRU Writes
>
> prod:/home/vikash $ onstat -F
>
> IBM Informix Dynamic Server Version 11.50.FC8W2 -- On-Line (Prim) -- Up
> 06:37:30 -- 83208912 Kbytes
>
> Fg Writes LRU Writes Chunk Writes
> 0 1647079 59511
>
> I am not sure if these intensive writing could be or problem, you
> recommended
> that 0.5 and 2 setting against our current 1 and 5, would it reduce the LRU
> rights
>

Reducing the lru_min/max_dirty settings will actually increase the LRU
writes and decrease the chunk writes. If you want to try moving more of
the IO to chunk writes during checkpoint time then you have to increase
these values instead.

>
> The onstat -g iof output is too big to post here can you help me understand
> what portion
> in that output will show me the delay in read/writes or a benchmark value
> below which
> we are in trouble
>
> sar -a
>
> HP-UX prodtu B.11.31 U ia64 10/16/13
>
> 00:00:00 iget/s namei/s dirbk/s
> 00:05:00 2 803 34
> 00:10:00 1 770 32
> 00:15:00 94 772 47
> 00:20:00 1 782 31
> 00:25:00 1 776 37
> 00:30:00 1 745 24
> 00:35:00 1 782 40
> 00:40:00 1 755 37
> 00:45:00 1 865 34
> 00:50:00 1 765 26
> 00:55:00 1 781 36
> 01:00:00 1 753 22
> 01:05:00 1 771 24
> 01:10:00 3 821 42
> 01:15:00 87 753 32
> 01:20:00 2 785 42
> 01:25:00 2 813 49
> 01:30:00 2 836 67
> 01:35:01 3 870 74
> 01:40:00 1 794 38
> 01:45:00 1 756 32
> 01:50:00 2 817 48
> 01:55:00 5 1238 50
> 02:00:00 1 750 20
> 02:05:00 1 775 24
> 02:10:00 1 741 22
> 02:15:00 99 743 21
> 02:20:00 2 740 26
> 02:25:00 1 762 24
> 02:30:00 1 731 20
> 02:35:00 6 941 51
> 02:40:00 9 1680 139
> 02:45:00 4 1006 222
> 02:50:00 8 1000 144
> 02:55:00 3 828 58
> 03:00:00 2 793 38
> 03:05:00 1 771 22
> 03:10:00 1 763 24
> 03:15:00 110 756 21
> 03:20:00 1 769 28
> 03:25:00 4 748 26
> 03:30:00 1 740 20
> 03:35:00 1 758 23
> 03:40:00 2 823 41
> 03:45:00 1 771 22
> 03:50:00 5 825 47
> 03:55:00 1 772 24
> 04:00:00 1 724 20
> 04:05:00 1 774 24
> 04:10:00 1 764 25
> 04:15:00 100 421 25
> 04:20:00 1 377 31
> 04:25:00 1 393 34
> 04:30:00 1 229 13
> 04:35:00 1 285 29
> 04:40:00 1 212 18
> 05:05:02 102 64 99
> 05:10:00 2 596 47
> 05:15:00 111 674 58
> 05:20:00 3 883 45
> 05:25:00 2 793 40
> 05:30:00 1 743 37
> 05:35:00 1 750 43
> 05:40:00 1 798 44
> 05:45:00 5 781 48
> 05:50:00 1 760 39
> 05:55:00 1 768 36
> 06:00:00 1 726 37
> 06:05:00 1 750 38
> 06:10:00 1 760 40
> 06:15:00 97 738 37
> 06:20:00 2 757 40
> 06:25:00 1 716 36
> 06:30:00 1 752 37
> 06:35:00 1 740 37
> 06:40:00 1 768 39
> 06:45:00 1 727 42
> 06:50:00 1 758 41
> 06:55:00 1 739 36
> 07:00:00 1 744 36
> 07:05:00 1 771 43
> 07:10:00 1 756 40
> 07:15:00 64 726 39
> 07:20:00 1 772 41
> 07:25:00 1 794 41
> 07:30:00 1 779 38
> 07:35:00 1 787 38
> 07:40:00 1 773 40
> 07:45:00 1 785 35
> 07:50:00 1 802 39
> 07:55:00 1 791 37
> 08:00:00 1 809 37
> 08:05:01 1 837 43
> 08:10:00 3 931 56
> 08:15:00 72 851 37
> 08:20:00 2 867 42
> 08:25:00 1 825 40
> 08:30:00 1 756 39
> 08:35:00 1 762 45
> 08:40:00 1 767 41
> 08:45:00 1 719 35
> 08:50:00 1 757 40
> 08:55:00 1 752 38
> 09:00:00 3 752 40
> 09:05:00 1 806 46
> 09:10:00 1 764 45
> 09:15:00 84 767 37
> 09:20:00 1 781 43
> 09:25:00 1 733 38
> 09:30:00 1 761 38
> 09:35:00 1 773 40
> 09:40:00 1 790 43
> 09:45:00 1 761 36
> 09:50:01 1 790 42
> 09:55:00 1 778 39
> 10:00:00 1 835 45
> 10:05:00 1 792 40
> 10:10:00 1 821 46
> 10:15:00 72 781 43
> 10:20:00 1 814 41
> 10:25:00 1 824 41
> 10:30:00 1 782 37
> 10:35:00 1 800 40
> 10:40:00 1 802 43
> 10:45:00 1 778 36
> 10:50:00 1 807 42
> 10:55:00 1 802 40
> 11:00:00 1 813 39
> 11:05:00 1 804 42
> 11:10:00 1 850 49
> 11:15:00 70 868 42
> 11:20:00 2 907 45
> 11:25:00 1 836 40
> 11:30:00 1 848 41
> 11:35:00 1 824 41
> 11:40:00 1 836 44
>
> Average 102 65 99
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>

--001a1133f726bdcd1504e8d9ac2e

Messages In This Thread

• Performance Impact after storage migration
  VIKAS HIVARKAR -- Wednesday, 9 October 2013, at 3:47 a.m.
  • Re: Performance Impact after storage migration
    Art Kagel -- Wednesday, 9 October 2013, at 6:45 a.m.
    • Re: Performance Impact after storage migration
      VIKAS HIVARKAR -- Friday, 11 October 2013, at 7:38 a.m.
      • Re: Performance Impact after storage migration
        Art Kagel -- Friday, 11 October 2013, at 10:26 a.m.
        • Re: Performance Impact after storage migration
          VIKAS HIVARKAR -- Wednesday, 16 October 2013, at 3:26 a.m.
          • Re: Performance Impact after storage migration
            Art Kagel -- Wednesday, 16 October 2013, at 7:05 a.m.
            • Re: Performance Impact after storage migration
              VIKAS HIVARKAR -- Friday, 18 October 2013, at 8:09 a.m.
              • Re: Performance Impact after storage migration
                Art Kagel -- Friday, 18 October 2013, at 8:12 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.