 |
IDS Forum
Re: A question of load . . .
Posted By: Chris Salch
Date: Thursday, 31 August 2006, at 2:36 p.m.
In Response To: Re: A question of load . . . (Eric Rowell)
( Unfortunatly, none of us thought to save any of that data when things
started slowing down, so all we've got is what we can remember. )
In answer to several:
3 oninit processes were the only things really eating cpu time. There
were a few other processes that would show up using a small amount of
cpu time but nothing excessive. We were getting a load average just
topping 2 at the time ( this has happened before without any significant
impact to the system )
iostat was showing about 40kbps - 60kbps across all the drives,
significantly lower than what I see now or even after everything calmed
back down and our scheduled backup was started.
Here's an onstat from shortly after if it means much, (The backup
process was started after the whole mess).
IBM Informix Dynamic Server Version 9.40.HC5 -- On-Line -- Up 13
days 09:21:20 -- 426136 Kbytes
Userthreads
address flags sessid user tty wait tout locks nreads
nwrites
d2767018 ---P--D 1 root - 0 0 0 633
4202
d276763c ---P--F 0 root - 0 0 0 0
77695
d2767c60 ---P--F 0 root - 0 0 0 0
11181
d2768284 ---P--F 0 root - 0 0 0 0
7118
d27688a8 ---P--F 0 root - 0 0 0 0
18300
d2768ecc ---P--F 0 root - 0 0 0 0
9348
d27694f0 ---P--F 0 root - 0 0 0 0
7618
d2769b14 ---P--F 0 root - 0 0 0 0
6462
d276a138 ---P--F 0 root - 0 0 0 0
4758
d276a75c ---P--- 13 root - 0 0 0 0
24
d276ad80 ---P--B 14 root - 0 0 0 65074
480
d276b9c8 Y--P--- 352236 carsu tC d8375840 0 1 1019
604
d276bfec ---P--D 17 root - 0 0 0 0
0
d276c610 Y--P--D 26 root - c4fd3b6c 0 0 0
0
d276cc34 Y--P--- 27 roedelm ASTRASER d3201420 0 1 0
0
d276d258 Y--P--- 28 roedelm ASTRASER d30a6358 0 1 0
0
d276d87c Y--P--- 384521 zijlstre tg d9e4c658 0 1 9
0
d276dea0 Y--P--- 390938 carsu tO d7df4560 0 4 0
0
d276eae8 Y--P--- 392296 erpeldic JOHN d70e6b20 0 1 0
0
d276fd54 Y--P--- 389877 carsu tkb d4f0b150 0 1 132
170
d2770378 Y--P--- 386225 sgps JOHN d9961308 0 1 1
0
d277099c ---PR-- 392439 academus - 0 0 1 10
0
d2770fc0 Y--P--- 391698 gullettj tnb d9e3a278 0 1 30
0
d27715e4 Y--P--- 392236 bakerl LINDABAK d53d67f8 0 1 0
0
d2771c08 Y--P--- 392438 academus - d502aa20 0 1 0
0
d277222c Y--P--- 26513 academus - d4865868 0 1 0
0
d2772850 Y--P--- 392379 sgps JOHN d9bcf078 0 1 0
0
d2772e74 Y--P--- 344045 carsu tG d71e18b8 0 1 8179
0
d2773498 Y--P--- 381962 trogdond t3 d43afe78 0 1 1
0
d2774704 Y--P--- 383934 hoffpowj teb d7e04de0 0 1 10
0
d2774d28 Y--P--- 387046 bakerl LINDABAK d600ad58 0 1 1
0
d277534c Y--P--- 389228 burrowsr t0 d7332a28 0 1 104
0
d2775970 Y--P--- 343854 murleyt TOBIMURL d35e5818 0 1 319
36
d2775f94 Y--P--- 390673 childree tI d8d7adf8 0 1 32
0
d2776bdc Y--P--- 352174 sackettp PAMSACKE d78af750 0 1 173
22
d2777200 Y--P--- 385510 zijlstre tg dc89a9b0 0 2 134
0
d2777824 Y--P--- 392149 sgps JOHN d93b61f8 0 1 0
0
d2777e48 Y--P--- 344941 holleyc DAL-REP1 d5aad900 0 1 581
46
d277846c Y--P--- 773 carsu tf d45ae1b0 0 1 0
0
d27790b4 Y--P--- 344192 carsu tt d74e7bc8 0 1 1795
0
d27796d8 Y--P--- 375850 hyattr RENEEHYA d48798b8 0 1 1548
54
d277b58c Y--P--- 357910 hueyt TERESAHU d3b50968 0 1 1467
134
d277bbb0 Y--P--- 387128 carsu tl d66aef80 0 1 225
0
d277c1d4 Y--P--- 390897 holleyc DAL-REP1 d75015b0 0 1 1
0
d277da64 Y--P--- 343646 vestalt tq d4df9da8 0 1 2
0
d277e6ac Y--P--- 155417 carsu tD d59b3a20 0 1 0
0
d277f2f4 Y--P--- 343715 sarverl tu d4331c58 0 1 1221
0
d277ff3c Y--P--- 392116 sgps JOHN d761ce30 0 1 0
0
d2780560 Y--P--- 343793 cowartj JULIANCO d39af2b8 0 1 1494
722
d2781df0 Y--P--- 392277 smeetond - d64dec78 0 1 0
0
d2782a38 ---PR-- 384779 wolffd - 0 0 3 16752
7880
d278305c Y--P--- 390841 johnsonh tO d548c7c0 0 1 3
0
d27842c8 Y--P--- 390978 carsu t8 d3682c60 0 1 0
0
d27848ec Y--P--- 390741 carsu t8 d8b67fa8 0 1 157
0
d2785b58 ------- 391626 informix tty2a11 0 0 0 312628
0
d278617c Y--P--- 382952 carsu tk d91fb7d8 0 1 251
0
d27867a0 Y--P--- 252531 carsu t4 d9534b28 0 2 0
0
d2786dc4 Y--P--- 376405 gullettj tnb d70c1558 0 1 7
0
d27873e8 Y--P--- 381469 wolffd - d600acf8 0 1 2
0
d2787a0c Y--P--- 353053 meltonc t9 d7057fb8 0 1 1
0
d2788654 Y--P--- 392449 carsu td d478b828 0 1 869
0
d278929c Y--P--- 392295 erpeldic JOHN d66c4c78 0 1 0
0
d278a508 Y--P--- 390298 carsu tk d8b814f8 0 1 2738
0
d278ab2c Y--P--- 346340 wessonc tl dc89aad0 0 1 132
0
d278b150 Y--P--- 389623 sgps JOHN d39afb58 0 1 0
0
d278b774 Y--P--- 392066 keilersv ti d4e43ca0 0 1 4
0
d278bd98 Y--P--- 387915 carsu teb d91fbea0 0 1 1195
0
d278d004 Y--P--- 348675 bradfork tgb d70c17b8 0 1 3
0
d278dc4c Y--P--- 344032 pattersk tG d9be5680 0 1 2
0
d278eeb8 Y--P--- 344035 carsu tG d7123220 0 1 13
0
d2790124 Y--P--- 390760 carsu t8 d4c13748 0 1 3255
0
d2790748 Y--P--- 388516 carsu tnb d676d478 0 1 2992
0
d27919b4 Y--P--- 384549 carsu tg d3255478 0 4 6536
40
d2791fd8 Y--P--- 344477 bakerl tR da0f4e88 0 1 10
0
d27925fc Y--P--- 343677 vestalt tq d42de718 0 2 2182
0
d2792c20 Y--P--- 390425 sgps JOHN d4f87150 0 1 0
0
d2793244 Y--P--- 391011 suarezj - d94736f8 0 2 251
4
d27944b0 Y--P--- 117944 carsu th d4787428 0 1 0
0
d279571c Y--P--- 344043 carsu tG d73f1bd0 0 1 10003
634
d2796364 Y--P--- 385067 kunderp tv d4e31d40 0 1 7
0
d2796988 Y--P--- 392168 bakerl LINDABAK d3b64c98 0 1 11
0
d2796fac Y--P--- 745 hammd tf d5adaf68 0 1 0
0
d27975d0 Y--P--- 746 hammd tf d5af5ca0 0 1 0
0
d5c30ecc Y--P--- 392385 carsu tg d9f50e18 0 1 167
0
d5c31b14 Y--P--- 344484 carsu tR d9b80588 0 1 4042
58
d5c32138 Y--P--- 383938 carsu teb d4f872e8 0 4 6761
0
d5c3275c Y--P--- 392440 academus - d9961018 0 1 0
0
d5c339c8 Y--P--- 346791 mcgheeb tbb d70c1cb0 0 1 5
0
d5c33fec Y--P--- 384998 carsu teb d4df9558 0 1 0
0
d5c34c34 Y--P--- 376219 carsu tkb d8237630 0 1 9807
0
d5c35258 Y--P--- 390653 childree tI d77f2f98 0 1 0
0
d5c35ea0 Y--P--- 344041 carsu tG d624f140 0 1 7429
4780
d5c36ae8 ---P--- 392464 roedelm - 0 0 1 0
0
d5c3710c Y--P--- 792 carsu tf d4726018 0 1 0
0
d5c37730 Y--P--- 390893 ardend tJ d8b281b0 0 1 24
0
d5c38378 Y--P--- 390057 barnesc t8 d7d89930 0 1 1
0
d5c3899c Y--P--- 122431 jonesl tp d31a08d8 0 2 0
0
d5c38fc0 Y--P--- 346800 mcgheeb tbb d8ee0210 0 2 2471
140
d5c395e4 Y--P--- 392245 bakerl LINDABAK d93b6c48 0 1 2
0
d5c3a22c Y--P--- 348610 carsu tab d7e08d80 0 1 477
0
d5c3ae74 Y--P--- 390740 jonesmi MICHAELJ d4589a70 0 1 1
0
d5c3c704 Y--P--- 346765 hallv tab d93b6ac8 0 1 6
0
d5c3d34c ------- 391626 informix tty2a11 0 0 0 0
1472
d5c3e5b8 Y--P--- 389097 hueyt tvb d9b80078 0 2 112
2
d5c3ebdc Y--P--- 363978 carsu tgb d8d7a858 0 1 380
80
d5c3f824 Y--P--- 389756 jordanm MONIQUEJ d7e08f30 0 1 509
16
d5c40a90 Y--P--- 353058 meltonc t9 d4682d58 0 1 463
2
d5c410b4 Y--P--- 391922 childree tI d556d078 0 2 89
0
d5c416d8 Y--P--- 384710 sgps JOHN d39af3d8 0 1 0
0
d5c41cfc Y--P--- 376203 bullardk tkb d645f9b0 0 1 3
0
d5c42320 Y--P--- 343654 carsu tq d895c9c0 0 1 2605
0
d5c42944 Y--P--- 344199 carsu tt dc913d78 0 1 4232
0
d5c42f68 Y--P--- 391511 keilersv VIKKIKEI d94941b0 0 1 125
0
d5c43bb0 Y--P--- 390770 hornt TERESAHO d47e4d30 0 1 169
0
d5c447f8 Y--P--- 384926 carsu tg d35e56f8 0 1 0
0
d5c44e1c Y--P--- 389012 zijlstre tg d600fe20 0 1 11
0
d5c45440 Y--P--- 2608 academus - d45e64c0 0 1 0
0
d5c45a64 Y--P--- 344478 bakerl tR d5b3cd68 0 1 3127
186
d5c46088 Y--P--- 392300 erpeldic JOHN d8d7a918 0 1 0
0
d5c47918 Y--P--- 348782 carsu tgb d7e16af0 0 1 462
0
d5c47f3c Y--P--- 387612 carsu tpb d45e6520 0 1 0
0
d5c491a8 Y--P--- 392297 erpeldic JOHN d7d894f8 0 1 0
0
d5c49df0 Y--P--- 385072 carsu tv d4c1b490 0 1 12641
0
d5c4aa38 Y--P--- 343785 barnesc tx d7e089e8 0 1 2
0
d5c4b680 Y--P--- 382716 dennisg GINGERDE d89e8478 0 1 1591
6
d5c4c8ec Y-AP--M 391626 informix tty2a11 d5ff76c8 0 0 23
0
d5c4cf10 Y--P--- 348250 rueggt tfb d4535958 0 1 1
0
d5c4d534 Y--P--- 224913 carsu to d7e08ba0 0 1 0
0
d5c4e17c Y--P--- 384923 carsu tg d849be48 0 1 1096
0
d5c4e7a0 Y--P--- 69236 dennisg tc d73f1918 0 1 0
0
d5c4edc4 Y--P--- 392414 powellc t1 d9bea968 0 1 0
0
d5c4fa0c Y--P--- 389749 murleyt twb d4f9ed90 0 2 88
0
d5c518c0 Y--P--- 391759 lanec CINDYCHI d7e04e40 0 1 77
4
d5c51ee4 Y--P--- 381913 carsu t3 d64de2f0 0 1 194
0
d5c52508 Y--P--- 392194 bakerl LINDABAK d8375b60 0 1 0
0
d5c52b2c Y--P--- 344229 murleyt tt d981ba80 0 2 1823
0
d5c53150 Y--P--- 390983 carsu tO d4bc7b70 0 1 0
0
d5c53774 Y--P--- 390859 carsu tO d4a784f0 0 1 488
34
d5c549e0 Y--P--- 391152 carsu t0 d6022498 0 1 338
0
d5c55004 Y--P--- 392249 jordanm MONIQUEJ d5601150 0 1 158
0
d5c55c4c Y--P--- 391233 carsu t8 d685a8d8 0 1 663
0
d5c56894 Y--P--- 344027 sgps JOHN d478b5e8 0 1 1
0
d5c56eb8 Y--P--- 389009 burrowsr t0 d5123f10 0 1 0
0
d5c57b00 Y--P--- 344447 bakerl LINDABAK d4331d78 0 1 987
20
d5c58124 Y--P--- 392202 bakerl LINDABAK d9b993a8 0 1 0
0
d5c58748 Y--P--- 390958 carsu tO dc913d18 0 1 0
0
d5c5a5fc Y--P--- 390985 carsu tk d70e6210 0 1 107
0
d5c5b244 Y--P--- 382060 carsu ty d9e3a1b8 0 1 373
40
d5c5b868 Y--P--- 382092 carsu ty d7332c38 0 1 1164
0
d5c5be8c Y--P--- 343840 barnesc tx d93b62b8 0 1 952
0
d5c5c4b0 Y--P--- 142454 jonesl LIB1-DAT d94e3908 0 1 0
0
d5c5dd40 Y--P--- 392355 hyattr RENEEHYA d7770ef8 0 1 18
0
d5c5e988 Y--P--- 343720 carsu tu d82f1d98 0 1 2627
0
d5c5efac Y--P--- 344036 carsu tG d39af5b8 0 1 408
0
d5c5fbf4 Y--P--- 381482 wolffd - d84d1fb8 0 2 84
0
d88f963c Y--P--- 390702 jonesmi MICHAELJ d6b965f8 0 1 0
0
d88f9c60 Y--P--- 392388 vestalt TINAVEST d9ebf018 0 1 96
2
d88fa284 Y--P--- 346114 halll LH-132-X d32bf8a0 0 1 1179
0
d88faecc Y--P--- 391419 keilersv VIKKIKEI d7eff078 0 1 53764
0
d88fc138 Y--P--- 343657 carsu tq d600ac98 0 1 3833
0
d88fc75c Y--P--- 392289 erpeldic JOHN d93b6498 0 1 0
0
d88fec34 Y--P--- 392291 erpeldic JOHN d446c428 0 1 0
0
d88ff87c Y--P--- 382045 erpeldic ty d66ae0e8 0 1 4
0
d88ffea0 Y--P--- 354348 jonesmi MICHAELJ d93b66d8 0 1 544
30
d8900ae8 Y--P--- 348238 bradfork KENDRABR d4b6ebf8 0 1 604
24
d8901d54 Y--P--- 342633 rueggt TEXASRUE d86800d8 0 1 1003
0
d8902378 Y--P--- 348290 carsu tfb d478bf48 0 1 2914
16
d890299c Y--P--- 117940 carsu th d7e16e08 0 1 0
0
d8902fc0 Y--P--- 392239 solheim - d690b660 0 1 32
38
d8903c08 Y--P--- 363983 carsu tgb d7a087c0 0 1 0
0
d890422c Y--P--- 387605 erpeldic tpb d917aab8 0 1 1
0
d8904850 Y--P--- 385970 carsu tg d39b09b8 0 1 981
0
d8905498 Y--P--- 344470 bakerl LINDABAK d8d7a198 0 1 106
0
d8905abc Y--P--- 389019 carsu t0 d9ebf0d8 0 1 3173
0
d8906704 Y--P--- 392285 erpeldic JOHN d8795be0 0 1 0
0
d8906d28 Y--P--- 384990 carsu teb d9e3b748 0 1 211
0
d890734c Y--P--- 385324 carsu tbb d53606d0 0 1 118
0
d8907f94 Y--P--- 392390 vestalt TINAVEST d4cfff78 0 1 1
0
d8909824 Y--P--- 390056 barnesc t8 d89e8670 0 2 1101
0
d8909e48 Y--P--- 378057 carsu tC d43afed8 0 1 485
162
d890aa90 Y--P--- 392428 wadea td d31cda18 0 1 1
0
d890b6d8 Y--P--- 344495 carsu tR d478bd08 0 1 451
44
d890bcfc Y--P--- 390522 jonesmi MICHAELJ d5ad5f08 0 1 2490
0
d890d58c Y--P--- 356864 carsu tfb d55abba8 0 1 0
0
d890dbb0 Y--P--- 389234 carsu tkb d5ff7668 0 1 734
78
d890e1d4 Y--P--- 356863 carsu tfb d8d7abb8 0 1 239
88
d890ee1c Y--P--- 391006 moughond t6 d818cda8 0 1 17
0
d890f440 Y--P--- 382075 carsu ty d9e4ce90 0 1 0
0
d890fa64 Y--P--- 392403 academus - d9e4c150 0 1 0
0
d89106ac Y--P--- 389226 sackettp PAMSACKE d6d83350 0 1 2
0
d8910cd0 Y--P--- 391242 ardend tB d4a9f4f0 0 1 12
0
d8911918 Y--P--- 389745 murleyt twb d556d2c8 0 1 1
0
d8912560 Y--P--- 352242 mcgheeb BRENDAMC d676d220 0 1 23
14
d8914414 Y--P--- 362627 hornt TERESAHO d66c0ec0 0 1 388
16
d891505c Y--P--- 388506 carsu tnb d464d448 0 1 138
0
d8915680 Y--P--- 344042 carsu tG d68fcdf0 0 1 0
0
d8915ca4 Y--P--- 392375 dennisg tc d478b708 0 1 151
0
d89168ec Y--P--- 364227 rueggt tfb d685a878 0 1 182
36
d8917b58 Y--P--- 391253 holleyc DAL-REP1 d94534e0 0 1 1
0
d89187a0 Y--P--- 392427 academus - d4914688 0 1 0
0
d891a654 Y--P--- 359093 hallv tab d5360520 0 2 162
0
d891b29c Y--P--- 389017 carsu t0 d5c80760 0 1 401
0
d891c508 Y--P--- 389064 sackettp PAMSACKE d78cdf98 0 1 0
0
d891cb2c Y--P--- 389214 sackettp PAMSACKE d93b6b28 0 1 0
0
d891dd98 Y--P--- 363671 erpeldic JOHN d9e3bb28 0 1 228
20
d891e3bc Y--P--- 382875 fordjean tk d73a29c8 0 1 4
0
d891f004 Y--P--- 389649 sgps JOHN d3408370 0 1 0
0
d891f628 Y--P--- 343833 ardend DEADREAR d70c14f8 0 1 673
30
d891fc4c Y--P--- 390864 carsu t8 d8b81078 0 1 697
0
d89214dc Y--P--- 386721 carsu tl d9e3b868 0 1 296
0
d8923fd8 Y--P--- 391237 ardend tB d375bec8 0 1 0
0
d89245fc Y--P--- 391000 suarezj - d4225df8 0 1 26
0
212 active, 384 total, 371 maximum concurrent
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
9072698 7908428 2698037162 99.66 321497 1980538 35477827 99.09
isamtot open start read write rewrite delete commit
rollbk
1076597400 15853295 79159375 780472623 26762946 51068 53804 39526
155
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 98471.70 2583.33 127 427
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
1443280 28 2981507560 0 0 110 89461
548761
ixda-RA idx-RA da-RA RA-pgsused lchwaits
2346196 86301 4646343 7071512 16960
On Thu, 2006-08-31 at 13:32 -0400, Eric Rowell wrote:
> I would have to say without information from tools like "top", "ps",
> vmstat, and other OS related tools as well as an array of Informix
> onstats I wouldn't even know where to offer a suggestion.
>
> Because of the same type of issue in the past we have made a habit
> (for good or bad) to take an onstat -o filename.out. This allows us
> to go back to that point in time to look at informix at that time.
> Note: this requires disk space as large as the memory foot print of
> ids. We have a space just to save a bunch of these in times of need.
> Luckily we don't need them often.
>
> I would suggest looking for orphaned or runaway processes... Anything
> accumulating CPU time beyond what is normal... Ask your UNIX SA to
> look for this on regular bases... "ps -ef" Also watch for processes
> that have a really low priority... Not sure about HPUX but normally
> low means more resources to be given.
>
> But it sounds like you have covered most of the bases
>
> On 8/31/06, CHRIS SALCH <chrissalch@letu.edu> wrote:
> >
> > We have an HP9000/L2000 with 4 64bit PA-RISC chips running at 440Mhz and
> > 5 Gigs of ram. We're running Inforimx 9.40.HC5 on HPUX 11i. We ran
> > into an issue on Monday where our database engine seemed to grind to a
> > halt. The machine itself responded reasonably to everything but
> > database queries. It seemed like anything having to do with extracting
> > or querying data in the database took exceptionally long to respond.
> > Interacting with onstat did not. Unfortunately, no one thought to save
> > off a copy of what onstat dumped during the incident.
> >
> > Our application is a touch heterogeneous, we have everything from cognos
> > impromptu, MS Access "applications", a massive 4gl based application
> > using shm to connect, to perl dbi based connections running off the same
> > database. The majority of users probably connect using shm connections
> > through the 4gl application but, MS Access makes an excessive number of
> > connections per user of a given application. Everything but the MS
> > Access and Congos Impromptu reports/applications are running from the
> > same machine as the engine. This machine also has an apache web server
> > to handle some cgi scripts, mostly perl code and the source of most of
> > our perl dbi connections.
> >
> > When our problem occured, we had a reasonably heavy load from all
> > around. The machine had approximatly 560 or so proccess running total,
> > some 300-400 connections appearing in onstat at any one instant, and
> > three oninit processes eating up 90% of the cpu time solid for about an
> > hour. Simple queries would take 2 to 3 minutes to complete. There
> > were no apparent harware problems with the equipment, no excessive
> > numbers of locks or full logical logs, no single process or small group
> > of processes that looked like they were doing anything out of the
> > ordinary.
> >
> > Fortunately, everything seemed to calm down again about an hour latter.
> > As a note, that day represented an exceptionally high load in comparison
> > to our normal operations and this could all be related to having maxed
> > out what our hardware could handle.
> >
> > Is there any tuning that might be suggested for our system? I've
> > attached a copy of our config file. ( I've sense increased the frequency of
> my
> > logging cron job )
> >
snip
> >
> *******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
Messages In This Thread
- A question of load . . .
CHRIS SALCH -- Thursday, 31 August 2006, at 1:17 p.m.
- RE: A question of load . . .
Jack Parker -- Thursday, 31 August 2006, at 1:25 p.m.
- Re: A question of load . . .
Eric Rowell -- Thursday, 31 August 2006, at 1:32 p.m.
- Re: A question of load . . .
Chris Salch -- Thursday, 31 August 2006, at 2:36 p.m.
- Re: A question of load . . .
ART KAGEL, BLOOMBERG/ 731 LEXIN -- Thursday, 31 August 2006, at 2:42 p.m.
- Re: A question of load . . .
Norberto Valverde LLanos -- Thursday, 31 August 2006, at 3:37 p.m.
- Re: A question of load . . .
Dimitar Bachvarov -- Friday, 1 September 2006, at 6:25 a.m.
- Re: A question of load . . .
FRANK LAI -- Wednesday, 27 September 2006, at 5:34 p.m.
IDS Forum is maintained by Administrator with WebBBS 5.12.
|
|
