OK, here's what's going on:
- You didn't turn on PDQPRIORITY so the update statistics high command
has to scan all 71 partitions sequentially.
- You didn't increase the amount of memory allocated for sorting so it
only has 15MB available to sort 5 columns of keys totaling 960MB five
different ways (once ordering by each key) - DS_NONPDQ_QUERY_MEM without
PDQ or DS_TOTAL_MEMORY and DS_MAX_QUERIES with PDQPRIORITY > 0
- You didn't enable PSORT_NPROCS so the non-PDQ sorting is also single
threaded.
At the time you extracted the sqexplain output the update statistics
command was beginning to process the 4th sort of 5, so your between 60% &
80% done give or take a few percentages if the different keys are of
significantly different sizes.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Fri, Dec 9, 2016 at 10:22 AM, Habichtsberg, Reinhard <
RHabichtsberg@arz-emmendingen.de> wrote:
> Art:
>
> There are only a few lines in sqexplain.out:
>
> UPDATE STATISTICS:
> ==================
>
> Table: informix.xxxxxxxxx
> Mode: HIGH
> Number of Bins: 267 Bin size 67362549
> Sort data 964908.6 MB Sort memory granted 15.0 MB
> Estimated number of table scans 5
> PASS #1 pe_id
> PASS #2 pe_bearbeiter
> PASS #3 pe_pruefstart
> PASS #4 pe_wfstatusvor
> PASS #5 pe_re
> Scan 0 Sort 0 Build 34591 Insert 0 Close 0 Total 34591
> Completed pass 1 in 576 minutes 31 seconds
> Scan 0 Sort 0 Build 141308 Insert 0 Close 0 Total 141308
> Completed pass 2 in 2355 minutes 8 seconds
> Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total 36106
> Completed pass 3 in 601 minutes 46 seconds
>
> Btw: The table has 71 partitions. The table is always heavily used and has
> about seven million of write ops a day. Furthermore the table is part of
> many
> queries.
>
> Thank you,
> Reinhard.
>
> -----Ursprüngliche Nachricht-----
> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art
> Kagel
> Gesendet: Freitag, 9. Dezember 2016 14:34
> An: ids@iiug.org
> Betreff: Re: Long lasting dostats [38274]
>
> Reinhard:
>
> Welllllll, you've got a table that, at least on the surface, seems to have
> on
> the order of 10^10 rows and you don't have PDQPRIORITY or PSORT_NPROCS set
> so
> this is a single threaded operation on a table that has well over 30
> partitions (I keep losing count). It's just going to take time.
>
> You have -X set... I'd like to see the sqexplain file lines from this run
> (you
> can send these to me directly so as not to clog the list.
>
> Art
>
> Art S. Kagel, President and Principal Consultant ASK Database Management
> www.askdbmgt.com
>
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and
> do not reflect on the IIUG, nor any other organization with which I am
> associated either explicitly, implicitly, or by inference. Neither do those
> opinions reflect those of other individuals affiliated with any entity with
> which I am affiliated nor those of the entities themselves.
>
> On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard <
> RHabichtsberg@arz-emmendingen.de> wrote:
>
> > Hi all
> >
> > Informix 12.10.FC4W1XB, Solaris 10
> >
> > I'm running dostats on a very big table. It seems it won't come to an
> end.
> > Runs now since Monday, the 5th of December.
> >
> > Table myschema:
> >
> > CREATE TABLE "informix".xxxxxxxxxxxxx (
> >
> > pe_id SERIAL8(15712838227) NOT NULL,
> >
> > pe_re INTEGER,
> >
> > pe_re_lfd SMALLINT,
> >
> > pe_vo SMALLINT,
> >
> > pe_pos SMALLINT,
> >
> > pe_ebene CHAR(2),
> >
> > pe_pruefung SMALLINT,
> >
> > pe_pruefstart DATETIME YEAR TO SECOND,
> >
> > pe_pruefende DATETIME YEAR TO SECOND,
> >
> > pe_pruefart CHAR(1),
> >
> > pe_bearbeiter CHAR(8),
> >
> > pe_wfstatusvor DECIMAL(8,4),
> >
> > pe_wfstatusnach DECIMAL(8,4),
> >
> > pe_ergx SMALLINT,
> >
> > pe_ergxart CHAR(1),
> >
> > pe_berinr INTEGER,
> >
> > pe_textnr INTEGER,
> >
> > pe_folgeberinr INTEGER,
> >
> > pe_jobid INTEGER,
> >
> > pe_jobart CHAR(1),
> >
> > pe_pruefzeit DECIMAL(12,2),
> >
> > pe_pruefzeit_ges DECIMAL(12,2),
> >
> > pe_wfnr SMALLINT,
> >
> > pe_nachbeding SMALLINT,
> >
> > pe_folge_ga INTEGER,
> >
> > pe_lfdnr_verf INTEGER
> > ) FRAGMENT BY EXPRESSION
> >
> > PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < 2400000000
> > ) ) IN xxxxxxxxx23,
> >
> > PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < 2600000000
> > ) ) IN xxxxxxxxx24,
> >
> > PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < 2800000000
> > ) ) IN xxxxxxxxx25,
> >
> > PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < 3000000000
> > ) ) IN xxxxxxxxx26,
> >
> > PARTITION xxxxxxxxx27 ((pe_id >= 3000000000 ) AND (pe_id < 3200000000
> > ) ) IN xxxxxxxxx27,
> >
> > PARTITION xxxxxxxxx28 ((pe_id >= 3200000000 ) AND (pe_id < 3400000000
> > ) ) IN xxxxxxxxx28,
> >
> > PARTITION xxxxxxxxx29 ((pe_id >= 3400000000 ) AND (pe_id < 3600000000
> > ) ) IN xxxxxxxxx29, PARTITION xxxxxxxxx30 ((pe_id >= 3600000000 ) AND
> > (pe_id < 3800000000 ) ) IN xxxxxxxxx30,
> >
> > PARTITION xxxxxxxxx31 ((pe_id >= 3800000000 ) AND (pe_id < 4000000000
> > ) ) IN xxxxxxxxx31,
> >
> > PARTITION xxxxxxxxx14 ((pe_id >= 4000000000 ) AND (pe_id < 4200000000
> > ) ) IN xxxxxxxxx14,
> >
> > PARTITION xxxxxxxxx5 ((pe_id >= 4200000000 ) AND (pe_id < 4400000000 )
> > ) IN xxxxxxxxx5,
> >
> > PARTITION xxxxxxxxx13 ((pe_id >= 4400000000 ) AND (pe_id < 4600000000
> > ) ) IN xxxxxxxxx13,
> >
> > PARTITION xxxxxxxxx11 ((pe_id >= 4800000000 ) AND (pe_id < 5000000000
> > ) ) IN xxxxxxxxx11,
> >
> > PARTITION xxxxxxxxx22 ((pe_id >= 5000000000 ) AND (pe_id < 5200000000
> > ) ) IN xxxxxxxxx22,
> >
> > PARTITION xxxxxxxxx3 ((pe_id >= 5200000000 ) AND (pe_id < 5400000000 )
> > ) IN xxxxxxxxx3,
> >
> > PARTITION xxxxxxxxx21 ((pe_id >= 5400000000 ) AND (pe_id < 5600000000
> > ) ) IN xxxxxxxxx21,
> >
> > PARTITION xxxxxxxxx2 ((pe_id >= 5600000000 ) AND (pe_id < 5800000000 )
> > ) IN xxxxxxxxx2,
> >
> > PARTITION xxxxxxxxx1 ((pe_id >= 5800000000 ) AND (pe_id < 6000000000 )
> > ) IN xxxxxxxxx1,
> >
> > PARTITION xxxxxxxxx10 ((pe_id >= 6000000000 ) AND (pe_id < 6200000000
> > ) ) IN xxxxxxxxx10,
> >
> > PARTITION xxxxxxxxx15 ((pe_id >= 6200000000 ) AND (pe_id < 6400000000
> > ) ) IN xxxxxxxxx15,
> >
> > PARTITION xxxxxxxxx9 ((pe_id >= 6400000000 ) AND (pe_id < 6600000000 )
> > ) IN xxxxxxxxx9,
> >
> > PARTITION xxxxxxxxx8 ((pe_id >= 6600000000 ) AND (pe_id < 6800000000 )
> > ) IN xxxxxxxxx8,
> >
> > PARTITION xxxxxxxxx7 ((pe_id >= 6800000000 ) AND (pe_id < 7000000000 )
> > ) IN xxxxxxxxx7,
> >
> > PARTITION xxxxxxxxx6 ((pe_id >= 7000000000 ) AND (pe_id < 7200000000 )
> > ) IN xxxxxxxxx6,
> >
> > PARTITION xxxxxxxxx20 ((pe_id >= 7200000000 ) AND (pe_id < 7400000000
> > ) ) IN xxxxxxxxx20,
> >
> > PARTITION xxxxxxxxx19 ((pe_id >= 7400000000 ) AND (pe_id < 7600000000
> > ) ) IN xxxxxxxxx19,
> >
> > PARTITION xxxxxxxxx18 ((pe_id >= 7600000000 ) AND (pe_id < 7800000000
> > ) ) IN xxxxxxxxx18,
> >
> > PARTITION xxxxxxxxx17 ((pe_id >= 7800000000 ) AND (pe_id < 8000000000
> > ) ) IN xxxxxxxxx17,
> >
> > PARTITION xxxxxxxxx16 ((pe_id >= 8000000000 ) AND (pe_id < 8200000000
> > ) ) IN xxxxxxxxx16,
> >
> > PARTITION xxxxxxxxx4 ((pe_id >= 8200000000 ) AND (pe_id < 8400000000 )
> > ) IN xxxxxxxxx4,
> >
> > PARTITION xxxxxxxxx32 ((pe_id >= 8400000000 ) AND (pe_id < 8600000000
> > ) ) IN xxxxxxxxx32,
> >
> > PARTITION xxxxxxxxx33 ((pe_id >= 8600000000 ) AND (pe_id < 8800000000
> > ) ) IN xxxxxxxxx33,
> >
> > PARTITION xxxxxxxxx34 ((pe_id >= 8800000000 ) AND (pe_id < 9000000000
> > ) ) IN xxxxxxxxx34,
> >
> > PARTITION xxxxxxxxx35 ((pe_id >= 9000000000 ) AND (pe_id < 9200000000
> > ) ) IN xxxxxxxxx35,
> >
> > PARTITION xxxxxxxxx36 ((pe_id >= 9200000000 ) AND (pe_id < 9400000000
> > ) ) IN xxxxxxxxx36,
> >
> > PARTITION xxxxxxxxx37 ((pe_id >= 9400000000 ) AND (pe_id < 9600000000
> > ) ) IN xxxxxxxxx37,
> >
> > PARTITION xxxxxxxxx38 ((pe_id >= 9600000000 ) AND (pe_id < 9800000000
> > ) ) IN xxxxxxxxx38,
> >
> > PARTITION xxxxxxxxx39 ((pe_id >= 9800000000 ) AND (pe_id < 10000000000
> > ) ) IN xxxxxxxxx39,
> >
> > PARTITION xxxxxxxxx40 ((pe_id >= 10000000000 ) AND (pe_id <
> > 10200000000 )
> > ) IN
> > xxxxxxxxx40,
> >
> > PARTITION xxxxxxxxx41 ((pe_id >= 10200000000 ) AND (pe_id <
> > 10400000000 )
> > ) IN
> > xxxxxxxxx41,
> >
> > PARTITION xxxxxxxxx42 ((pe_id >= 10400000000 ) AND (pe_id <
> > 10600000000 )
> > ) IN
> > xxxxxxxxx42,
> >
> > PARTITION xxxxxxxxx43 ((pe_id >= 10600000000 ) AND (pe_id <
> > 10800000000 )
> > ) IN
> > xxxxxxxxx43,
> >
> > PARTITION xxxxxxxxx44 ((pe_id >= 10800000000 ) AND (pe_id <
> > 11000000000 )
> > ) IN
> > xxxxxxxxx44,
> >
> > PARTITION xxxxxxxxx45 ((pe_id >= 11000000000 ) AND (pe_id <
> > 11200000000 )
> > ) IN
> > xxxxxxxxx45,
> >
> > PARTITION xxxxxxxxx46 ((pe_id >= 11200000000 ) AND (pe_id <
> > 11400000000 )
> > ) IN
> > xxxxxxxxx46,
> >
> > PARTITION xxxxxxxxx47 ((pe_id >= 11400000000 ) AND (pe_id <
> > 11600000000 )
> > ) IN
> > xxxxxxxxx47,
> >
> > PARTITION xxxxxxxxx48 ((pe_id >= 11600000000 ) AND (pe_id <
> > 11800000000 )
> > ) IN
> > xxxxxxxxx48,
> >
> > PARTITION xxxxxxxxx49 ((pe_id >= 11800000000 ) AND (pe_id <
> > 12000000000 )
> > ) IN
> > xxxxxxxxx49,
> >
> > PARTITION xxxxxxxxx50 ((pe_id >= 12000000000 ) AND (pe_id <
> > 12200000000 )
> > ) IN
> > xxxxxxxxx50,
> >
> > PARTITION xxxxxxxxx51 ((pe_id >= 12200000000 ) AND (pe_id <
> > 12400000000 )
> > ) IN
> > xxxxxxxxx51,
> >
> > PARTITION xxxxxxxxx52 ((pe_id >= 12400000000 ) AND (pe_id <
> > 12600000000 )
> > ) IN
> > xxxxxxxxx52,
> >
> > PARTITION xxxxxxxxx53 ((pe_id >= 12600000000 ) AND (pe_id <
> > 12800000000 )
> > ) IN
> > xxxxxxxxx53,
> >
> > PARTITION xxxxxxxxx54 ((pe_id >= 12800000000 ) AND (pe_id <
> > 13000000000 )
> > ) IN
> > xxxxxxxxx54,
> >
> > PARTITION xxxxxxxxx55 ((pe_id >= 13000000000 ) AND (pe_id <
> > 13200000000 )
> > ) IN
> > xxxxxxxxx55,
> >
> > PARTITION xxxxxxxxx12 ((pe_id >= 4600000000 ) AND (pe_id < 4800000000
> > ) ) IN xxxxxxxxx12,
> >
> > PARTITION xxxxxxxxx56 ((pe_id >= 13200000000 ) AND (pe_id <
> > 13400000000 )
> > ) IN
> > xxxxxxxxx56,
> >
> > PARTITION xxxxxxxxx57 ((pe_id >= 13400000000 ) AND (pe_id <
> > 13600000000 )
> > ) IN
> > xxxxxxxxx57,
> >
> > PARTITION xxxxxxxxx58 ((pe_id >= 13600000000 ) AND (pe_id <
> > 13800000000 )
> > ) IN
> > xxxxxxxxx58,
> >
> > PARTITION xxxxxxxxx59 ((pe_id >= 13800000000 ) AND (pe_id <
> > 14000000000 )
> > ) IN
> > xxxxxxxxx59,
> >
> > PARTITION xxxxxxxxx60 ((pe_id >= 14000000000 ) AND (pe_id <
> > 14200000000 )
> > ) IN
> > xxxxxxxxx60,
> >
> > PARTITION xxxxxxxxx61 ((pe_id >= 14200000000 ) AND (pe_id <
> > 14400000000 )
> > ) IN
> > xxxxxxxxx61,
> >
> > PARTITION xxxxxxxxx62 ((pe_id >= 14400000000 ) AND (pe_id <
> > 14600000000 )
> > ) IN
> > xxxxxxxxx62,
> >
> > PARTITION xxxxxxxxx63 ((pe_id >= 14600000000 ) AND (pe_id <
> > 14800000000 )
> > ) IN
> > xxxxxxxxx63,
> >
> > PARTITION xxxxxxxxx64 ((pe_id >= 14800000000 ) AND (pe_id <
> > 15000000000 )
> > ) IN
> > xxxxxxxxx64,
> >
> > PARTITION xxxxxxxxx65 ((pe_id >= 15000000000 ) AND (pe_id <
> > 15200000000 )
> > ) IN
> > xxxxxxxxx65,
> >
> > PARTITION xxxxxxxxx66 ((pe_id >= 15200000000 ) AND (pe_id <
> > 15400000000 )
> > ) IN
> > xxxxxxxxx66,
> >
> > PARTITION xxxxxxxxx67 ((pe_id >= 15400000000 ) AND (pe_id <
> > 15600000000 )
> > ) IN
> > xxxxxxxxx67,
> >
> > PARTITION xxxxxxxxx68 ((pe_id >= 15600000000 ) AND (pe_id <
> > 15800000000 )
> > ) IN
> > xxxxxxxxx68,
> >
> > PARTITION xxxxxxxxx69 ((pe_id >= 15800000000 ) AND (pe_id <
> > 16000000000 )
> > ) IN
> > xxxxxxxxx69,
> >
> > PARTITION xxxxxxxxx70 ((pe_id >= 16000000000 ) AND (pe_id <
> > 16200000000 )
> > ) IN
> > xxxxxxxxx70,
> >
> > PARTITION xxxxxxxxx_rem remainder IN xxxxxxxxx_rem EXTENT SIZE 1000000
> > NEXT SIZE 1000000 LOCK MODE ROW;
> >
> > {
> >
> > Please review extent sizing and adjust to allow for growth.
> > }
> >
> > REVOKE ALL ON "informix".xxxxxxxxxx FROM public; CREATE UNIQUE INDEX
> > "informix".xxxxxxxxxx_31 ON "informix".xxxxxxxxxx (
> >
> > pe_id ASC
> > ) USING btree;
> >
> > CREATE INDEX "informix".xxxxxxxxxx_32 ON "informix".xxxxxxxxxx (
> >
> > pe_pruefstart ASC,
> >
> > pe_jobid ASC
> > ) USING btree;
> >
> > CREATE INDEX "informix".xxxxxxxxxx_33 ON "informix".xxxxxxxxxx (
> >
> > pe_re ASC,
> >
> > pe_re_lfd ASC,
> >
> > pe_vo ASC,
> >
> > pe_pos ASC,
> >
> > pe_wfstatusnach ASC
> > ) USING btree;
> >
> > CREATE INDEX "informix".xxxxxxxxxx_34 ON "informix".xxxxxxxxxx (
> >
> > pe_wfstatusvor ASC,
> >
> > pe_pruefung ASC,
> >
> > pe_pruefstart ASC
> > ) USING btree;
> >
> > CREATE INDEX "informix".xxxxxxxxxx_35 ON "informix".xxxxxxxxxx (
> >
> > pe_bearbeiter ASC,
> >
> > pe_pruefstart ASC
> > ) USING btree;
> >
> > GRANT SELECT, UPDATE, INSERT, DELETE, INDEX ON xxxxxxxxxx TO "public"
> > AS "informix"; GRANT SELECT ON xxxxxxxxxx TO "r_all" AS "informix";
> > GRANT SELECT, UPDATE, INSERT, DELETE ON xxxxxxxxxx TO "rw_all" AS
> > "informix";
> >
> > ALTER TABLE xxxxxxxxxx ADD CONSTRAINT PRIMARY KEY (
> >
> > pe_id
> > ) CONSTRAINT "informix".pk_sd_xxxx_22_neu;
> >
> > Size: 1.390,645 GB rows: 13.512.969.517
> >
> > Dostats logging:
> >
> > dostats -d xxxxx -t xxxxxxxxx -G -P 0 -X -v 2 -E --isolation d
> >
> > Verbose mode level 2 ENABLED....
> > ISOLATION LEVEL: DIRTY READ.
> > Initially drop distributions DISABLED...
> > Stored Procedure processing DISABLED...
> > Shared Memory Connection handling DISABLED...
> > Expanded Reporting ENABLED....
> > Expanded Time Reporting ENABLED....
> > Force Distributions Only ENABLED....
> > Distributions Only clauses ENABLED....
> > System catalog processing DISABLED...
> > Table Name Template SUPPLIED (xxxxxxxxx)...
> > Host Name NOT SUPPLIED...
> > Command Execution IMMEDIATE...
> > SET EXPLAIN ENABLED...
> > Lock mode set to wait 10 seconds.
> > PDQPRIORITY for table processing set to: 10.
> > Distribution Aging limit set to: 30 days Table browse threshhold set
> > to: 15.000000%
> >
> > Warning! Optimized servers produce statistics fastest with
> > PSORT_NPROCS set.
> >
> > Warning! Optimized servers produce statistics fastest with
> > PSORT_DBTEMP set.
> >
> > Warning! Optimized servers produce statistics fastest with DBUPSPACE set.
> >
> > Working on database: xxxxx.
> >
> > Working on table xxxxxxxxx:
> >
> > pe_pruefstart, pe_jobid(LOW)...SUCCESSFUL (0 hrs 41 mins 40.459 Secs)
> >
> > pe_id(LOW)...SUCCESSFUL (1 hrs 25 mins 22.479 Secs)
> >
> > pe_re, pe_re_lfd, pe_vo, pe_pos, pe_wfstatusnach(LOW)...SUCCESSFUL (11
> > hrs 34 mins 40.682 Secs)
> >
> > pe_bearbeiter, pe_pruefstart(LOW)...SUCCESSFUL (3 hrs 38 mins 41.680
> > Secs)
> >
> > pe_wfstatusvor, pe_pruefung, pe_pruefstart(LOW)...SUCCESSFUL (6 hrs 42
> > mins
> > 33.324 Secs)
> >
> > pe_id, pe_re, pe_pruefstart, pe_bearbeiter, pe_wfstatusvor(HIGH)...
> >
> > Update Stats low needed about 24 hrs. Since then Update Statistics
> > high is running and need about 92 hrs until now.
> >
> > How can I determine if there is a chance that the run comes to an end
> > and if so, how long will it last.
> >
> > TIA
> >
> > Reinhard
> >
> >
> > ************************************************************
> > *******************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --089e013d18c017585b054339cfc5
>
>
> ************************************************************
> *******************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
> ************************************************************
> *******************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--089e013d18c013a5a505433bda76
