IDS Forum

Re: onload consuming shared memory

Posted By: Johan Backlund
Date: Thursday, 24 June 2004, at 4:15 p.m.

In Response To: onload consuming shared memory (Johan Backlund )

Thanks for the responses I have received. However, I want to emphasize
that the primary problem is the fact that onload causes IDS to allocate
huge amounts of shared memory - as I said, we were seeing 16-20MB being
allocated every minute in the last half hour.

Therefore, changing the shared memory allocation parameters might
improve the situation for a short time, but if IDS continues to
allocate 20MB per minute during the last 90 minutes of the load, we
will hit the total shared memory limit for a 32-bit application on
HP-UX 11 including all tweaking. I would like to know what the memory
is to be used for, so that we can predict the amount of shared memory
we will need.

The speed of the load is ok if it comes in under 5 hours, even if it
would be great to cut it down. Correct me if I am wrong, but since we
are using onload, I am assuming that indes are created after load
automagically? I can't see how indexes could be updated row by row when
data is loaded page by page in binary format.

We have not had the time to do another test. We have verified that the
data is ok with oncheck, just to get that out of the equation. As I see
it, we will probably resort to using dbimport instead, and take the hit
of having a very long system down in favour of getting our migration
done.

Again, thanks for you responses.

/Johan


--- JHAYS2@sears.com wrote:
> Johan,
>
> One more thing. Contiguous memory is more efficient than fragmented
> memory. Consider increasing SHMVIRTSIZE and SHMADD for the load.
> They
> can always be put back to normal operating sizes after the load.
> When
> Informix initializes, it will attempt to grab SHMVIRTSIZE in one
> contiguous block. Also, if additional segments are needed, it is
> better
> to grab a large segment rather than many small segments.
>
> Also, see my previous msg about delaying index building until after
> the
> load. Your "net net" will be much better that way. Your load will
> zip
> through much faster without the indexes being updated as each row is
> added.
>
> Jamie
>
>
>
>
> "Johan Backlund " <jbacklund@yahoo.com>
> Sent by: forum.subscriber@iiug.org
> 06/20/2004 08:58 AM
>
>
> To: ids@iiug.org
> cc:
> Subject: onload consuming shared memory [3138]
>
>
> Hi all,
>
> we're just about to start a full database restore to recover from a
> failed onload session. I just want to try and determine what might
> have
> gone wrong to see if how we can do it better next time.
>
> We want to onload a table with blob data of about 300000 rows and
> 11GB
> of onload files size. The onload ran for some three hours with an
> estimated running time of about 4.5 hours. All looked just fine,
> except
> maybe that we had thought it would be a little faster than that.
>
> Then the load stopped with "Not enough core" as error message from
> onload. The online log complains on ENOMEM when trying to allocate
> another virtual segment. I looked at onstat -g seg and it had
> allocated
> some 30 extra segments. Interestingly enough, the allocation of new
> segments came mostly from the last half hour, where it allocated a
> new
> segment about once every minute. Even more interesting, each segment
> was a little larger than the previous one (it started on 16MB
> segments
> and ended with sizes almost up at 20MB...).
>
> This system is on 9.21.HC7X3 on an HP-UX 11.
>
> I've got a couple of questions that you may be able to answer:
>
> (a) Why does onload need this memory? Can we avoid it somehow?
>
> (b) If we cannot avoid, can we predict how much it might need? I
> mean,
> even if we increase the size of each allocated segment to lower the
> number of segments, we still need to know that we will stay within
> the
> total limits of the machine (OS).
>
> (c) Anyone seen virtual segments increasing in size as more of them
> are
> allocated...looks like kind of buggy behaviour to me?
>
> Well, gotta go back to the team and check that they are on track on
> the
> full restore.
>
> thanks for listening.
>
> Johan
>
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Mail Address AutoComplete - You start. We finish.
> http://promotions.yahoo.com/new_mail
>
>
>
>


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Messages In This Thread

• onload consuming shared memory
  Johan Backlund -- Sunday, 20 June 2004, at 9:58 a.m.
  • Re: onload consuming shared memory
    JHAYS2@sears.com -- Monday, 21 June 2004, at 10:29 a.m.
    • RE: onload consuming shared memory
      Schmitz, Rob B [ITS] -- Monday, 21 June 2004, at 11:42 a.m.
  • Re: onload consuming shared memory
    JHAYS2@sears.com -- Monday, 21 June 2004, at 10:47 a.m.
    • RE: onload consuming shared memory
      Alexey Sonkin -- Monday, 21 June 2004, at 3:19 p.m.
  • Re: onload consuming shared memory
    Johan Backlund -- Thursday, 24 June 2004, at 4:15 p.m.
    • RE: onload consuming shared memory
      Michaelis, Daniel -- Thursday, 24 June 2004, at 5:34 p.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.