Back to this message -- I think there are still some points to discuss.

[[ note I'm re-directing this to tech-userlevel -- I don't think there
are any real kernel issues here, at least not in the direction I'm
taking the discussion...  :-) ]]

[ On Monday, November 29, 2004 at 01:15:43 (+0100), theo borm wrote: ]
> Subject: Re: memory (re-)allocation woes
>
> What strikes me as odd is that if I switch off swap altogether, the program
> only aborts (is killed with an out of swap message) after 
> (re-)allocating about
> half the amount of memory, so in principle, having less than half the amount
> of physical memory in swap even aggravates the problem....


Ah ha!  I had not realized that your system might not have adequate swap
space.


When a program is killed because it happened to be the process trying to
use more VM than the system could accomodate, well that's an entirely
separate issue from anything to do with now malloc(), realloc(),
etc. work internally.


First off you should re-tune your test system so that it has ample swap
space to allow all the normally running procesess in your test
environment to exceed their hard RLIMIT_DATA maximum sizes (and perhaps
also adjust the default RLIMIT_DATA via /etc/login.conf if necessary).


> But please keep in mind that with whatever rlimit you specify, there is
> always a way to make the system *kill* some of the programs doing
> reallocations by simply running enough of them.

That's why there's also an RLIMIT_NPROC.  :-)

Tuning a gerneral purpose system to allow users to do what they need to
do while at the same time preventing the worst of abuses by an
unfriendly user is a process of compromise.  Ultimately in unix-like
systems, and especially in BSD-like systems, external security policies
must be implemented to prevent the worst of potential abuses.  I.e. the
traditional "big stick" policies to prevent them from happening or even
being tried in the first place.


> after an initial swapping all disk (and network) activity ceased.

Ah, well I think that's sometimes another outcome from running the
system out of swap space....   A separate issue.....


> > The process wasn't killed was it?  realloc() returned NULL and it exited
> > of its own accord by design of its code.
> 
> It *was* killed. Running a single process eating memory is not a problem;
> this will run into one third of the /set/ resource limits /before/ being killed.
> Running *multiple* processes eating memory, with the sum of their resource
> limits exceeding one third of the available *will* result in random killing.
> (and probably in some pathological cases the system thrashing itself)

I think you're just seeing the effects of running out of swap space.
Add more!  ;-)

Once your system has been adequately tuned to run your test program then
the process should always exit of its own accord after an ENOMEM error
from realloc()


> The realloc only references all those pages because of its implementation, and
> as its implementation is so silly (that it references all those pages), the OS might
> just as well have refused reallocation in the first place.
> 
> I get the impression that there are three fundamental issues here: first the OS
> overcommits memory, probably assuming that most programs allocate memory
> that they never intend to use, second the implementation of realloc is such that
> all memory allocated /is/ actually used, and third the implementation of free
> is such that freed memory is not released to the OS immediately, but apparently
> held back somewhere. Right now (IMHO) this makes NetBSD qualify for the
> label "only suitable for small jobs"

Well, "small jobs IFF they use realloc() a lot with large blocks of RAM".

(Also note that NetBSD 1.5 and subsequent release use malloc()/realloc()
implementations from FreeBSD....  though it seems some significant fixes
sill need to be brought over too, as well as maybe some from OpenBSD as
well -- they should all exhibit the same behaviour with your test
program, assuming all other things are equal, e.g. RAM, swap space, etc.)


First remember that realloc() has to copy all the memory if it has to
move the block -- i.e. there's no way to avoid having realloc() touch
all the allocated pages, especially not in the way the pathalogical test
program uses it.

Second note that as per the malloc(3) manual page the free pages will
only be returned to the OS if the "hint" option is explicitly turned on.

Realloc() could probably also be fixed to be smarter so that it would
move a block into an adjacent free block (or contiguous free blocks)
when the combined size of the adjacent and current blocks would suffice
for the new allocation.  That should solve the "only 1/3 utilization"
problem exhibited by the pathalogical test case of your sample program.

The current implementation of realloc() certainly is quite naive in how
it always makes a new allocation, always moves the data, and always
frees the old block (if the size must change -- that's its only smart
part).  :-)

It would be interesting though to first try to determine whether or not
such a fix would be a significant improvement for any more common uses
of realloc().  Given that all the *BSDs have been using the naive
realloc() for quite some time without too much complaint I'm not sure
there would be a noticable improvement in general.

Note that one of the most common uses of realloc() I've seen and used
myself in a lot of code is to resize a larger general purpose buffer
_down_ to just exactly the size it needs to be once it has been filled
with data.  The other common use is to extend a buffer, though generally
I'd guess that only one or two extensions are ever made, unlike the test
program which just keeps going and going and going....

-- 
						Greg A. Woods

+1 416 218-0098                  VE3TCP            RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>          Secrets of the Weird <woods@weird.com>