Re: Linux compat and swap

[tlaronde%polynum.com@localhost]

Hello Mike,

On Thu, Apr 23, 2020 at 10:56:15AM +0100, Mike Pumford wrote:
> 
> 
> On 23/04/2020 08:18, Sad Clouds wrote:
> >On Wed, 22 Apr 2020 20:46:08 +0200
> >tlaronde%polynum.com@localhost wrote:
> >
> >>(This can be also simply a memory leak in the program but since the
> >>RAM is not exhausted...).
> >
> >Without much detail about how much memory, swap, etc your system is
> >using, it's anyone's guess really. You say "RAM is not exhausted" but
> >that why would the system be swapping? Maybe you're not looking at it
> >at the right time, it is possible for a process to malloc() large
> >amount of memory and then quickly release it, by the time you look at
> >it, some processes got swapped out, but the memory got released to the
> >system.
> >
> I agree with all this but I'd add one more thing to the mix that might help
> your system ride out this better.
> 
> If you have both memory intensive and filesystem intensive processes running
> on a NetBSD system the kernel filesystem cache can end up evicting programs
> running in the background that are inactive which then take a LONG time to
> recover. For a system with a reasonable amount of memory I found the
> vm.filemin and vm.filemax needed to be tweaked so that filesystem cache was
> more likely to be tweaked than program code.
> 
> I set:
> vm.filemax=10
> vm.filemin=1
> in /etc/sysctl.conf for a system with 16GB of RAM.
> 

Thank you! This is the answer to one of my major questions: I was
assuming that processes were getting the priority against file
caching so that if there was memory for file caching it would be
automatically released for processing when resources were getting
scarce.

> Before I made those changes pkgsrc builds of things like rust and firefox
> would end up with over 60% of the RAM dedicated to FS cache forcing the swap
> out of the jenkins server process that was running the build! It would also
> do the same to live firefox/thunderbird processes running on the same system
> and pretty much the only way back was a reboot as these processes could
> never swap themselves back in in a sane amount of time :(
> 
> I've also seen the same sort of odd memory consumption behaviour in scripts
> that 'just' process text files. I once managed to write a perl script that
> parsed firewall logs that ended up consuming all ram on the system for a
> reasonable sized log. I never noticed it on FreeBSD and NetBSD (I just put
> the disk activity down to the fact it was crunching a large log file).
> However when I tried to run the same script on Linux it would never complete
> as Linux wasn't able to deal with its memory hog nature at the time (this
> was years ago on Linux kernel 2.0).

I'm also still a bit puzzled by the behavior of the program since I use
it as a filter and it should read one line at a time (and doesn't need
to keep whatever information to correlate lines)...

Best regards,
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                       http://www.sbfa.fr/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C