Re: Linux compat and swap

[Mike Pumford <mpumford%mudcovered.org.uk@localhost>]

On 23/04/2020 11:48, tlaronde%polynum.com@localhost wrote:
> [Please forgive me to give supplementary information gathered for all
> answers and "before". And thank you to all for answering.]
>
> To be more precise:
>
> The node is not a desktop node but a server one (so there is no X11
> running, so no firefox or whatever), with two main functions:
>
> 1) A file server;
> 2) A processing node for data (GIS) manipulations, since there can be
> huge amounts of data to process, so it is more efficient to process
> where it is saved due to the network capabilities.
In my case the build machines main purpose is as a build server (hence 
the jenkins server which is a java process). I use it as a desktop when 
developing things as its my most powerful BSD build machine and its 
easier to develop when you can use a browser to check API references etc 
so its desktop use is intermittent. So they were just examples of memory 
hungry processes that got swapped out unexpectedly rather than the only 
ones.

> The node' uptime is 225 days (since some disk replacements) serving
> during office time files to 2 to 6 nodes. And it is processing data
> (sometimes during office time; generally outside office time).
>
> It has 3862 MB of avail memory.
>
> When using a program to convert coordinates, this program being a Linux
> binary, the program bailed out, 3 times in a row, after processing 20M
> records. Splitting the input data under this amount, I managed to
> process everything.
Could you be running into the data size limits for the process? ulimit 
-d is data size, -s is stack size. If a process hits those limits it may 
be killed even if there IS memory available to protect the rest of the 
system.

Here's a comparison from the bash ulimit on BSD and linux:

BSD system has 2GB of RAM:
$ ulimit -a
number of threads               (-T) 1024
socket buffer size       (bytes, -b) unlimited
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 524288
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) 669921
max memory size         (kbytes, -m) 2009764
open files                      (-n) 2048
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 4096
cpu time               (seconds, -t) unlimited
max user processes              (-u) 300
virtual memory          (kbytes, -v) unlimited

Ubuntu 18 Linux (VM with 8GB of RAM):
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31721
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31721
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Note the fact that linux has datasize unlimited by default. That BSD 
datasize does not scale with system memory. Its still 512MB even on my 
system with 16GB of RAM. All my build wrappers for pkgsrc do ulimit -d 
to allow for the compile processes to grow really large (which they do :( )

> => This is why I jumped to the conclusion that the program had memory
> leak because processing should be one line of input at a time so
> whatever the number of lines, I fail to see why it would need more
> memory for more records.
Perhaps its badly written. ;) As I said in an earlier e-mail I had a log 
crunching program that leaked in the same sort of way. It did store 
small amounts of data for each line processed but the leak was actually 
caused by a logic error in a core loop than inadvertently caused memory 
to be kept around for every line processed.
> I found, afterwards, in /var/log/messages, that this program was killed
> because "out of swap".
>
>
> Concerning my sentence "RAM is not exhausted", my assumption (another
> one...) was that when memory for processing was getting scarced, file
> caching was reduced since there seems to be a basic "page" device for
> files: the file on disk. So for me, when there was still a huge amount
> of memory for file caching there was still a huge amount of memory as
> spare for programs at the detriment of file caching hence speed.
See above about process limits. you might find ulimit -d unlimited will 
let the process use more RAM and possibly even run to completion.

Mike