Re: The sad state of pkgsrc [was Re: pkgsrc NetBSD 7.0.1_PATCH/x86_64 2016-09-04 11:03]

[David Holland <dholland-pkgtech%netbsd.org@localhost>]

On Wed, Sep 14, 2016 at 02:21:38PM +0000, David Holland wrote:
 > On Wed, Sep 07, 2016 at 11:35:01PM +0200, Joerg Sonnenberger wrote:
 >  > > OK, for sdcc3, it fails for me too with a 2GB VA limit. It builds
 >  > > with a 3GB limit, and it doen't seem to use really use 3GB of RAM
 >  > > (I didn't see my system swapping or reclaiming that much memory
 >  > > from the file cache).  I'm not sure what could be done to fix
 >  > > this. Nor if limiting the VA to 2GB is a good idea :)
 >  > 
 >  > Looking a bit further, it seems to be gplib that fails. That's
 >  > ridiculous -- a linker for a 16bit target trying to allocate more than
 >  > 2GB space. I wonder if there is a negative integer cast to unsigned
 >  > somewhere or so.
 > 
 > It's worse than that: gplib is ar. I have just looked inside it and
 > added some diagnostic stuff, but based on what I've found my guess is
 > that the compiler and/or gputils assembler is emitting an invalid
 > object file that causes gplib to to choke.
 >
 > I also have a patch coming up to make the sdcc3 lib build not
 > blatantly ignore build errors.

Nope nope nope, all wrong.

The build message is
   gp_calloc() -- Could not allocate 4194304 bytes of memory. {gpmemory.c.LINE-308, b_memory_put()}
   gmake[6]: [Makefile:32: pic16f1773.o] Error 1 (ignored)
   error: "pic16f1773.o" is not a valid object file

What the build does is
   -$(CC)
   -$(LIB)
and gputils's error thing exits on failure. What this means is that
the calloc error is from the assembler, and then the second message is
gplib reporting that pic16f1773.o doesn't exist.

I didn't look in the gputils assembler, but I did run an sdcc3 build
with gputils logging all memory allocations, which sort | uniq -c'd 
are:
      1 [[calloc: 167640 * 88]]
      1 [[calloc: 1905 * 16]]
      1 [[calloc: 475 * 152]]
      1 malloc: [[13554]]
      1 malloc: [[34528]]
      1 malloc: [[69056]]
      1 malloc: [[7440]]
      1 malloc: [[85057]]
      2 [[calloc: 1 * 144]]
      2 malloc: [[312]]
      4 [[calloc: 1 * 112]]
      4 malloc: [[128]]
     13 [[calloc: 1 * 1416]]
    474 [[calloc: 65536 * 64]]   <---
    475 [[calloc: 1 * 152]]
    476 [[calloc: 1 * 24]]
    477 [[calloc: 1 * 40]]
    477 [[calloc: 40 * 40]]
    931 malloc: [[16]]
   1428 [[calloc: 1 * 88]]
   9828 malloc: [[24]]
   27350 malloc: [[32]]

The problem is that there are 474 allocations of 65536 64-byte
objects, which totals close to 2G. It's one of these that fails when
you crank down the memory limit.

I haven't looked into where they're coming from and I need to go do
other things, but someone else should be able to take it up from
here.

-- 
David A. Holland
dholland%netbsd.org@localhost