I'm not super familiar with the details of the pipe implementation in
NetBSD, but I know it's been re-worked since NetBSD-1.6.X code to take
advantage of new features in the virtual memory subsystem and process
scheduling.  Since NetBSD and FreeBSD have diverged in this area of the
kernel significantly, it's not surprising that you get different behaviors.
If I had to wager a guess, I'd bet that if you ran your code on NetBSD-1.5
or earlier, you'd see behavior more similar to FreeBSD's behavior than with
NetBSD 2.0 and later.  (The reason I say NetBSD-1.5 and not 1.6 is because
I believe some of the initial pipe changes made it into 1.6 code, while the
rest went into 2.0 code.)
-Brian
On Dec 2,  8:32pm, Vincent Stemen wrote:
} Subject: Re: Possible bug relating to malloc()/realloc(), popen(), and rea
} On Thu, Dec 02, 2004 at 08:34:53PM -0500, Dan LaBell wrote:
} > 
} > On Thursday, December 2, 2004, at 01:58 PM, Vincent Stemen wrote:
} > 
} > >On Thu, Dec 02, 2004 at 03:25:35PM +0100, Martin Husemann wrote:
} > >>On Wed, Dec 01, 2004 at 07:05:40PM -0600, Vincent Stemen wrote:
} > >>>but the pointer I get back from realloc() does
} > >>>not point to the beginning of the data allocated by the first 
} > >>>malloc()
} > >>>as it should.
} > >>
} > >>Why do you think it should?
} > >
} > >Because the manual on realloc() says it should and it would not be
} > >very useful otherwise.  Either way, as pointed out by other replies, I
} > >was misinterpreting the result.  That was not the problem.
} > >
} > >
} > 
} > From K&R 2nd ed.  "realloc returns a pointer to the new space or
} > NULL if the request cannot be satisfied ..."
} > 
} > It's allowed to relocate the space,  ( in which case is copies from
} 
} Yes, I understand that.  Sorry if I didn't word that part clearly
} enough.  When I said the return from realloc() did not point to the
} beginning of the data, I didn't mean that I expected it to point to
} the same address, but to the beginning of the data even if it was
} relocated.  Since I thought you could depend on the amount of data
} read from the popen() pipe, I was not advancing my pointer after the
} first read when it was shorter than my specified block size, causing
} the first block of data to be overwritten.  I misinterpreted that as
} an incorrect pointer being returned from realloc().
} 
} So the actual difference between NetBSD and FreeBSD seems to be in
} read() when reading from the pipe.  I don't understand why, on NetBSD,
} it always limits the first read to 1024 bytes and not any of the
} following reads, and why it does not do it when I step through it
} slowly with the debugger.
} 
} Your other points are interesting though.  Thanks.
} 
} 
} > the from old location to the new location ) or it wouldn't be useful,
} > in that it would break trivially, if there is any memory fragmentation.
} > Really, you only get the same address back in special cases.
} > 1)  The memory allocated is at the edge of of the brk/sbrk line, when
} >     malloc increases the break, it's still contiguous, so the regions 
} > only
} >     increases in size, you get the same pointer address back.
} > 2)  Because of other free()'s,  malloc sees there is contiguous memory
} >     available, and no brk/sbrk call is necessary.
} > 3)  (special case of 1) Malloc actually brk/sbrk'ed more memory than it
} >      needed, last malloc,  just so every malloc doesn't mean a syscall 
} > to
} >      sbrk()  (for efficiency ).
} > 
} > If it isn't a special case then realloc is almost the same as  
} > malloc(newsize), and bcopy(old,new,oldsize) , free(old), return (new).  
} > Basically, I consider realloc a convenience function that does 
} > (malloc,bcopy,free,) except that it has "window optimizations" (if 
} > that's the right term), where in some circumstances the bcopy can 
} > avoided.
} > 
} > I don't know what freebsd uses for malloc, but implementations vary,
} > gnu's will even try to use mmap for large chunks, as opposed to sbrk()  
} > -- at any rate if freebsd's brk()'ed for more memory on the first call 
} > than netbsd,
} > that may be why.   I probably would go-ahead and use read() and 
} > write(), like you are doing, to avoid any need to flush with stdio ( 
} > although I think flush is more important on write than read, but I 
} > would read in PIPEBUF chunks.
} > As far as the other issues,  I wonder how cat is writing it out, and
} > if anything changes if you run dd if=testfile bs=1028 -- I bet cat is 
} > linebuffering with stdio.
} > 
} > Actually looking at the source, its memcpy not bcopy, and seems like we 
} > are using the same malloc, as I see freebsd notices... and I'm 
} > surprised how tiny PIPEBUF is, 512 bytes, is that configurable?  If its 
} > posix I still think its low for atomic writes...
} > I don't know if I actually helped, but I thought I'd weigh in w/ some 
} > c/unix lore.
} 
} -- 
} Vincent Stemen
} Avoid the VeriSign/Network Solutions domain registration trap!
} Read how Network Solutions (NSI) was involved in stealing our domain name.
} http://www.InetAddresses.net
} 
>-- End of excerpt from Vincent Stemen