> > As far as I understand, this is to reserve a lot of space first 
> > (MEMORY_DISK_ROOT_SIZE), and then later tell the kernel how much space 
> > really to allocate. Try embedding your 11MB filesystem in the 51MB kernel.
> 
> OK, now I set MEMORY_DISK_ROOT_SIZE=15000 which gives 15000*512/1024 = 7500 k
> The image is 5112 k 

Not sure, but it doesn't seem like memory reserved by MEMORY_DISK_ROOT_SIZE
is ever freed (mdsetimage won't do it that is). So I'd say first create
the disk image, then set MEMORY_DISK_ROOT_SIZE to match (roud up to closest
blocksize multiple).

I may be wrong on this though, didn't look too deep...

> #mdsetimage -v -s netbsd custom.img
> [... works fine ...]
> 
> Running without -s gives:
> # mdsetimage -v netbsd custom.img
> [... works fine ...]
> 
> Seems like -s sets the reserved space to the size of the image.

Yes, see also /usr/src/gnu/usr.sbin/mdsetimage/mdsetimage.c

> I am not sure how big the mounted memory fs is since I have problems
> with kernel refusing to execute /sbin/init
> It stops right before that.
> So I made a simple printf C code and replaced init with it
> # sbin/init 
> Tell YazzY you started!
> 
> exec /sbin/init: error 8
                   ^^^^^^^

This is (from execve(2)):

     [ENOEXEC]          The new process file has the appropriate access per-
                        mission, but has an invalid magic number in its
                        header.

Maybe you forgot some COMPAT_<foo> in your kernel config?

	Regards,
		-- Jachym Holecek