Re: 2025Q1 lang/rust build failure on illumos (ENOMEM)

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Wed, 16 Apr 2025 13:13:58 +0200
> From: Hans Rosenfeld <hans%netbsd.org@localhost>
> 
> thread '<unnamed>' panicked at std/src/sys/pal/unix/stack_overflow.rs:476:13:
> failed to allocate a guard page: Not enough space (os error 12)
> note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
> fatal runtime error: initialization or cleanup bug
> rustc exited with signal: 6 (SIGABRT) (core dumped)
> 
> "os error 12" is ENOMEM, but that doesn't make much sense to me. The
> OpenIndiana system has no stack limit or any other memory limits set,
> and it's barely using half of its 128GB of physical memory.
> 
> Does anyone have an idea what's going on here?

Here's the relevant code, install_main_guard_default, to map a stack
guard page in the method chosen by Rust on illumos, including the
panic at line 476:

https://github.com/rust-lang/rust/blob/1.83.0/library/std/src/sys/pal/unix/stack_overflow.rs#L454-L487

It uses mmap with MAP_FIXED to map a stack guard page at a specific
location, but mmap returns ENOMEM to indicate that the requested
address could not be mapped (not that it was in use already -- that is
handled as if you did munmap first -- but that you can't mmap the
address at all).

Previously, in 1.81.0 which built (according to, e.g.,
https://mail-index.NetBSD.org/pkgsrc-bulk/2025/01/12/msg026447.html),
I suspect this code was simply not used on illumos.  And sure enough,
when I look at the diff between 1.81.0 and 1.83.0 of this file, there
are illumos-related changes.  It looks like something may have been
enabled back in September some time between 1.81.0 and 1.83.0 on
illumos:

https://github.com/rust-lang/rust/commit/54672ac392e3a4af640f59db01e93200b96c97d5
https://github.com/rust-lang/rust/issues/128568

I bet if you revert the change it will work.  But I don't know enough
about illumos to say why it might have worked in upstream testing but
not for you -- there tends to be a bit of variation in how operating
systems handle stack guard pages.  Someone who knows illumos details
should take a look to make sure what Rust is doing is sensible.

For comparison, here's how stack guard pages work on NetBSD:

https://man.NetBSD.org/stack.7

It's possible that illumos behaves essentially the same as one of the
other operating systems listed here in the install_main_guard dispatch
logic, and should use that instead of install_main_guard_default:

https://github.com/rust-lang/rust/blob/1.83.0/library/std/src/sys/pal/unix/stack_overflow.rs#L360-L378

Past discussion about the guard page logic in Rust on NetBSD (and how
it is fundamentally similar on many other operating systems):

https://github.com/rust-lang/rust/pull/122002