Re: Booting on a Powerbook6,8 (was Re: Installing 8.0 on a Powerbook6,8)

[Jason Thorpe <thorpej%me.com@localhost>]

> On Feb 12, 2021, at 1:43 PM, Valery Ushakov <uwe%stderr.spb.ru@localhost> wrote:
> 
> This is probably not the specific node, see e.g. Joerg's mail where if
> was failing ofr him in a different OF_finddevice:
> 
>  http://mail-index.netbsd.org/port-macppc/2019/10/30/msg002659.html
> 
> 
> So between the weird xcf vs elf issue, the libsa allocator issue, and
> the observed OF_finddevice issues I suspect that something goes awry
> at some point when we interact with OFW and all of the above are just
> different manifestations of the same underlying problem.  But my
> ppc-fu is weak, apple's ofw is actively hostile to inspection as it
> has the forth headers stripped, and I don't have a serial console on
> my test mini g4, so it was quite awkward to work on this even when me
> and the mini where in the same epsilon neighborhood (and it's
> impossible now when I'm not).

Ok, so I spent a little more time working on this and I think it boils down to the fact that we just don't implement a correct virt-mode OpenFirmware client on PowerPC.

Some history (forgive me if I get some of the details wrong, I'm really digging back into the archives here)...

Back when Wolfgang originally committed the PowerPC port, I believe he was working on a FirePower MX (I also had one of these, and met up with a few of the FirePower engineers at their building in Redwood City).  These were PReP type systems with an OpenFirmware implementation from FirmWorks.  The FirePower ran Windows NT (in little-endian mode) in addition to the NetBSD PowerPC port (in big-endian mode).

Wolfgang originally used the real-mode OpenFirmware client interface.  This is by far the simplest way to interact with OpenFirmware, because you just switch off the MMU when you call it (well, and make sure you're running on a 1:1-mapped stack that's valid with the MMU on or off).  Technically, you are supposed to implement ofw-to-client callbacks for real-mode if the client is managing physical memory, but apparently Wolfgang got away without doing so because the kernel was primarily just calling "getprop", "peer", and "child" and the like to navigate around the device tree, which generally don't require the firmware to allocate memory.

Unfortunately, the early FirmWorks implementation had "some bugs" (and the FirePower unit I had definitely qualified as "early").  One of the issues was that some versions would leave the MMU turned on (and I believe had a BAT table that just mapped the entire address space 1:1) even if real-mode? was set to true.  This is the origin of FIRMWORKSBUGS and ofwreal.S, which sort of emulates real-mode by saving off the kernel MMU context, re-loading the MMU context that the firmware was using, and when switching back to the kernel's MMU context when the firmware call returned.

Anyway, at some point, we switched to just using virt-mode, because it's actually the BEST way to interface with OpenFirmware, but continued to do the MMU-swap-a-roo because, I guess, it was just easier.  Note at no time, at least to my recollection, did we implement the client callbacks you're supposed to provide when using the virt-mode client interface.

Anyway, these days, we actually go through the exercise of getting the MMU translations that OpenFirmware is using and entering those into an "ofw_pmap", and doing the rough equivalent of pmap_activate(&ofw_pmap) before jumping to the OpenFirmware entry point.  This means that whatever translations were in use by the firmware before the kernel was started will remain valid when calling into the firmware.  Cool.

But it seems we still don't seem to implement the client callbacks.  And herein lies the problem.  Quoth the PowerPC Processor Binding to IEEE 1275-1994 Revision 2.1 (November 6, 1996):

[Section 4.2.6. Client Interface (Virtual-Mode)]
.
.
.
Hence there //shall// be callback services provided by the client for use by Open Firmware for such actions; see section 8.5.2.

Now, because we don't provide a callback to the firmware, the firmware is just assuming that it still controls the MMU.  Normally, at least for the really simple stuff that we're doing, this is fine because the device tree and a bunch of other things were already mapped by the firmware because hey, we booted!

But!  What I think is happening on my system is that the MMU is getting touched by a way the firmware's not expecting and this happens to be causing *trouble* on my machine.  It would actually be instructive for me to run the experiment of removing the 1GB DIMM and booting with just the 256MB that's soldered onto the mainboard.  Anyway, the spec is pretty clear, if the client is going to touch the MMU *and* call into the firmware, then the client has to have registered a client call back between "taking over MMU" and "calling into the firmware after doing so".  This way any translations that the firmware needs to establish while the client has control of the MMU will be properly handled by the client (i.e. the kernel).

Anyway, I think that OpenBSD boots on this machine because they're just doing the MMU-switch-a-roo all the time.  This is a situation of "they're also getting lucky, just slightly luckier" ... because they're saving and restoring the entire MMU context on each firmware call, they're skirting the problem. (BUT, interestingly enough, they are NOT the ISI or DSI vectors which means jumping into outer-f***ing-space if firmware happens to need a BAT or PTE spill. You've got to ask yourself one question. Do I feel lucky? I dunno, maybe the firmware is swapping the vectors at entry/exit like it's supposed to do in real-mode?).  They also make the offending call that blows up my machine at a slightly earlier point in the bootstrap process, and bypass the save/restore dance around calling the firmware entry point at that particular stage of the process.

Anyway, the correct solution is to properly implement a complete virt-mode client interface.  This means:

- Pre-allocating memory for the pmap data structures that would hold the firmware's mappings, using the "claim" and "map" methods.

- Fetching all of the translations, populating the kernel pmap with those translations using the pre-allocated structures.

- Registering a callback once we're running on the kernel's MMU context, and implementing the "map", "unmap", and "translate" client callbacks (so as to perform those actions on behalf of OpenFirmware since we told it we own the MMU).

Once that is done, then calling OpenFirmware is a simple matter ... it's just a function call to a library, nothing more.  All of the calling glue in assembly can simply go away completely.  There are a couple of correct examples in the tree for other platforms:

- The shark port does *exactly* the correct thing (kudos to the folks at DEC), although it's of course following the ARM processor binding, so some of the details are slightly different).

- sparc64 does the right stuff too, although that one is obfuscated a bit behind their "prom_*()" API, which hides some of the differences because the older OpenProm and the properly-1275 OpenBOOT.

So while examples exist, it's not a small task, unfortunately, and is going to require the early bootstrap code in the PowerPC port to be refactored somewhat.  But I think it is the correct way to solve the problem (and will probably eliminate the need for some of the strange special cases we have scattered about).

-- thorpej