Re: Installing NetBSD on Jetson Nano

[Dan MacDonald <allcoms%gmail.com@localhost>]

I installed Jetpack 4.5 a while back which changed the boot process
for the Nano and now allows booting from USB which the recent Armbian
releases take advantage of. The Jetpack 4.5 release notes say:

"Boot firmware for all Jetson Nano developer kits updated to relocate
boot firmware to integrated QSPI-NOR."

https://forums.developer.nvidia.com/t/jetpack-4-5-production-release-with-l4t-32-5/166475

That's as much as I can tell you about the Nano boot process.

On Sun, Jan 23, 2022 at 2:51 AM Jason Mitchell <jmitchel%bigjar.com@localhost> wrote:
>
> Okay, this is confusing to me. There's no U-Boot at all, unless it's further on. Where does the DOS partition start? Maybe hexdump up to the start of the DOS partition?
>
> HTH,
>
> Jason M.
>
> Get BlueMail for Android
> On Jan 21, 2022, at 5:17 PM, Dan MacDonald <allcoms%gmail.com@localhost> wrote:
>>
>> $ hexdump -C -n 4194304
>> Armbian_22.02.0-trunk.0016_Jetson-nano_jammy_edge_5.16.2.img
>> 00000000  fa b8 00 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |................|
>> 00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|
>> 00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|
>> 00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |.........|...t..|
>> 00000040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L.....|.........|
>> 00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>> *
>> 000001b0  00 00 00 00 00 00 00 00  5e 12 03 bd 00 00 00 00  |........^.......|
>> 000001c0  41 00 83 03 e0 ff 00 80  00 00 00 20 32 00 00 00  |A.......... 2...|
>> 000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>> *
>> 000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
>> 00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>> *
>> 00400000
>>
>> On Fri, Jan 21, 2022 at 9:04 PM Jason Mitchell <jmitchel%bigjar.com@localhost> wrote:
>>>
>>>
>>>  Dan,
>>>
>>>  This document says there are four separate boot loaders. I can't tell for certain, but they might also need to be on the boot media.
>>>
>>>  https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/bootflow_jetson_nano.html
>>>
>>>  The output of hexdump -C of the first 2 or 4 megabytes of the Jetson Armbian image should tell us if Armbian has extra boot code we need.
>>>
>>>  HTH,
>>>
>>>  Jason M.
>>>
>>>  Get BlueMail for Android
>>>  On Jan 20, 2022, at 2:49 PM, Dan MacDonald <allcoms%gmail.com@localhost> wrote:
>>>>
>>>>
>>>>  Hi Jason
>>>>
>>>>  I'm closer to getting NetBSD to boot thanks to your help but no dice yet!
>>>>
>>>>  I can see that the latest NetBSD current builds do have the required
>>>>  .dtb included for the Jetson Nano so its looking more promising but
>>>>  now I have two or three more questions.
>>>>
>>>>  There are now two images available for NetBSD arm64 current,
>>>>  arm64mbr.img.gz and arm64.img.gz. I think its pretty safe to presume
>>>>  that arm64.img is the UEFI image and hence not the one I want. I'd
>>>>  expect arm64mbr.img.gz is the one I should be using but it would be
>>>>  nice to know for sure because neither boots OOTB. The arm64mbr image
>>>>  is the only one that my Linux laptop can read a partition of, the boot
>>>>  partition.
>>>>
>>>>  Then we have the u-boot tarball which contains two files in
>>>>  uboot/share/u-boot/jetson-nano - u-boot  which is about 7 MB and
>>>>  u-boot-dtb.bin which is about 600 K. I have tried dd'ing both onto the
>>>>  arm64mbr image, one at a time and reflashing before trying the next,
>>>>  but neither booted. I also tried copying u-boot-dtb.bin into a few
>>>>  dirs on the boot partition - in the root dir, in the DTB dir and in
>>>>  DTB/Nvidia - no boot yet!
>>>>
>>>>  I asked in #netbsd and someone thinks I may have to use a USB cable to
>>>>  flash the u-boot-dtb.bin onto the nano but that was all they had to
>>>>  offer in the way of help and I've not found any guides to installing
>>>>  NetBSD on the Jetson TK1 yet either.
>>>>
>>>>  Thanks again for your help!
>>>>
>>>>  On Thu, Jan 20, 2022 at 12:23 PM Jason Mitchell <jmitchel%bigjar.com@localhost> wrote:
>>>>>
>>>>>
>>>>>
>>>>>   Dan,
>>>>>
>>>>>   Responses below.
>>>>>
>>>>>   HTH,
>>>>>
>>>>>   Jason M.
>>>>>
>>>>>   On 1/20/22 3:34 AM, Dan MacDonald wrote:
>>>>>>
>>>>>>
>>>>>>   Hi Jason
>>>>>>
>>>>>>   Thanks for your feedback.
>>>>>>
>>>>>>   So it sounds like I may be able to get away with writing the Tegra
>>>>>>   u-boot to the arm64 image, once I have an image that includes the
>>>>>>   tegra210-p3450-0000.dtb I require.
>>>>>>
>>>>>>   I have seen tegra210-p3450-0000.dtb on one of the NetBSD FTP sites. I
>>>>>>   don't suppose I might be able to copy that dtb onto the NetBSD 9.2
>>>>>>   arm64 image and expect it to work?
>>>>>
>>>>>
>>>>>
>>>>>   The dtb file needs to be on the DOS partition and it needs to be where
>>>>>   u-boot expects it. If you build the u-boot package you should be able to
>>>>>   see what dtb filename u-boot wants and what directory it should be in.
>>>>>   This will be in the u-boot config file that gets created in the work
>>>>>   directory under sysutils/u-boot-jetson-nano.
>>>>>
>>>>>>     I'd rather not have to but I
>>>>>>   presume I'm going to have to build the latest NetBSD from source in
>>>>>>   order to get an image that includes this dtb. Is there a guide to
>>>>>>   building the aarch64 port of NetBSD 10 anywhere?
>>>>>
>>>>>
>>>>>
>>>>>   I don't see why you would need to build current just to get the dtb
>>>>>   file. You might need to run -current to support the Jetson, but since
>>>>>   there are NetBSD daily builds, you should just be able to download an
>>>>>   image from:
>>>>>   http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/binary/gzimg/
>>>>>
>>>>>   If you want a guide to compiling NetBSD google (compiling the kernel
>>>>>   using build.sh). It's chapter 31 of the NetBSD guide (I think).
>>>>>
>>>>>>   I'm hoping Jared will see this and be able to give exact instructions
>>>>>>   so we can get this process documented either on the NetBSD Tegra page
>>>>>>   or on the NetBSD wiki. The NetBSD web site could really do with a
>>>>>>   overhaul- there seems to be lots of vital info missing and it looks
>>>>>>   like a relic from the 90s.
>>>>>>
>>>>>>   On Wed, Jan 19, 2022 at 10:37 PM Jason Mitchell <jmitchel%bigjar.com@localhost> wrote:
>>>>>>>
>>>>>>>
>>>>>>>   On 1/19/22 4:43 AM, Dan MacDonald wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>   I've not been able to find any instructions or confirmation that it is
>>>>>>>>   possible to install NetBSD on the Jetson Nano other than a NetBSD
>>>>>>>>   mailing list post from Jared McNeil saying that I would need to use
>>>>>>>>   the tegra210-p3450-0000.dtb:
>>>>>>>>
>>>>>>>>   http://mail-index.netbsd.org/port-arm/2020/01/31/msg006479.html
>>>>>>>>
>>>>>>>>   I’ve tried the NetBSD 9.2 evbarm-aarch64 image and it wouldn’t boot
>>>>>>>>   for me, likely because it doesn't include the tegra210-p3450-0000.dtb.
>>>>>>>>
>>>>>>>>   Would I need to build NetBSD 10 from source to install it on my Jetson
>>>>>>>>   Nano or are there pre-built NetBSD arm64 images somewhere that do
>>>>>>>>   include the required dtb and u-boot? I don't think the Jetson Nano
>>>>>>>>   supports UEFI.
>>>>>>>>
>>>>>>>>   Recent versions of Jetpack (the reference OS for Jetson devices) and
>>>>>>>>   Armbian let me boot straight from USB 3. Might this also be possible
>>>>>>>>   with NetBSD or will I only be able to boot from MicroSD?
>>>>>>>>
>>>>>>>>   Thanks
>>>>>>>
>>>>>>>
>>>>>>>   Hello,
>>>>>>>
>>>>>>>         From what I understand the boot process for ARM devices goes like this:
>>>>>>>
>>>>>>>   1) Device power on
>>>>>>>
>>>>>>>   1b) There could be a step here if there's some nvidia specific code on
>>>>>>>   the SPI flash or in ROM
>>>>>>>
>>>>>>>   2) Load u-boot from SPI flash or boot device*
>>>>>>>
>>>>>>>   3) Load EFI code on the DOS partition and execute it
>>>>>>>
>>>>>>>   4) The EFI code looks for a kernel on an FFS partition and executes it.
>>>>>>>
>>>>>>>         The questions I don't know the answer to are:
>>>>>>>
>>>>>>>   1) Does the Jetson Nano have u-boot installed in SPI flash?
>>>>>>>
>>>>>>>   2) If u-boot is present on the boot media and in SPI flash which is
>>>>>>>   loaded first?
>>>>>>>
>>>>>>>   3) Where does the Jetson expect u-boot to be on the boot drive
>>>>>>>
>>>>>>>         It looks like you're using a generic image which probably doesn't
>>>>>>>   have u-boot on it. There's a package in pkgsrc (and also in pkgsrc-wip)
>>>>>>>   containing u-boot for the Jetson nano (sysuttils/u-boot-jetson-nano). If
>>>>>>>   you downloaded that you could copy it onto a microSD card that contained
>>>>>>>   the base NetBSD ARM image (arm64.img.gz). I thought the package would
>>>>>>>   print the dd command necessary to install the u-boot file, but it
>>>>>>>   doesn't look like it does. I think:
>>>>>>>
>>>>>>>         dd if=u-boot of=/dev/*disk* seek=64     is the command for other
>>>>>>>   platforms.
>>>>>>>
>>>>>>>         To boot NetBSD off of a USB device I'm nearly certain you'll need
>>>>>>>   to load u-boot into the SPI flash. Maybe the Jetson will look for u-boot
>>>>>>>   on a USB device , I don't know.
>>>>>>>
>>>>>>>         My understanding of this is incomplete, but hopefully this will
>>>>>>>   shed some light on the subject.
>>>>>>>
>>>>>>>   HTH,
>>>>>>>
>>>>>>>   Jason M.
>>>>>>>
>>>>>>>   * u-boot lives in unallocated space between the MBR and the DOS
>>>>>>>   partition starting at 0x8000 on the disk (at least on some ARM chips)
>>>>>
>>>>>
>>>>>