Booting Enzian pt. 1: The BMC

This is the first post in a series where we look at how Enzian boots. In future posts we will look at the many stages the CPU and FPGA go through before we end up with a fully operational Enzian with the ECI coherency links up.

However, before we can get there, we need to boot a part of modern server systems that is often forgotten: The baseboard management controller, or BMC.

The BMC - a computer in the computer

The BMC in Enzian is an Enclustra Mercury ZX5 module which in turn is built around a Xilinx Zynq 7000 SoC. Here’s a photo of the module, with a Swiss 1-franc coin (about the size of a US quarter) for scale:

The Zynq SoC itself has 2 ARM Cortex A9 cores and an FPGA, while the Mercury module adds 1GB of RAM, 64MB of NOR flash and 512MB of NAND flash. The BMC is plugged into the main Enzian EATX board, next to where the QSFP28 daughter card sits. It has a USB port and a dedicated 1Gb/s Ethernet interface, both brought out on the Enzian rear panel.

First-stage boot loader

The BMC is powered by a 5V stand-by power rail so it boots as soon as the power supply is connected and executes its BootROM code (see Chapter 6 in the Zynq-7000 Manual).

The configuration of the boot mode pins on the SoC determine from where the BootROM code will load the first stage bootloader. The Enzian mainboard pinstraps the BMC such that the boot device is the Quad-SPI NOR flash. From there the BootROM code loads the first-stage bootloader, which in our case is the second program loader (SPL) for U-Boot

The SPL in turn loads a flattened device tree (FIT) image from the same flash memory, which contains a bitstream for the BMC’s FPGA and the U-Boot image. The SPL first programs the FPGA and then goes on to load U-Boot itself.

Strictly speaking, the Zynq boot process does not require the BMC’s FPGA to be programmed this early. However, on the current revision of Enzian this is necessary: all the serial consoles, including the one for the BMC itself, are routed through the BMC FPGA so until this is programmed we are flying blind. We definitely want a serial console when we enter U-Boot, so our SPL has to program the BMC FPGA. We’d probably change this in a future version of the Enzian mainboard, but it works for now.

U-Boot

If all went well, we now have the UARTs routed to the USB connector on the Enzian, and can get serial output from the BMC. U-Boot now starts:

FPGA image loaded from FIT


U-Boot 2020.01 (Dec 10 2021 - 10:55:23 +0000) Enclustra Mercury

CPU:   Zynq 7z015
Silicon: v3.1
Model: Enclustra ZX5-15-2I-D10 SOM
DRAM:  ECC disabled 1 GiB
Flash: Flash MIO pins configured to NAND mode
0 Bytes
NAND:  512 MiB
MMC:
Loading Environment from SPI Flash... Flash MIO pins configured to QSPI mode
SF: Detected s25fl512s_256k with page size 256 Bytes, erase size 256 KiB, total 64 MiB
OK
In:    serial@e0001000
Out:   serial@e0001000
Err:   serial@e0001000
SF: Detected s25fl512s_256k with page size 256 Bytes, erase size 256 KiB, total 64 MiB
Net:
ZYNQ GEM: e000b000, mdio bus e000b000, phyaddr 3, interface rgmii-id
eth0: ethernet@e000b000
Hit any key to stop autoboot:  0
Mercury>

Our current default for U-Boot is to boot a Linux distribution from NAND flash. On the line that reads:

Flash: Flash MIO pins configured to NAND mode

– we can see a peculiarity of the Enclustra Mercury module: the SPI NOR flash controller and the NAND flash controller share pins. To access the NAND flash U-Boot therefore needs to first reconfigure the pin multiplexers. It can then load the Linux kernel and a devicetree and boot it. The root file system resides in a UBI partition on the NAND flash as well but we let Linux deal with mounting it.

U-Boot can do more than load Linux from NAND flash though: The auto-boot process can be interrupted, dropping us into the U-Boot console. We can now load Linux and the device tree from somewhere else, for example a TFTP server over the network or a memory stick in the BMC’s USB connector. We can also load a new bitstream if we so desire, which is handy for BMC software development or updating the OS and bitstream on the BMC’s flash.

Instead of Linux we can of course also boot an entirely different OS on the BMC. One of our research topics is how to make BMC software more secure and trustworthy. In exploratory projects we have successfully booted seL4 on the BMC and we are investigating how such a trustworthy BMC firmware/software stack would look like. We’ll write about that in a future post.

BMC Linux

For now, the Enzians at ETH Zurich run Linux on their BMCs, specifically OpenBMC which is built using the Yocto Project. The rest of the default boot sequence is therefore not really Enzian specific: first the kernel comes up, configures the hardware including booting the second core and mounts the root file system residing on NAND flash. Then systemd takes over and orchestrates the rest of the system startup before we get dropped into the Linux login prompt:

Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) 0.1.0 enzianbmc ttyPS1

enzianbmc login:

And there it is. Note that, at this stage, we haven’t even powered up the rest of the Enzian board, including the ThunderX-1 CPU and the big FPGA…