Fedora on Litex+RocketChip, Self-Hosting Demo

Gabriel L. Somlo <somlo at cmu dot edu>, 2023-01-11

1. Build Instructions

This demo was put together on a LambdaConcept ECPIX-5 85F board.

Pre-built images (ecpix-5 bitstream, 32-GB sdcard dd image) are available from: litex_rocket_fedora_prebuilt.tar.xz

FOSDEM 2023 slide deck here.

1.1. Bitstream

Once LiteX and its dependencies are fully installed, building the bitstream requires a simple, straightforward command line:

litex-boards/litex_boards/targets/lambdaconcept_ecpix5.py --build \
	--cpu-type rocket --cpu-variant fulld --sys-clk-freq 50e6 \
	--with-ethernet --with-sdcard \
	--yosys-flow3 --nextpnr-timingstrict --nextpnr-seed $RANDOM
	[ --csr-csv ./csr.csv --csr-json ./csr.json ]

NOTE: Drop --nextpnr-timingstrict, or manually iterate over the nextpnr portion of the build sequence to generate bitstream that fully passes timing constraints. This process generates a bitstream file named lambdaconcept_ecpix5.svf.

1.2. Firmware

Based on the generated csr.csv and/or csr.json files from earlier, as well as the sample .dts file included with the pre-elaborated Verilog code for RocketChip in pythondata-cpu-rocket, manually construct an ecpix5_fedora.dts device tree source file. This file must also include a chosen section which specifies the kernel command line and the location of the Fedora initial ram disk in physical memory. Compile the device tree table:

dtc -O dtb ecpix5_fedora.dts -o ecpix5_fedora.dtb

Next, grab OpenSBI and build the firmware blob, in a way that includes the above DTB file:

make CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=generic \
	FW_FDT_PATH=/path/to/ecpix5_fedora.dtb

This generates a file named fw_jump.bin, which is one of the three "blobs" loaded into DRAM by the built-in LiteX BIOS.

1.3. Fedora

Grab a pre-built Fedora-RISCV image, e.g.
Fedora-Developer-37-20221130.n.0-nvme.raw.img.xz.
The following steps will show how to lay it out onto a 32-GB SDCard:

# unpack
xz -d Fedora-Developer-37-20221130.n.0-nvme.raw.img.xz

# partition sdcard to look like this:
# echo -e "p\nq" | fdisk /dev/sdc
#   Cmd (m for help): Disk /dev/sdc: 29.73 GiB, 31927042048B, 62357504sec
#   Disk model: Micro SD/M2
#   Units: sectors of 1 * 512 = 512 bytes
#   Sector size (logical/physical): 512 bytes / 512 bytes
#   I/O size (minimum/optimal): 512 bytes / 512 bytes
#   Disklabel type: dos
#   Disk identifier: 0x67f480f9
#
#   Device     Boot   Start      End  Sectors  Size Id Type
#   /dev/sdc1          2048  2099199  2097152    1G  6 FAT16
#   /dev/sdc2       2099200 62333951 60234752 28.7G 83 Linux

# format sdcard partitions:
mkfs.vfat /dev/sdc1
mkfs.ext4 /dev/sdc2

# mount pre-built Fedora image (source) and sdcard partitions (destination):
modprobe nbd max_part=8
qemu-nbd -f raw --connect=/dev/nbd1 ./Fedora-Developer-37-20221130.n.0-nvme.raw.img
mkdir src_boot src_root dst_boot dst_root
mount /dev/nbd1p1 src_boot
mount /dev/nbd1p2 src_root
mount /dev/sdc1 dst_boot
mount /dev/sdc2 dst_root

# copy root partition:
rsync -a src_root/ dst_root/

# grab UUID of root and boot partitions on sdcard:
blkid /dev/sdc*

# edit dst_root/etc/fstab
#  update UUID for "/" and "/boot", also change fstype of "/boot" to "vfat"

# increase login timeouts:
sed -i '/Timeout/s/120/240/' dst_root/usr/lib/systemd/system/user@.service
sed -i '/TIMEOUT/s/60/200/; /TIMEOUT/s/^#//' dst_root/etc/login.defs
sed -i '/Grace/s/2/4/; /Grace/s/^#//' dst_root/etc/ssh/sshd_config

# populate boot partition:
cat > dst_boot/boot.json <<- "EOT"
	{
		"initrd.fed":  "0x83000000",
		"Image.fed":   "0x80200000",
		"fw_jump.bin": "0x80000000"
	}
	EOT

# unmount partitions:
for i in src_boot src_root dst_boot dst_root; do
  umount $i
done

NOTE: The /boot partition still requires the addition of the firmware, kernel image, and initial ramdisk files, which currently must be customized manually.

1.3.1. About the Kernel Command Line

Recall the chosen section of ecpix5_fedora.dts, specifically the bootargs line containing the kernel command line:

console=liteuart earlycon=liteuart,0x12006800 swiotlb=noforce ro root=/dev/mmcblk0p2 LANG=en_US.UTF-8
systemd.default_timeout_start_sec=360s systemd.unit=multi-user.target enforcing=0

An explanation is in order for some of those arguments:

systemd.default_timeout_start_sec=360s -- This is the equivalent of modifying DefaultTimeoutStartSec in the systemd configuration file, but with the added advantage of affecting both instances of systemd: the one running from the initial ram disk, and the one running from the real root filesystem after pivoting. Since we are on a very slow computer running at 50MHz, this is needed to tell systemd to be patient and not time out waiting for various services to start!
systemd.unit=multi-user.target -- This tells Fedora to boot in text (runlevel 3 equivalent) mode, instead of the default graphical (runlevel 5 equivalent) mode for which the downloaded image was configured. Once booted, this default can be changed by running systemctl set-default multi-user.target, after which the kernel command-line parameter is no longer necessary.
enforcing=0 -- This tells Fedora to start SELinux in non-enforcing mode, which is necessary because rsync-ing files from the downloaded image to the SD card likely destroyed the SELinux labels associated with the files and directories on the root partition. Running restorecon -Rp / after the system boots should fix that; however, relabeling takes about 9 hours on a system this slow, and will get broken again each time we mount the SD card on another system to copy files to or from it, so it's probably best to stay in non-enforcing mode during development and experimentation.

1.4. Kernel

The Fedora image referenced above ships with kernel 6.0.10-300.0.riscv64.fc37.riscv64, which does not include IRQ suport for LiteUART (the LiteX serial/tty interface). Also, some of the following list of default kernel configuration options cause the kernel to crash during boot on LiteX (specifics are still TBD):

CONFIG_PRINTK_INDEX=y
CONFIG_SOC_STARFIVE=y
CONFIG_ERRATA_THEAD=y
CONFIG_KEXEC_FILE=y
CONFIG_COMPAT_32BIT_TIME=y
CONFIG_MODULE_UNLOAD_TAINT_TRACKING=y
CONFIG_NF_FLOW_TABLE_PROCFS=y
CONFIG_FW_LOADER_COMPRESS_ZSTD=y
CONFIG_EFI_COCO_SECRET=y
CONFIG_NVME_AUTH=y
CONFIG_NVME_TARGET_AUTH=y
CONFIG_SCSI_FLASHPOINT=y
CONFIG_FB_EFI=y
CONFIG_IMA_KEXEC=y
CONFIG_CRYPTO_ECDH=y
CONFIG_SYSTEM_BLACKLIST_AUTH_UPDATE=y

Therefore, on a QEMU fedora-riscv64 VM deployed from the same pre-built downloaded disk image as above, build a custom kernel:

git clone https://github.com/litex-hub/linux
cd linux; git checkout litex-rebase
cp /boot/config-6.0.10-300.0.riscv64.fc37.riscv64 .config
make olddefconfig
# comment out the above CONFIG_* options
make -j8
make modules_install
make install

Copy the resulting /usr/lib/modules/XYZ folder to the root partition of the SD card (e.g., /dev/sdc2). Copy the resulting /boot/vmlinuz-XYZ and /boot/initramfs-XYZ.img to the /boot partition of the SD card as Image.fed and initrd.fed, respectively. Update ecpix5_fedora.dts to reflect the new size of the initrd image, and rebuild fw_jump.bin before also adding it to the /boot partition of the SD card.

2. Running Fedora on Litex+Rocket, on the ECPIX-5 board

This demo shows Fedora booting, installing the FOSS toolchain used to build its own bitstream in the first place, and running that toolchain to build a blinky bitstream for its own underlying board (the ECPIX-5), as shown in the following video.
NOTE: The md5sum of the top.svf bitstream, right before it is sent to the board, is shown as bae0d78c2132eaed400184681aabb618:

NOTE: Toward the end of the (slow, 4h:45m) ASCII recording, the md5sum of the generated top.svf bitstream matches the one deployed above, showing that it really was generated on the ECPIX-5 board itself!