Fedora on Litex+RocketChip, Self-Hosting Demo

Gabriel L. Somlo <somlo at cmu dot edu>, 2023-01-11

1. Build Instructions

This demo was put together on a LambdaConcept ECPIX-5 85F board.

Pre-built images (ecpix-5 bitstream, 32-GB sdcard dd image) are available from: litex_rocket_fedora_prebuilt.tar.xz

FOSDEM 2023 slide deck here.

1.1. Bitstream

Once LiteX and its dependencies are fully installed, building the bitstream requires a simple, straightforward command line:
litex-boards/litex_boards/targets/lambdaconcept_ecpix5.py --build \
	--cpu-type rocket --cpu-variant fulld --sys-clk-freq 50e6 \
	--with-ethernet --with-sdcard \
	--yosys-flow3 --nextpnr-timingstrict --nextpnr-seed $RANDOM
	[ --csr-csv ./csr.csv --csr-json ./csr.json ]
NOTE: Drop --nextpnr-timingstrict, or manually iterate over the nextpnr portion of the build sequence to generate bitstream that fully passes timing constraints. This process generates a bitstream file named lambdaconcept_ecpix5.svf.

1.2. Firmware

Based on the generated csr.csv and/or csr.json files from earlier, as well as the sample .dts file included with the pre-elaborated Verilog code for RocketChip in pythondata-cpu-rocket, manually construct an ecpix5_fedora.dts device tree source file. This file must also include a chosen section which specifies the kernel command line and the location of the Fedora initial ram disk in physical memory. Compile the device tree table:
dtc -O dtb ecpix5_fedora.dts -o ecpix5_fedora.dtb
Next, grab OpenSBI and build the firmware blob, in a way that includes the above DTB file:
make CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=generic \
This generates a file named fw_jump.bin, which is one of the three "blobs" loaded into DRAM by the built-in LiteX BIOS.

1.3. Fedora

Grab a pre-built Fedora-RISCV image, e.g.
The following steps will show how to lay it out onto a 32-GB SDCard:
# unpack
xz -d Fedora-Developer-37-20221130.n.0-nvme.raw.img.xz

# partition sdcard to look like this:
# echo -e "p\nq" | fdisk /dev/sdc
#   Cmd (m for help): Disk /dev/sdc: 29.73 GiB, 31927042048B, 62357504sec
#   Disk model: Micro SD/M2
#   Units: sectors of 1 * 512 = 512 bytes
#   Sector size (logical/physical): 512 bytes / 512 bytes
#   I/O size (minimum/optimal): 512 bytes / 512 bytes
#   Disklabel type: dos
#   Disk identifier: 0x67f480f9
#   Device     Boot   Start      End  Sectors  Size Id Type
#   /dev/sdc1          2048  2099199  2097152    1G  6 FAT16
#   /dev/sdc2       2099200 62333951 60234752 28.7G 83 Linux

# format sdcard partitions:
mkfs.vfat /dev/sdc1
mkfs.ext4 /dev/sdc2

# mount pre-built Fedora image (source) and sdcard partitions (destination):
modprobe nbd max_part=8
qemu-nbd -f raw --connect=/dev/nbd1 ./Fedora-Developer-37-20221130.n.0-nvme.raw.img
mkdir src_boot src_root dst_boot dst_root
mount /dev/nbd1p1 src_boot
mount /dev/nbd1p2 src_root
mount /dev/sdc1 dst_boot
mount /dev/sdc2 dst_root

# copy root partition:
rsync -a src_root/ dst_root/

# grab UUID of root and boot partitions on sdcard:
blkid /dev/sdc*

# edit dst_root/etc/fstab
#  update UUID for "/" and "/boot", also change fstype of "/boot" to "vfat"

# increase login timeouts:
sed -i '/Timeout/s/120/240/' dst_root/usr/lib/systemd/system/user@.service
sed -i '/TIMEOUT/s/60/200/; /TIMEOUT/s/^#//' dst_root/etc/login.defs
sed -i '/Grace/s/2/4/; /Grace/s/^#//' dst_root/etc/ssh/sshd_config

# populate boot partition:
cat > dst_boot/boot.json <<- "EOT"
		"initrd.fed":  "0x83000000",
		"Image.fed":   "0x80200000",
		"fw_jump.bin": "0x80000000"

# unmount partitions:
for i in src_boot src_root dst_boot dst_root; do
  umount $i
NOTE: The /boot partition still requires the addition of the firmware, kernel image, and initial ramdisk files, which currently must be customized manually.
1.3.1. About the Kernel Command Line
Recall the chosen section of ecpix5_fedora.dts, specifically the bootargs line containing the kernel command line:
console=liteuart earlycon=liteuart,0x12006800 swiotlb=noforce ro root=/dev/mmcblk0p2 LANG=en_US.UTF-8
systemd.default_timeout_start_sec=360s systemd.unit=multi-user.target enforcing=0
An explanation is in order for some of those arguments:

1.4. Kernel

The Fedora image referenced above ships with kernel 6.0.10-300.0.riscv64.fc37.riscv64, which does not include IRQ suport for LiteUART (the LiteX serial/tty interface). Also, some of the following list of default kernel configuration options cause the kernel to crash during boot on LiteX (specifics are still TBD):
Therefore, on a QEMU fedora-riscv64 VM deployed from the same pre-built downloaded disk image as above, build a custom kernel:
git clone https://github.com/litex-hub/linux
cd linux; git checkout litex-rebase
cp /boot/config-6.0.10-300.0.riscv64.fc37.riscv64 .config
make olddefconfig
# comment out the above CONFIG_* options
make -j8
make modules_install
make install
Copy the resulting /usr/lib/modules/XYZ folder to the root partition of the SD card (e.g., /dev/sdc2). Copy the resulting /boot/vmlinuz-XYZ and /boot/initramfs-XYZ.img to the /boot partition of the SD card as Image.fed and initrd.fed, respectively. Update
ecpix5_fedora.dts to reflect the new size of the initrd image, and rebuild fw_jump.bin before also adding it to the /boot partition of the SD card.

2. Running Fedora on Litex+Rocket, on the ECPIX-5 board

This demo shows Fedora booting, installing the FOSS toolchain used to build its own bitstream in the first place, and running that toolchain to build a blinky bitstream for its own underlying board (the ECPIX-5), as shown in the following video.
NOTE: The md5sum of the top.svf bitstream, right before it is sent to the board, is shown as bae0d78c2132eaed400184681aabb618:

NOTE: Toward the end of the (slow, 4h:45m) ASCII recording, the md5sum of the generated top.svf bitstream matches the one deployed above, showing that it really was generated on the ECPIX-5 board itself!