Tutorial for setting up Microwatt chroot and running simulations

Useful Links (External):

Useful links (Libre-SOC):

Other Tutorials (Libre-SOC):

  • First steps for working with PowerISA instructions Libre-SOC page: firststeps

Video Tutorial

43min tutorial was made and uploaded to Youtube, covering some of the material you'll find on this page.

Development environment scripts

If you haven't already, clone Libre-SOC's development environment setup scripts. These are bash scripts, and greatly simplify the time it takes to create a:

  • Stable environment
  • With all software and libraries at specific versions (which are known to work).

These attributes are absolutely critical, and no support will be provided, unless you use these scripts to setup a development environment. This helps us fix any bugs in the scripts, and make sure everyone runs on the same page.

$ git clone https://git.libre-soc.org/git/dev-env-setup.git

Do look through the code before running any of those scripts. This is your own legal responsibility (to not run arbitrary code off of the internet) and we take no responsibility or accept any liability whatsoever for your use or misuse of them.

It is expected for you to use Debian for the host OS (anything else is unsupported: many contributors have repeatedly gotten into trouble by not following this advice), while all the chroots - which are developed very specifically for "reproducible builds" - run Debian 10 (Buster).

Setting up chroot

Scripts we will be using for the setup are:

  • mk-deb-chroot, cp-scripts-to-chroot for chroot setup
  • install-hdl-apt-reqs, verilator-install, hdl-tools-yosys for working with Microwatt

(Current limitation for mk-deb-chroot, is that you must be the first user on the host machine, having user ID 1000.)

Commands to run in terminal to setup a new chroot environment for microwatt simulations.

$ cd dev-env-setup
$ sudo bash
# ./mk-deb-chroot microwatt
# ./cp-scripts-to-chroot microwatt
# exit
$ schroot -c microwatt
(microwatt):$ cd dev-env-setup
(microwatt):$ sudo bash
(microwatt):# ./install-hdl-apt-reqs
(microwatt):# ./verilator-install
(microwatt):# ./hdl-tools-yosys
(microwatt):# exit
(microwatt):$ cd ~/src/
(microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
(microwatt):$ cd microwatt
(microwatt):$ git checkout verilator_trace

Make sure verilator binaries in $PATH:

(microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
(microwatt):$ export GHDLSYNTH=ghdl

(GHDLSYNTH needs to be redefined because the Makefile has default ghdl.so, but somewhere else '.so' gets appended. You may see the following error if you don't redefine: ERROR: Can't load module ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**) IRC

Compiling the verilator sim for Microwatt

Verilator creates a fairly fast simulation by converting the HDL design to C++, and then compiling a binary which the user runs.

To compile the verilator simulation, first set verilator as the target for the makefile:

(microwatt):$ export FPGA_TARGET=verilator

Before compiling, you can change the THREADS variable in the makefile, which will allow the compiled verilator simulation binary to use more than 1 thread (make sure to check how many CPU threads you have before changing this!)

To compile the verilator simulation binary, call make with the microwatt-verilator rule.

(microwatt):$ make microwatt-verilator

Compiling hello world code

We need some code to actually run on the core, so start with the 'hello world'. Instructions assume you're still in the microwatt directory.

(microwatt):$ cd hello_world
(microwatt):$ make

A hello_world.bin should be generated (the final binary to be loaded), as well as an .elf file, and .hex (representing the binary data as hex text strings).

To view the symbol table (useful to see where various sections of the binary begin):

(microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
(microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf

-h shows just the section headers, -x shows all headers.

And to view the disassembly (great for learning about the PowerISA instructions, and for associating the binary hex with actual instructions), you can view the automatically generated hello_world.as file.

Command to generate the disassembly:

(microwatt):$ powerpc64le-linux-gnu-objdump -D hello_world.elf

For more information about objdump (common utility, not just for PowerISA), see the manual pages.

(microwatt):$ man powerpc64le-linux-gnu-objdump

The binary is ready to go, now it can be loaded into the simulation.

Simulation

Command line args

To find out the microwatt-verilator arguments, you can check with -h arg:

(microwatt):$ ./microwatt-verilator -h

Some of the arguments are explained in further sections.

Running

Run the microwatt-verilator binary, with hello_world/hello_world.bin as an argument:

(microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin

time is a utility you can use to measure how long it takes to run the sim.

A pretty ASCII art of a lightbulb should be printed, and then the user can type any characters, which will be echoed back. To end the simulation press Ctrl+C.

If no characters are appearing after about 20 seconds, stop the simulation, as there might be other issues.

Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge) takes 53 seconds to print the ASCII lightbulb.

On another dev's machine, ASUS KGPE D16, this takes just over a minute.

(You'll find that uart printout is one of the longer parts of the simulation in general.)

Analysing results after simulation

The following files will be generated during the sim:

  • bram.dump - Shows the PC address and instruction being executed. If the sim hangs without any printing, view this file, as the processor may have hit an exception etc. Grows in size as the sim runs.

  • bram.snapshot.[NUMBER], verilator.save.[NUMBER] - Snapshot files of the contents of bram and verilator model respectively. Can be used to resume the simulation. The number on the end corresponds to the tick time (i.e. bram.snapshot.1999990/verilator.save.1999990). First the verilator model is loaded, and then the bram contents are loaded. See lines #65-108 and #189-195 of the microwatt-verilator.cpp file. Pass the tick number on the end of the filename with the '-s' flag:

    (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990

You'll get a message like this:

loading hello_world/hello_world.bin at 0x0 size 0x1888
loading bram.snapshot.1999990 at 0x0 size 0x10000000
restored at 1999990

These snapshots are generated at intervals of every 2,000,000 ticks.

  • microwatt-verilator.vcd - GTKWave waveform file, allowing you to look at processor signals and transitions during simulation. Pass -d flag to microwatt-verilator binary:

    (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d

NOTE: Trace dumping will generate a large VCD file (about 6GB for the hello world example)!

If you want GTKWave to load it faster, convert to fst first:

(microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
(microwatt):$ gtkwave microwatt-verilator.fst

Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific to the GTKWave tool.

Micropython

The Microwatt repo comes with a pre-compiled micropython binary (version 1.12), which you can try out after confirming 'hello world' works. Bear in mind, not all features of python will be available. Such as floating-point numbers.

For micropython to work, you'll need to increase the RAM size in the makefile. Go to the microwatt-verilator makefile, and comment out the following lines:

MEMORY_SIZE=8192
RAM_INIT_FILE=hello_world/hello_world.hex

And uncomment the following:

MEMORY_SIZE=393216
RAM_INIT_FILE=micropython/firmware.hex

This will increase the RAM size from 8KiB to 384KiB. The RAM_INIT_FILE in these examples isn't doing anything, however good practice to follow.

Clean up generated files, and recompile:

(microwatt):$ make clean
(microwatt):$ make microwatt-verilator

Once the binary has been built, run the same way as before, but point to the micropython firmware binary:

(microwatt):$ microwatt-verilator micropython/firmware.bin

On the same system as above, with 1 thread, it took 49 seconds to get to the micropython shell.

Verilator runtime commands

A few examples:

# Show the version of verilator being used
(microwatt):$ ./microwatt-verilator +verilator+version

Building microwatt-verilator using the Libre-SOC core

In the Makefile, you need to set EXTERNAL_CORE to true, and copy the generated core from soc repo to microwatt. (If you use a separate chroot to generate Libre-SOC cores, then you'll need to copy from that chroot to microwatt chroot from host.

cd /path/to/soc
make microwatt_external_core
cp external_core_top.v /path/to/microwatt

Then compile verilator sim binary as before:

cd ~/src/microwatt/
export FPGA_TARGET=verilator
export GHDLSYNTH=ghdl
make microwatt-verilator

Running Linux kernel

To run Linux on Microwatt, you'll need two binaries:

  • The sdram_init.bin, which is easy to compile (no additional software required).

  • The dtbImage.microwatt device tree Linux kernel. This can be compiled (see below), or a copy can be downloaded from: https://ftp.libre-soc.org/dtbImage.microwatt.

Like with the micropython example, you'll need to increase the RAM size in the makefile, and recompile the microwatt-verilator binary.

Uncomment the following:

MEMORY_SIZE=536870912

Which will change the RAM size to 512KiB.

As there is no dtbImage.microwatt.hex, you can leave RAM_INIT_FILE unchanged.

Building the kernel - TODO:

(Please don't build the kernel yourself, until you've tested with the existing kernel linked above!)

On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu, you can omit CROSS_COMPILE and ARCH in this case

apt install gcc-powerpc64le-linux-gnu
apt install flex bison lz4
git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
cd microwatt
wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
patch -p1 < microwatt-linux-5.7.patch
wget https://ftp.libre-soc.org/rootfs.cpio
CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt

This will produce a file microwatt/arch/powerpc/boot/dtbImage.microwatt

Building sdram_init.bin

This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if cross compilation is used. It is assumed you're already in ~/src/microwatt/ directory.

(microwatt):$ cd litedram/gen-src/sdram_init/
(microwatt):$ make

The resulting binary will be in the obj/ directory.

Running the simulation

Make sure to return back to src/microwatt/.

(microwatt):$ cd ~/src/microwatt/
(microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
(microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt

This will take some time...

Sysconn information

TODO WIP integrate from https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html

Sysconn is a module which includes information about the SoC, and the info is printed at the start of the simulation.

Time benchmarks

microwatt-verilator was compiled with 3 threads for faster simulation.

  • Time to finish printing Sysconn info: about 1min
  • Time to allocate bytes to kernel: ?
  • Time to login prompt: about 1 hour
  • Time to user shell: ?

TODO: buildroot

  • https://github.com/shenki/buildroot/commits/microwatt
  • https://codeconstruct.com.au/docs/microwatt-orangecrab/

FPGA Development - TODO: Need checking

Building the bitstring for OrangeCrab

cd microwatt
export FPGA_TARGET=ORANGE-CRAB
export GHDLSYNTH=ghdl
make microwatt.bit

flashing the bitstring to the OrangeCrab

make prog # this will run OpenOCD

Notes for ulx3s

notes for how to compile for ulx3s

git clone https://github.com/kost/fujprog
(follow build procedure shown in fujprog README)
git clone https://git.libre-soc.org/git/microwatt.git
git checkout -b verilator_trace
export FPGA_TARGET=ulx3s
make microwatt.svf
fujprog microwatt.svf

Notes for nextpnr-xilinx

superceded: see page nextpnr-xilinx and devscript https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD

for compiling nextpnr-xilinx and making it useable for nmigen to compile for the digilent arty-a7-100t, requires a little futzing around, using the symbiflow version of prjxray-db instead of the one recommended as a submodule

git clone https://github.com/gatecat/nextpnr-xilinx
cd nextpnr-xilinx
git checkout cd8b15db6ff5c1a7f10a9e
git submodule init
git submodule update
cd xilinx/external
mv prjxray-db prjxray-db-no
git clone https://github.com/SymbiFlow/prjxray-db
cd prjxray-db
git checkout 0a0addedd73e7
cp ./artix7/xc7a100t/*.json \
   ./artix7/xc7a100tcsg324-1
cd ../../..
cmake -DARCH=xilinx .
make
make install
python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
mkdir -p /usr/share/nextpnr/xilinx-chipdb
cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
cp -aux xilinx/external/prjxray-db /usr/share/nextpnr

Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface

This uses OrangeCrab's built-in USB interface, rather than needing a separate USB-serial adapter. see the following for further details:

running orangecrab-examples before flashing microwatt

See https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules

If the OrangeCrab is running in DFU mode, lsusb will show:

1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2

OrangeCrab has two DFU devices:

Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"

Then clone and patch orangecrab-examples:

git clone https://github.com/orangecrab-fpga/orangecrab-examples
patch -p1 < orangecrab-examples.diff

To build and flash the example:

pushd orangecrab-examples/nmigen
python3 blink.py
popd
sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0