1. Introduction
  2. Checking we're all ready and set
  3. Running the very first test
  4. Dissecting the test of interest
  5. A quick look at ADD instruction test
  6. Diving into the instruction execution
  7. Markdown enters the scene
  8. Modifying the pseudocode

Introduction

This tutorial intends to shed some light at the first steps for a newcomer. Note that this tutorial is a work in progress; feel free to update it. This tutorial assumes that the environment and and repositories are set. The information on environment can be found at HDL workflow page, and an easy set of commands is at the devscripts page

In this tutorial, we will perform these steps:

  1. Checking we're all ready and set.
  2. Running the first test and observing its results.
  3. Updating the test so that it enables/disables some functions.

Note that this tutorial is ISA-centric, since the idea of adding it was born by the times there were several newcomer-oriented tasks around the decoder. Many of key concepts, however, are not ISA-specific, so there is a hope that this guide can be helpful for other tasks.

Checking we're all ready and set

Once we established the environment and cloned openpower-isa repository, we can run the first test we'll use as an example to observe. Since we're starting from decoder, we will work with openpower-isa repository. If you followed the HDL guidelines strictly, you can chroot into libresoc environment and work with the repository immediately, by changing the directory to ${HOME}/src/openpower-isa.

schroot -c libresoc /bin/bash
cd "${HOME}/src/openpower-isa"

If for some reason openpower-isa repository is not yet in your src subdirectory, you need to clone the relevant repository. The recommended way to do it is to use hdl-dev-repos script.

The environment is quite newcomer-friendly: the test we intend to dissect is implemented in pure Python, so there is no need to build anything before running the test. All we need is a good ol' Python; however, before proceeding, make sure that Python is at least of version 3.7.2.

python3 --version

If the version is less than 3.7.2, consider re-visiting the HDL workflow page and checking that all preparations are done correctly. Again, the preferred way is using simple scripts to keep the life easy. In this case, the script of interest is install-hdl-apt-reqs.

Running the very first test

Once we have the repository and the needed Python version, we can launch our first test suite (assuming we're already inside openpower-isa repository):

python3 src/openpower/decoder/isa/test_caller.py > /tmp/log

The test takes a while to complete; once it's ready, you should observe that it exits with success, and all tests from suite report no errors. In case some tests report failures, consider raising a question in #libre-soc on irc.libera.chat, or in mailing list. If you choose the latter, again, consider visiting HDL workflow page, to be aware of mailing lists etiquette and preferences. Note, however, that some tests can be skipped; this is acceptable.

If everything works fine, that's a good sign; let's proceed to the first routine we'd like to dissect.

Dissecting the test of interest

The script we launched contained several tests; amongst others, it checks that addition instruction works correctly. Since we're going to look through this instruction, let's for this time disable other tests so that we would run only test related to addition. Let's edit the test_caller.py script and mark all tests except addition one as those which must be skipped. This goal can be achieved by two ways.

  1. The first option is to rename all methods in DecoderTestCase class so that the names start with prefix other than test_.
  2. Alternatively, and this is the recommended way, tests can be temporarily disabled via @unittest.skip decorator applied to the method. This method is mentioned in HDL workflow as well (section 10.4).

Regardless of the option considered, test_add, which we're looking at, should be kept intact. Once all tests but test_add are renamed or skipped, re-run the test:

python3 src/openpower/decoder/isa/test_caller.py > /tmp/log

This time the suite should complete much faster. Let's observe the log and look for a line call add add. This line comes from src/openpower/decoder/isa/caller.py, and gives us an important information: we're calling an add instruction, and its assembly mnemonic is add as well.

So far so good; we dropped other tests, and now look at the test for add instruction. Now we're ready to check how the instruction behaves.

A quick look at ADD instruction test

Let's return to the test and the logs. What the test for add instruction does? For reader's convenience, the overall test_add code is duplicated here:

    def test_add(self):
        lst = ["add 1, 3, 2"]
        initial_regs = [0] * 32
        initial_regs[3] = 0x1234
        initial_regs[2] = 0x4321
        with Program(lst, bigendian=False) as program:
            sim = self.run_tst_program(program, initial_regs)
            self.assertEqual(sim.gpr(1), SelectableInt(0x5555, 64))

What do we see here? First of all, we have an assembly listing, consisting of exactly one instruction, "add 1, 3, 2". Then we establish the initial values for registers we're going to work with. After that, we instantiate the program with the assembly listing, execute the program, and compare the state of the first general-purpose register (aka GPR) with some predefined value (64-bit integer with value 0x5555).

Now let's turn to logs to see how they correspond to what we see. The lines of interest are reading reg RA 3 0 and reading reg RB 2 0. These, unsurprisingly, are two registers (3 and 2) which act as input parameters to add instruction; the result is then placed as an output into register 1.

Note that the initial values for registers 3 and 2 are 0x1234 and 0x4321 respectively, and this matches to the input parameters in the logs:

inputs [SelectableInt(value=0x1234, bits=64), SelectableInt(value=0x4321, bits=64)]

The simulator performs the actual computation, obtaining the result, and then updates the general-purpose register we used as an output parameter:

results (SelectableInt(value=0x5555, bits=64),)
writing gpr 1 SelectableInt(value=0x5555, bits=64) 0

In the end, we see that our assertion indeed passes:

__eq__ SelectableInt(value=0x5555, bits=64) SelectableInt(value=0x5555, bits=64)

You can play around the test, e.g. modify the input/output registers (there are 32 GPRs, so there's a plethora of combinations possible). In the next chapter, we're going to take a deeper look and cover some bits of implementation.

Diving into the instruction execution

One of interesting aspects we saw in the previous chapters is that whenever the test executes (or, more precisely, simulates) some instructions, there's some logic beneath these actions. Let's take a look at execution flow; for now, we won't dive into that many details, but rather take a quick look. We already saw that there are some logs coming from the src/openpower/decoder/isa/caller.py script; but how do we end up there? By the time of writing, all tests inside test_caller.py use internal method called run_tst_program. This method, in turn, calls run_tst, and this is the place where the magic happens. In process nested function, we actually simulate that our instructions are executed one-by-one, literally calling yield from simulator.execute_one(). And this method inside the simulator instance belongs to src/openpower/decoder/isa/caller.py script. Inside execute_one method, the most crucial part is yield from self.call(opname) call; and, if we take a look inside of the call method, we will see that, aside of the aforementioned log (log("call", ins_name, asmop)), this function also takes care of the rest of the magic. This includes a lot of manipulations before and after executing instruction, but the crucial part is quite simple:

        # execute actual instruction here (finally)
        log("inputs", inputs)
        results = info.func(self, *inputs)
        log("results", results)

The part of the most interest is info.func call; we'll take a look at it in the next chapter.

Markdown enters the scene

If we investigate the info.func object, we'll discover that, in case of our add instruction, it is actually fixedarith.op_add function. Where does it come from?

This function is defined inside fixedarith.py, which is generated from the corresponding Markdown file, fixedarith.mdwn. On the first glance, one can find an idea to generate the code from the documentation somewhat surprising; however, this is perfectly reasonable, since we try to stay as close to the documentation as possible. Some Markdown files, like fixedarith.mdwn, contain the pseudocode in exactly the same form as it is present in the ISA docs. Let's take a look at fixedarith.mdwn and find our instruction.

As we can see in # Add section, there are four add variants:

  • add RT,RA,RB (OE=0 Rc=0)
  • add. RT,RA,RB (OE=0 Rc=1)
  • addo RT,RA,RB (OE=1 Rc=0)
  • addo. RT,RA,RB (OE=1 Rc=1)

All variants are covered in the relevant OpenPOWER ISA. As of time of writing, the most recent edition is 3.0C, and it is available here:

https://ftp.libre-soc.org/PowerISA_public.v3.0C.pdf

The instructions of our interest can be found exactly where we'd expect them to be found, at chapter 3.3: Fixed-Point Facility Instructions. We can see here the instruction encoding, as well as its form (XO) and all information we already found in fixedarith.mdwn file.

All these details are available in some form as part of info variable we observed before (at caller.py file). The info variable is, actually, a simple instance of named tuple; the overall structure of this named tuple is left as an exercise for the reader.

Modifying the pseudocode

We won't dive into all gory details of the decoder, at least for now. What is important for us at this stage is, how can we affect the generated code? As we saw recently, the markdown file is somehow converted into the Python code; but how is the conversion done?

This is done by the script called pywriter. Again, we omit the exact details, since this script clearly deserves its own documentation; the only crucial information that pywriter script uses PLY (hint: Python-Lex-Yacc) in order to get the work done, and, thanks to PLY magic, it is able to convert ISA pseudocode into something more Pythonic.

For illustrative purposes, let's modify the pseudocode so that, instead of addition, it performs a subtraction. Let's find the pseudocode for add instructions inside fixedarith.mdwn...

Pseudo-code:

    RT <- (RA) + (RB)

and modify it to:

Pseudo-code:

    RT <- (RA) - (RB)

OK, we changed the pseudocode inside the Markdown, but it hasn't yet changed a word inside the corresponding Python file (fixedarith.py). If we attempt to re-run the unit test, nothing happens, since Python code is kept intact.

In order to force the source code re-generation, pywriter noall fixedarith command is used. Once pywriter completes its task, we can re-run the test, and it should fail. The logs show a different result.

Work out by hand what 0x1234 - 0x4321 in 64-bit unsigned arithmetic is, and change the assert in the unit test to match. Re-run: the test should now pass. These manipulations, again, are left as an exercise for the reader.

This chapter concludes the First Steps tutorial. We'll take a deeper look at pseudocode generation when we will dive into an example of modifying BCD instructions.