0. 概述

0.1 RISC-V 官方的测试分两个层级

Unit tests for RISC-V processors: riscv-tests
RISC-V Architecture Test SIG(SIG: Special Interest Group)
- 核心是 RISCOF, RISCOF 支撑了架构测试的工作流。
  - RISCOF [Repo] [Doc] is a RISC-V Architectural Test Framework.
  - RISCOF - The RISC-V Compatibility Framework is a python based framework which enables testing of a RISC-V target (hard or soft implementations) against a standard RISC-V golden reference model using a suite of RISC-V architectural assembly tests.
- 相关的其他库
  - RISCV-CONFIG [Repo] [Doc]
  - RISCV-ISAC [Repo] [Doc]
  - RISCV-CTG [Repo] [Doc]

0.2 相关缩写

riscv-arch-test： RISC-V Architecture Test
riscv-isac: RISC-V ISA Coverage
riscv-ctg: RISC-V Compatibility Test Generator
RISCOF: RISC-V Compatibility Framework

1. 单元测试

使用 riscv-test 生成的测试程序，先要搞清楚的问题：
- 怎样退出？ ---- target 中, 使用编号为 93 的 ecall 表示程序运行结束.
- 怎样判断结果？---- target 上 application 的退出值为 0 表示 pass。
生成单元测试 case

$ git clone https://github.com/riscv/riscv-tests
$ cd riscv-tests
$ git submodule update --init --recursive
$ autoconf
$ ./configure --prefix=$RISCV/target
$ make
$ make install

运行 riscv tests 中 rv32ui-p-* 的 test cases

#!/bin/bash

set -e

# Specify the folder path
# RISCV is the path of the RISCV toolchain installation
folder_path="$RISCV/target/share/riscv-tests/isa"

echo -e "\n========= Running $0 starting ========="

cargo build

file_count=0

# Loop through all files in the specified folder
for file in "$folder_path"/rv32ui-p-*; do
    # Extract the filename without the path
    filename=$(basename -- "$file")
    
    # Check if the filename does not end with .dump
    if [[ $filename != *.dump ]]; then
        # If the filename does not end with .dump, process the file
        echo -e "\nRun $filename in rrv-iss."
        if [[ $filename == rv32ui-p-fence_i ]]; then
            continue
        fi

        file_count=$((file_count + 1))

        # ./target/debug/rrv-iss -l debug -f "$file" -i tmp.instr
        output=$(./target/debug/rrv-iss -f "$file")
        exit_code=$(echo "$output" | grep "Target application exit code:" | awk -F': ' '{print $2}')

        echo "$output"
        if [ "$exit_code" -ne 0 ]; then
            echo -e "\nTarget application exit code is not 0, it is $exit_code"
            echo "Total number of processed files: $file_count"
            echo -e "Test FAILED"
            echo -e "========= Running $0 Done =========\n"
            exit 1
        fi
    fi
done

echo -e "\nTotal number of processed files: $file_count"
echo -e "Test PASSED"
echo -e "========= Running  $0 Done =========\n"

2. 探索架构测试流程

RISC-V Architecture Test：riscv-arch-test 中包含了 framework，老的 framework 是基于 Makefile 的，RISCOF 是 riscv-arch-test 使用的新 framework
RISCOF 的工作流串起了riscv-arch-test 的各个部分。RISCOF 有两种工作流：
- 面向架构测试 case 使用者的工作流
  - 两个输入：yaml 格式的 RISCV CONFIG 文件；User DUT plugin
  - riscv-config 把用户提供的 yaml 文件转换为 standardized/normalized YAML spec
  - Test Selector 根据 std yaml 和 Test DataBase yaml 生成 Test-list yaml file
  - 根据 std yaml spec 和 test-list 在 reference model 和 DUT model 逐个运行 test case
  - reference model 和 DUT model 运行 test case时，会在内存中生成签名
    - produces a signature in the memory region of the test which captures the essence that particular test
  - RISCOF generates an HTML report which provides details of the implementation and tests that were passed/failed by the implementation.
- 面向架构测试 case 开发者的工作流
  - 手写或者使用 RISCV-CTG 生成 test cases，test cases 需要遵守 Test Format Spec
  - 以 coverage mode 运行 RISCOF 来分析 test cases 的 coverage 和 quality
    - coverage mode 只会在 reference model 上运行 test cases
    - 把 execution trace 和 CGF-File (Cover Group Format File) 作为 RISCV-ISAC 的输入, RISCV-ISAC 输出 coverage report。
  - 开发好的 test cases 可以 pull request 到 riscv-arch-test/riscv-test-suite

2.1 准备 RISCOF 的运行依赖

RISCOF 中 import 了 riscv-isac, riscv-isac 被移入 riscv-arch-test/riscv-isac
通过 riscv-arch-test 可以找到除了 riscv-test 之外的所有测试相关仓库
riscv-ctg 被移入 riscv-arch-test/riscv-ctg
这里主要参考 riscv-arch-test::README.md::Getting Started 和 RISCOF::Quickstart Doc

安装 RISCOF 相关软件

python venv
install RISCOF

~/python-venv$ sudo apt install python3-venv
~/python-venv$ python3 -m venv python3.6.0

zsw@zsw:~/xtools/source_tools/tests_for_riscv/riscof$ source ~/python-venv/python3.6.0/bin/activate

(python3.6.0) zsw@zsw:~/xtools/source_tools/tests_for_riscv/riscof$ git checkout -b local_dev d38859f
(python3.6.0) zsw@zsw:~/xtools/source_tools/tests_for_riscv/riscof$ pip3 install --editable .
(python3.6.0) zsw@zsw:~/xtools/source_tools/tests_for_riscv/riscof$ riscof --version
RISC-V Architectural Test Framework., version 1.25.3

~~install riscv-ctg: (python3.6.0) :~/xtools/source_tools/tests_for_riscv/riscv-arch-test/riscv-ctg$ pip install --editable .~~
install riscv-isac: (python3.6.0) :~/xtools/source_tools/tests_for_riscv/riscv-arch-test/riscv-isac$ pip install --editable .
install SAIL (SAIL C-emulator)

$ sudo apt-get install build-essential libgmp-dev z3 pkg-config zlib1g-dev
# $ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)"
$ wget https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh
$ chmod +x install.sh
$ sudo ./install.sh

$ opam init -y --disable-sandboxing
$ eval $(opam env --switch=default)
# $ opam switch create ocaml-base-compiler.4.06.1
$ opam switch create ocaml-base-compiler
$ eval $(opam env --switch=ocaml-base-compiler)
$ opam update && opam upgrade
$ opam install sail -y
$ eval $(opam config env)

$ git clone https://github.com/riscv/sail-riscv.git
$ cd sail-riscv
$ make && ARCH=RV32 make

$ ln -s `realpath c_emulator/riscv_sim_RV32` ~/xtools/riscv/bin/riscv_sim_RV32
$ ln -s `realpath c_emulator/riscv_sim_RV64` ~/xtools/riscv/bin/riscv_sim_RV64

2.2 使用 spike 和 sail 来验证 riscof 的工作流程

2.2.1 创建 Env Files 和 Architectural Tests

inputs to the RISCOF framework

A RISCV-CONFIG based YAML specification of the ISA choices made by the user.Details on writing the specific YAML spec can be found here : Spec Documentation
A user DUT plugin, Building your Model Plugin

步骤记录

浏览器中打开 https://github.com/riscv/riscv-arch-test 会跳转到 https://github.com/riscv-non-isa/riscv-arch-test , 需要修改 RISCOF 源码，重新 install RISCOF

(python3.6.0) zsw@zsw:~/xtools/source_tools/tests_for_riscv/riscof$ git diff --staged
diff --git a/riscof/constants.py b/riscof/constants.py
index 3cff010..c0e9e6e 100644
--- a/riscof/constants.py
+++ b/riscof/constants.py
@@ -13,8 +13,8 @@ suite = os.path.join(root,"suite/")
 #        if os.path.isfile(full_path):
 #            cgf_file.append(full_path)

-https_url = 'https://github.com/riscv/riscv-arch-test.git'
-ssh_url = 'git@github.com:riscv/riscv-arch-test.git'
+https_url = 'https://github.com/riscv-non-isa/riscv-arch-test.git'
+ssh_url = 'git@github.com:riscv-non-isa/riscv-arch-test.git'

 framework_db = os.path.join(root, "framework/database.yaml")
 cwd = os.getcwd()
 
(python3.6.0) zsw@zsw:~/xtools/source_tools/tests_for_riscv/riscof$ pip install --editable .

Create Neccesary Env Files
Clone ~~or update~~ the Architectural Tests

zsw@zsw:~/workspace/ws_develop/riscof_work_space/example$ source ~/python-env/python3.6.0/bin/activate

# Create Neccesary Env Files
(python3.6.0) zsw@zsw:~/workspace/ws_develop/riscof_work_space/example$ riscof setup --dutname=spike

# Clone the Architectural Tests
zsw@zsw:~/xtools/source_tools/tests_for_riscv$ git clone https://github.com/riscv-non-isa/riscv-arch-test.git

# or update the Architectural Tests
# (python3.6.0) zsw@zsw:~/workspace/ws_develop/riscof_work_space/example$ riscof --verbose info arch-test --dir ~/xtools/source_tools/tests_for_riscv --update

(python3.6.0) zsw@zsw:~/workspace/ws_develop/riscof_work_space/example$ riscof setup --dutname=spike
    INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
    INFO | using riscv_isac version : 0.18.0
    INFO | using riscv_config version : 3.18.3
    INFO | Setting up sample plugin requirements [Old files will be overwritten]
    INFO | Creating sample Plugin directory for [DUT]: spike
    INFO | creating /home/zsw/workspace/ws_develop/riscof_work_space/example/spike
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/model/model_platform.yaml -> /home/zsw/workspace/ws_develop/riscof_work_space/example/spike
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/model/riscof_model.py -> /home/zsw/workspace/ws_develop/riscof_work_space/example/spike
    INFO | creating /home/zsw/workspace/ws_develop/riscof_work_space/example/spike/env
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/model/env/model_test.h -> /home/zsw/workspace/ws_develop/riscof_work_space/example/spike/env
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/model/env/link.ld -> /home/zsw/workspace/ws_develop/riscof_work_space/example/spike/env
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/model/model_isa.yaml -> /home/zsw/workspace/ws_develop/riscof_work_space/example/spike
    INFO | Creating sample Plugin directory for [REF]: sail_cSim
    INFO | creating /home/zsw/workspace/ws_develop/riscof_work_space/example/sail_cSim
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/sail_cSim/__init__.py -> /home/zsw/workspace/ws_develop/riscof_work_space/example/sail_cSim
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/sail_cSim/riscof_sail_cSim.py -> /home/zsw/workspace/ws_develop/riscof_work_space/example/sail_cSim
    INFO | creating /home/zsw/workspace/ws_develop/riscof_work_space/example/sail_cSim/env
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/sail_cSim/env/model_test.h -> /home/zsw/workspace/ws_develop/riscof_work_space/example/sail_cSim/env
    INFO | copying /home/zsw/xtools/source_tools/tests_for_riscv/riscof/riscof/Templates/setup/sail_cSim/env/link.ld -> /home/zsw/workspace/ws_develop/riscof_work_space/example/sail_cSim/env
    INFO | Creating Sample Config File
    INFO | **NOTE**: Please update the paths of the reference and plugins in the config.ini file

修改生成文件中的交叉编译工具名称,改动的文件见下面的 git diff 输出

diff --git a/sail_cSim/riscof_sail_cSim.py b/sail_cSim/riscof_sail_cSim.py
index 3ee8659..c253dd6 100644
--- a/sail_cSim/riscof_sail_cSim.py
+++ b/sail_cSim/riscof_sail_cSim.py
@@ -41,8 +41,8 @@ class sail_cSim(pluginTemplate):
     def initialise(self, suite, work_dir, archtest_env):
         self.suite = suite
         self.work_dir = work_dir
-        self.objdump_cmd = 'riscv{1}-unknown-elf-objdump -D {0} > {2};'
-        self.compile_cmd = 'riscv{1}-unknown-elf-gcc -march={0} \
+        self.objdump_cmd = 'riscv64-unknown-elf-objdump -D {0} > {2};'
+        self.compile_cmd = 'riscv64-unknown-elf-gcc -march={0} \
          -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles\
          -T '+self.pluginpath+'/env/link.ld\
          -I '+self.pluginpath+'/env/\
@@ -63,11 +63,11 @@ class sail_cSim(pluginTemplate):
             self.isa += 'f'
         if "D" in ispec["ISA"]:
             self.isa += 'd'
-        objdump = "riscv{0}-unknown-elf-objdump".format(self.xlen)
+        objdump = "riscv64-unknown-elf-objdump".format(self.xlen)
         if shutil.which(objdump) is None:
             logger.error(objdump+": executable not found. Please check environment setup.")
             raise SystemExit(1)
-        compiler = "riscv{0}-unknown-elf-gcc".format(self.xlen)
+        compiler = "riscv64-unknown-elf-gcc".format(self.xlen)
         if shutil.which(compiler) is None:
             logger.error(compiler+": executable not found. Please check environment setup.")
             raise SystemExit(1)
diff --git a/spike/riscof_spike.py b/spike/riscof_spike.py
index 1e5ed76..8c791eb 100644
--- a/spike/riscof_spike.py
+++ b/spike/riscof_spike.py
@@ -72,7 +72,7 @@ class spike(pluginTemplate):
        # Note the march is not hardwired here, because it will change for each
        # test. Similarly the output elf name and compile macros will be assigned later in the
        # runTests function
-       self.compile_cmd = 'riscv{1}-unknown-elf-gcc -march={0} \
+       self.compile_cmd = 'riscv64-unknown-elf-gcc -march={0} \
          -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles -g\
          -T '+self.pluginpath+'/env/link.ld\
          -I '+self.pluginpath+'/env/\

2.2.2 Running RISCOF

validateyaml, testlist, run

riscof 是比较 signatures 文件,不是 trace 文件

# CWD --- (python3.6.0) zsw@zsw-xa:~/workspace/ws_develop/riscof_work_space/example

# The RISCOF run is divided into three steps as shown in the overview Figure. 
# The first step is to check if the input yaml files are configured correctly. 
# This step internally calls the riscv-config on both the isa and platform yaml files indicated in the config.ini file.
$ riscof validateyaml --config=config.ini

$ export RISCV_ARCH_TEST_ROOT=/home/zsw/xtools/source_tools/tests_for_riscv/riscv-arch-test

# The next step is to generate the list of tests that need to be run on the models.
$ riscof testlist --config=config.ini --suite=${RISCV_ARCH_TEST_ROOT}/riscv-test-suite/ --env=${RISCV_ARCH_TEST_ROOT}/riscv-test-suite/env

# The last step is to run the tests on the each of the models and compare the signature values to guarantee correctness.
$ riscof run --config=config.ini --suite=${RISCV_ARCH_TEST_ROOT}/riscv-test-suite/ --env=${RISCV_ARCH_TEST_ROOT}/riscv-test-suite/env

9. 运行 RISC-V 官方的测试（上）

9. 运行 RISC-V 官方的测试（上）

0. 概述

0.1 RISC-V 官方的测试分两个层级

0.2 相关缩写

1. 单元测试

2. 探索架构测试流程

2.1 准备 RISCOF 的运行依赖

安装 RISCOF 相关软件

2.2 使用 spike 和 sail 来验证 riscof 的工作流程

2.2.1 创建 Env Files 和 Architectural Tests

inputs to the RISCOF framework

步骤记录

2.2.2 Running RISCOF

validateyaml, testlist, run

推荐阅读更多精彩内容