The core idea of DiffTest is: for two implementations based on the same specification, given the same defined input, their behaviors should be consistent. Back to the processor design, for the two implementations according to the riscv manual, given the same correct program, their state changes (registers, memory) should be consistent, two implementations:
One of them is our CPU;
Alternatively, choose a simulator as a reference implementation. After each party executes an instruction, they check the status of their respective registers and memory. If they find that the status is inconsistent, they immediately report an error and stop the execution of the client program (equivalent to making an assert judgment on each instruction).
Instruction level = each instruction executed is verified
Can convert any program into an instruction-level test and assert the status
Support for programs that never end, such as OS
No need to know the outcome of the procedure in advance
Because we are comparing the behavior of instruction execution, not the semantics of the program
The biggest use of DiffTest is:When a decoding or execution bug occurs after running hundreds or thousands of instructions, how can we quickly find the first erroneous instruction? This is a hellishly difficult problem.
In this article, we choose the following as an example:
QEMU as a reference object (REF)
NEMU as a Test Object (DUT)
Of course, if the processor hardware needs to be tested, the processor can also be used as the DUT.
2 DiffTest supported by NEMU
make menuconfig
1
Go to Testing and Debugging -> Enable differential testing -> Reference design, as follows:
When NEMU is used as DUT, 5 types of simulators can be selected as reference simulators REF.
QEMU, dynamic library mode, the code is located in NEMU/tools/qemu-dl-diff/
QEMU, Socket mode, the code is located in NEMU/tools/qemu-socket-diff/
KVM, the code is located in NEMU/tools/kvm-diff/
NEMU
SPIKE
The author has not conducted any specific research on the latter three types, so this article will not introduce them.
3 QEMU as REF (dynamic library method)
You can use NEMU/tools/qemu-dl-diff/, compiled into the dynamic library riscv64-qemu-so. The relationship between NEMU, dynamic library, and QEMU:
Dynamic library: riscv64-qemu-so, exported the difftest_xx series function interfaces.
QEMU: In the qemu-system-riscv64 executable file, there are cpu_xx, gdb_xx and qemu_xx series function interfaces.
NEMU: riscv64-nemu-interpreter program, calls the dynamic library, and the dynamic library calls QEMU.
specific process:
The NEMU executable program riscv64-nemu-interpreter loads riscv64-qemu-so by calling the dlopen function, and parses out the difftest_xx series function symbols for subsequent calls.
Then, NEMU calls the difftest_init function for initialization.
In the difftest_init function, the dlopen function will be called again to load the qemu-system-riscv64 executable file and parse the cpu_xx, gdb_xx and qemu_xx series function symbols. These function symbols are actually the underlying implementation codes of the difftest_xx series functions.
From the qemu-system-riscv64 file, the parsed functions include the main function, which is then called by the difftest_init function to start the qemu program.
Then, if everything goes well, you can call the qemu function to compare the instruction results.
4 QEMU as REF (Socket mode)
You can use NEMU/tools/qemu-socket-diff/, compiled into the dynamic library riscv64-qemu-so. The relationship between NEMU, dynamic library, and QEMU:
Dynamic library: riscv64-qemu-so, exported the difftest_xx series function interfaces.
QEMU: In the qemu-system-riscv64 executable file, there are cpu_xx and gdb_xx series function interfaces.
NEMU: riscv64-nemu-interpreter program, calls the dynamic library, the dynamic library sends the command data packet through the socket, QEMU receives the data packet and parses it, and decides which function to call based on the specific command.
specific process:
The NEMU executable program riscv64-nemu-interpreter loads riscv64-qemu-so by calling the dlopen function, and parses out the difftest_xx series function symbols for subsequent calls.
Then, NEMU calls the difftest_init function for initialization.
In the difftest_init function, a child process is forked () and QEMU is started by calling the execlp function; the parent process continues to execute, connects to QEMU through the socket, and calls the init_isa function to initialize.
Then, if everything goes well, the command results can be compared through the socket.
