In order to run tests in the QUTest test system, you need the following three components:
This section describes how to install and use the "qutest.py script runner" component.
The qutest.py
script runner can be used standalone, without installation in your Python system (see Examples below).
qutest.py
script runner is included in the QTools™ collection. Also, the QTools™ collection for Windows already includes Python (3.x), so you don't need to install anything extra.You can install the latest qutest.py
script runner with the standard pip
package manager.
qutest
package is no longer available in the PyPi package index due to the overcomplicated "two-factor authentication".Instead, you can direct pip
to install directly from the qutest directory (e.g., /qp/qtools/qutest):
pip install /qp/qtools/qutest/qutest.tar.gz
Alternatively, you can direct pip
to install from Quantum Leaps GitHub:
pip install https://github.com/QuantumLeaps/qtools/releases/latest/download/qutest.tar.gz
If you are using qutest.py
script runner as a standalone Python script, you invoke it as follows:
python3 <path-to-qutest-script>/qutest.py <command-line-options>
Alternatively, if you've installed qutest.py
with pip
, you invoke it as follows:
qutest <command-line-options>
The Python test scripts are executed by the qutest.py
script runner (located in qtools/qutest/
folder), with the following usage:
qutest.py
script runner command-line options have been expanded and changed at version 7.2.0. Unfortunately, it was not possible to preserve the backwards compatibility with the earlier versions.usage: python qutest.py [-h] [-v] [-e [EXE]] [-q [QSPY]] [-l [LOG]] [-o [OPT]] [scripts ...] QUTest test script runner positional arguments: scripts List (comma-separated) of test scripts to run options: -h, --help show this help message and exit -v, --version Display QUTest version -e [EXE], --exe [EXE] Optional host executable or debug/DEBUG -q [QSPY], --qspy [QSPY] optional qspy host, [:ud_port][:tcp_port] -l [LOG], --log [LOG] Optional log directory (might not exist yet) -o [OPT], --opt [OPT] xciob: x:exit-on-fail,i:inter,c:qspy-clear,o:qspy-save-txt,b:qspy-save-bin More info: https://www.state-machine.com/qtools/qutest.html
where:
[scripts] | optional specification of the Python test scripts to run. If not specified, qutest.py will try to run all *.py files in the current directory as test scripts. |
-e[exe] | optional specification of the host executable to launch for testing embedded code on the host computer. If exe is not specified a remote (embedded) target is assumed. The special value debug or DEBUG means that qutest.py will start in the debug mode, in which it will NOT launch the host executable and it will wait for the Target reset and other responses from the Target. |
-q[host][:ud_port][:tcp_port] | optional host-name/IP-address:udp_port:tcp_port for the machine running the QSpy utility. If not specified, the default is localhost:7701:6601 . |
-l[log] | Optional directory for output of the log file (directory will be created if it does not exist yet). |
-o[opt] | options for the QUTest script runner to perform the following actions: x : exit on first failed test i : interactive mode c : clear the QSPY screen just after attaching o : save the QSPY text output to a file (in the QSPY launch directory) b : save the QSPY binary output to a file (in the QSPY launch directory) |
-q192.168.1.100
, -oci
).
[1] python3 %QTOOLS%\qutest\qutest.py [2] python3 %QTOOLS%\qutest\qutest.py -- test_mpu.py [3] python3 %QTOOLS%\qutest\qutest.py -ebuild/test_dpp.exe [4] python3 %QTOOLS%\qutest\qutest.py -ebuild/test_dpp.exe -q192.168.1.100 -l../log -oco [5] qutest -qlocalhost:7702 -oxc -- test_qk.py,test_mpu.py [6] python3 %QTOOLS%\qutest\qutest.py -eDEBUG -- test_mpu.py
[1]
runs all test scripts (*.py
) in the current directory on a remote target connected to QSPY host utility.
[2]
runs the test script test_mpu.py
in the current directory on a remote target connected to QSPY host utility.
[3]
runs all test scripts (*.py
) in the current directory and uses the host executable build/test_dpp.exe
(test fixture).
[4]
runs all test scripts (*.py
) in the current directory, uses the host executable build/test_dpp.exe
(test fixture), and connects to QSPY running on a machine with IP address 192.168.1.100
. Also produces QUTest log (-l) in the directory ../log
. Also clears the QUTest screen before the run (-oc) and causes QSPY to save the text output to a file (-oo)
[5]
runs "qutest" (installed with pip) to execute the test scripts test_qk.py,test_mpu.py
in the current directory, and connects to QSPY at UDP-host:port localhost:7701
.
[6]
runs "qutest" in the DEBUG mode to execute the test script test_mpu.py
in the current directory.
[1] python3 $(QTOOLS)/qutest/qutest.py [2] python3 $(QTOOLS)/qutest/qutest.py -- test_mpu.py [3] python3 $(QTOOLS)/qutest/qutest.py -ebuild/test_dpp [4] python3 $(QTOOLS)/qutest/qutest.py -ebuild/test_dpp -q192.168.1.100 -l../log -oco [5] qutest -qlocalhost:7702 -oxc -- test_qk.py,test_mpu.py [6] python3 $(QTOOLS)/qutest/qutest.py -eDEBUG -- test_mpu.py
[1]
runs all test scripts (*.py
) in the current directory on a remote target connected to QSPY host utility.
[2]
runs the test script test_mpu.py
in the current directory on a remote target connected to QSPY host utility.
[3]
runs all test scripts (*.py
) in the current directory and uses the host executable build/test_dpp
(test fixture).
[4]
runs all test scripts (*.py
) in the current directory, uses the host executable build/test_dpp
(test fixture), and connects to QSPY running on a machine with IP address 192.168.1.100
. Also produces QUTest log (-l) in the directory ../log
. Also clears the QUTest screen before the run (-oc) and causes QSPY to save the text output to a file (-oo)
[5]
runs "qutest" (installed with pip) to execute the test scripts test_qk.py,test_mpu.py
in the current directory, and connects to QSPY at UDP-host:port localhost:7701
.
[6]
runs "qutest" in the DEBUG mode to execute the test script test_mpu.py
in the current directory.
Similar to the Python interpreter itself, the qutest.py
test runner has two basic modes: script (default) and interactive. In the default script mode, the test runner executes the test scripts as fast as possible and exits. In the interactive mode, the test runner pauses and executes commands interactively from the terminal input. The interactive mode is activated by providing the -oi
command-line option. Interactive mode is a good way to play around and try the system's response to various commands. Also, interactive mode can be used to provide stimuli for debugging the test fixture.
Once activated, the interactive mode works as follows:
In either case, the interactive mode is shown by the presence of the >>>
command prompt. During an interactive session the output produced by the text fixture is displayed after each interactive command entered. The output is not checked against any expectations. In fact, the expect() command itself makes no sense in the interactive mode. Also, all tests that are interactive are considered "SIKPPED" and are not evaluated as "PASS" or "FAIL".
Up-arrow
and Down-arrow
keys on your keyboard to navigate through the list of previous commands. You end the interactive session by providing an "empty" command, that is by pressing the <Enter>
key right at the >>>
command promptHere is an example of an interactive session with test scripts provided. (Note the annotations added after the //
comments).
C:\qp\qpc\examples\qutest\qhsmtst\test>make OPT=i // <--- NOTE OPT=i python3 C:\qp\qtools/qutest/qutest.py -ebuild/test_qhsmtst.exe -q -l -oci -- QUTest unit testing front-end 7.2.1 running on Python 3.10.1 Copyright (c) 2005-2022 Quantum Leaps, www.state-machine.com Attaching to QSpy (localhost:7701) ... OK Run ID : 230111_174610 Target : build/test_qhsmtst.exe ==================================[Group 1]================================== test_qhsm-funct.py [ 1]-------------------------------------------------------------------------- QHsmTst_init [ PASS ( 0.1s) ] [ 2]^------------------------------------------------------------------------- QHsmTst dispatch >>> dispatch('A_SIG') // <--- interactive command 0000000127 BSP_DISPLAY s1-A; 0000000128 BSP_DISPLAY s11-EXIT; 0000000129 BSP_DISPLAY s1-EXIT; 0000000130 BSP_DISPLAY s1-ENTRY; 0000000131 BSP_DISPLAY s1-INIT; 0000000132 BSP_DISPLAY s11-ENTRY; 0000000133 Trg-Done QS_RX_EVENT >>> dispatch('B_SIG') // <--- interactive command 0000000134 BSP_DISPLAY s1-B; 0000000135 BSP_DISPLAY s11-EXIT; 0000000136 BSP_DISPLAY s11-ENTRY; 0000000137 Trg-Done QS_RX_EVENT >>> // <--- end of interactive session interactive test shows as "skipped" ---> [ SKIPPED ] ==================================[Group 2]================================== test_qhsm-struct.py [ 3]-------------------------------------------------------------------------- QHsmTst init [ PASS ( 0.1s) ] [ 4]^------------------------------------------------------------------------- QHsmTst dispatch >>> // <--- end of interactive session [ SKIPPED ] ==================================[ SUMMARY ]================================= Target ID : 230111_170758 (QP-Ver=721) Log file : Groups : 2 Tests : 4 Skipped : 2 [ 2 4 ] Failed : 0 ==============================[ OK ( 20.1s) ]===============================
As required for safety certification, the qutest.py
test runner can generate permanent records of the runs by producing log files. This feature is enabled by the -l<log-dir>
command-line option.
LOG=
symbol while invoking make
), for example: [1] make LOG=. [2] make LOG=../log [3] make LOG=c:/cert-qp/logs
[1]
generates QUTest log file in the current directory (.
)[2]
generates QUTest log file in the ../log
directory (relative to the current directory)
[3]
generates QUTest log file in the absolute directory c:/cert-qp/logs
The following following listing shows the generated log file:
The following screen shot shows the qutest.py screen output in which a log file has been produced (with the -l.
command-line option):
And the following listing shows the generated log file:
Run ID : 221221_161550 Target : build/test_qutest.exe ===================================[group]==================================== test_assert.py This test group contains tests that intenionally FAIL, to exercise failure modes of the QUTest system. [ 1]-------------------------------------------------------------------------- Expected assertion [ PASS ( 0.1s) ] [ 2]-------------------------------------------------------------------------- Unexpected assertion (should FAIL!) @test_assert.py:22 exp: "0000000002 COMMAND CMD_A 0" got: "0000000002 =ASSERT= Mod=test_qutest,Loc=100" ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ![ FAIL ( 0.2s) ] [ 3]-------------------------------------------------------------------------- Simple passing test [ PASS ( 0.1s) ] [ 4]-------------------------------------------------------------------------- Wrong assertion expectation (should FAIL!) @test_assert.py:32 exp: "0000000002 =ASSERT= Mod=test_qutest,Loc=200" got: "0000000002 =ASSERT= Mod=test_qutest,Loc=100" ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ![ FAIL ( 1.1s) ] [ 5]-------------------------------------------------------------------------- Simple passing test [ PASS ( 0.1s) ] =================================[ SUMMARY ]================================== Target ID : 221221_161031 (QP-Ver=720) Log file : ./qutest221221_161550.txt Groups : 1 Tests : 5 Skipped : 0 FAILED : 2 [ 2 4 ] ==============================[ FAIL ( 2.7s) ]===============================
Additionally, it is also possible to generate logs files from the QSPY output. This feature is enabled by the -oo
and -ob
command-line options.
-oo
saves the QSPY text output generated during the qutest.py run. QSPY saves the output in the directory from which it was launched.-ob
saves the QSPY raw binary output received from the test fixture during the qutest.py run. QSPY saves the output in the directory from which it was launched.OPT=
symbol while invoking make
), for example: [1] make OPT=o [2] make OPT=b [3] make OPT=ob
[1]
generates QSPY text output (in the QSPY launch directory)[2]
generates QSPY binary output (in the QSPY launch directory)[3]
generates both QSPY text and binary outputs (in the QSPY launch directory)