In any real-life project, getting the code written, compiled, and successfully linked is only the first step. The system still needs to be tested, validated, and optimized for best performance and resource consumption. A single-step debugger is frequently not helpful because it stops the system and exactly hinders seeing live interactions within the application. Clogging up high-performance code with printf statements is usually too intrusive and simply unworkable in many embedded systems, because printf formatting and output via a serial port happen exactly in the most time-critical paths through the code.

So the questions are: How can you monitor the behavior of a running real-time system without degrading the system itself? How can you discover and document elusive, intermittent bugs that are caused by subtle interactions among concurrent components? How do you design and execute repeatable unit tests and integration tests of your system? How do you ensure that a system runs reliably for long periods of time and achieves optimal performance?

Techniques based on software tracing can answer many of these questions. Software tracing is a method for obtaining diagnostic information in a live environment without the need to stop the application to get system feedback. Software tracing always involves some form of target system instrumentation to log interesting discrete events for subsequent retrieval from the system and analysis.

Remarks: Software tracing is particularly effective and powerful in combination with the event-driven reactive programming model, such as the one implemented in QP™ real-time frameworks. Due to the inversion of control, a real-time framework controls almost all interesting interactions in the system, so an instrumented real-time framework can provide much more comprehensive and detailed information than can any traditional RTOS kernel.

What is it?

QP/Spy™ is a software tracing and testing system specifically designed for embedded systems, such as single chip microcontrollers. The job of QP/Spy is to capture information about an embedded code's execution and send it to the host computer with minimal impact on the real-time performance of the embedded code.

Software tracing data records flowing from Target to the Host

Remarks: QP/Spy can also (optionally) send commands and data to the embedded target (see Bi-Directional QP/Spy), which serves as the basis for Unit Testing and for Visualization and Monitoring subsystems of the QP/Spy system.

How does it work?

In a nutshell, working with QP/Spy™ is similar to peppering the code with printf or similar (like sprintf) statements for logging and debugging, except that QP/Spy™ is much less intrusive, more lightweight, portable and selective than the primitive printf. Additionally, unlike the printf output, the data produced by QP/Spy™ contains the data integrity and continuity checks, so the host computer can tell if it receives corrupt or incomplete data.

The main advantages of the QP/Spy™ tracing system over peppering the code with printf statements are:

When you use printfs, the data formatting and sending occur in the time-critical paths through the embedded code. In contrast, QP/Spy produces raw binary data, so all the time-consuming formatting is removed from the embedded system and is done after the fact in the host computer.
in QP/Spy, data logging and sending to the host are separated so that the embedded system can typically perform the transmission outside of the time-critical path, for example in the idle processing of the embedded CPU.
Data produced by printf gives you no clue if the data gets corrupted or lost in transmission. In contrast, the QP/Spy transmission protocol checks for data integrity and continuity.
The QP/Spy tracing provides flexible filtering mechanisms, which allow you to selectively trace only the aspects of the system that you choose, and to suppress the data that is not interesting at the moment.
The QP/Spy tracing is implemented as macros that are active only in the "Spy" build configuration and are inactive in the Release or Debug build configurations. This means that you can safely leave the instrumentation in the code for future maintenance, development and testing.
The QP/Spy trace data contains precise, high-granularity timestamps, so you can tie the data to the common timeline.
The code size of the target-resident component in QP/Spy is merely a few hundred bytes, which contrasts with several kilobytes of code required by a full-blown printf formatter.

The picture below shows a typical setup for software tracing. The embedded Target system is executing instrumented code, which logs the trace data into a RAM buffer inside the Target. From that buffer the trace data is sent over a data link to a Host computer, which stores, displays, and analyzes the information. This configuration means that software tracing always requires two components: a "Target resident component" for generating and sending the trace data (QS in QP/Spy™), and a "Host resident component" to receive, decompress, visualize, and analyze the data (QSPY in QP/SPy™).

Typical setup for software tracing with QP/Spy™

Note: Software tracing instrumentation logs interesting discrete events that occur in the target system. These discrete events will be called trace records, to avoid confusing them with the application-level events.

A good tracing solution, such as QP/Spy, is minimally intrusive, which means that it can provide visibility into the running code with minimal impact on the target system behavior. Properly implemented and used, it will let you diagnose a live system without interrupting or significantly altering the behavior of the system under investigation.

Of course, it's always possible that the overhead of software tracing, no matter how small, will have some effect on the target system behavior, which is known as the probe effect (a.k.a. the "Heisenberg effect"). To help you determine whether that is occurring, you must be able to configure the instrumentation in and out, both at compile-time as well as at run-time.

To minimize the "probe effect", a good trace system performs efficient, selective logging of trace records using as little processing and memory resources of the target as possible. Selective logging means that the tracing system provides user-definable, fine-granularity filters so that the target-resident component only collects events of interest, and you can filter as many or as few instrumented events as you need. That way you can make the tracing as noninvasive as necessary.

To minimize RAM usage, the target-resident trace component typically uses a circular trace buffer that is continuously updated, and new data overwrites the old when the buffer "wraps around" due to limited size or transmission rate to the host. This reflects the typically applied last-is-best policy in collecting the trace data. In order to focus on certain periods of time, software trace provides configurable software triggers that can start and stop trace collection before the new data overwrites the old data of interest in the circular buffer.

To further maximize the amount of data collected in the trace buffer, the Target-resident component typically applies some form of data compression to squeeze more trace information into the buffer and to minimize the bandwidth required to uplink the data to the Host.

However, perhaps the most important characteristic of a flexible software tracing system is the separation of trace logging (what is being traced) from the data transmission mechanism (how and when exactly the data is sent to the Host). This separation of concerns allows the transmissions to occur in the least time-critical paths of the code, such as the idle loop. Also, clients should be able to employ any data transmission mechanism available on the Target, meaning both the physical transport layer (e.g., serial port, SPI, USB, Ethernet, etc.) as well as implementation strategy (polling, interrupt, DMA, etc.). The tracing facility should tolerate and be able to detect any RAM buffer overruns due to bursts of tracing data production rate or insufficient transmission rate to the host.

Finally, the tracing facility must allow consolidating data from all parts of the system, including concurrently executing threads and interrupts. This means that the instrumentation facilities must be reentrant (i.e., both thread-safe and interrupt-safe). Also, to be able to correlate all this data, most tracing systems provide precise time-stamping of the trace records.

Bi-Directional Connection to the Target

While traditional software tracing systems support only uni-directional output of trace data from the embedded Target to a Host computer, QP/Spy supports bi-directional communication to the target as well. This capability allows users to send commands and data to the Target and form the basis for Unit Testing and Visualization and Monitoring of embedded Targets.

Bi-directional data exchange between the Target and the Host

UDP Socket Extension

The QP/Spy system provides a UDP socket, which is open for communication with various Front-Ends (GUI-based or "headless"). Currently, the UDP connection point is used by the QUTest headless (console-based) front-end and the GUI-based QView front-end.

QP/Spy Session Example

To give you a better idea how QP/Spy works, the listing below shows some example output from a QP/Spy session. The left-hand side shows the raw, binary output generated by the target-resident component (QS). The right-hand side shows the human-readable format generated from the same data by the host-resident component (QSPY). The compression ratio between the binary and textual outputs in this data sample is about 3.7.

Example of the QP/Spy output

The following sections explain the concepts and components of QP/Spy™:

Getting StartedQP/Spy™ Data Protocol