QP/C  7.4.0-rc.2
Real-Time Embedded Framework
No Matches
Software Tracing

Time ManagementNon-Preemptive Kernel

Concepts & Definitions

In any real-life project, getting the code written, compiled, and successfully linked is only the first step. The system still needs to be tested, validated, and tuned for best performance and resource consumption. A single-step debugger is frequently not helpful because it stops the system and exactly hinders seeing live interactions within the application. Clogging up high-performance code with printf() statements is usually too intrusive and simply unworkable in most embedded systems. So the questions are: How can you monitor the behavior of a running real-time system without degrading the system itself? How can you discover and document elusive, intermittent bugs that are caused by subtle interactions among concurrent components? How do you design and execute repeatable unit and integration tests of your system? How do you ensure that a system runs reliably for long periods of time and gets top processor performance? Techniques based on Software Tracing can answer many of these questions.

What is Software Tracing?

Software Tracing is a method for obtaining diagnostic information in a live environment without the need to stop or even significantly slow down the application to get the system feedback. Software Tracing always involves some form of a target system instrumentation to log the relevant discrete occurrences inside the target for subsequent retrieval from the target to the host computer and analysis. An example of Software Tracing is sprinkling the code with printf() statements, which are a crude tracing instrumentation in this case. Of course, a professional Software Tracing system can be far less intrusive and more powerful than the primitive printf().

In the context of safety certification, the Software Tracing system described in this part of the Software Requirement Specification plays a critical role. It provides the backbone of testing, verification, and validation strategies for both QP Framework and QP Applications.

The Figure SRS-QS-SET below shows a typical setup for Software Tracing, which always consists of two components:

  1. Target-resident component for generating and sending the trace data. This component might also receive commands from the host to execute on the target; and
  2. Host-resident component to receive, parse, validate, visualize, and analyze the data.

Figure SRS-QS-SET: Typical setup for Software Tracing.
The discrete occurrences logged by a Software Tracing system are sometimes referred to as "events". However, in the context of an event-driven framework like QP, these occurrences will be called trace records, to avoid confusing them with the application-level events.

Software Tracing & Active Objects

Software Tracing is particularly effective and powerful in combination with the event-driven Active Object model of computation. Due to the inversion of control, a running application built of Active Objects is a highly structured affair where all important system interactions funnel through the underlying event-driven framework. This arrangement offers a unique opportunity for applying Software Tracing in a framework like QP.

Figure SRS-QS-FUN: Software tracing in QP Framework

Tracing instrumentation inside just the QP Framework provides an unprecedented wealth of information about the running system, far more detailed and comprehensive than any traditional RTOS can provide (because RTOS is not based on control inversion.) The Software Trace data from the framework alone can produce:

[1] detailed state machine activity in all Active Objects and other state machines in the system.

[2] detailed information about event posting/publishing, queuing, recycling. This also includes complete, time-stamped sequence diagrams.

[3] detailed information about real-time kernel activity, like: kernel objects, context switches, scheduler, etc.

[4] custom application-specific trace records

This ability can form the foundation of the whole testing strategy for QP Application. In addition, individual Active Objects are natural entities for unit testing, which you can perform simply by injecting events into the Active Objects and collecting the generated execution trace. Software Tracing at the framework level makes all this comprehensive information available to the application developers, even with no instrumentation added to the application-level code.

QP/Spy Software Tracing System

QP/Spy is a Software Tracing and testing system specifically designed for and embedded in the QP Framework. As shown in Figure SRS-QS-STR, QP/Spy consists of the following components:

  1. Target-resident component called QS. This component is part of QP Framework and is the main subject of this Software Requirements Specification. The QS target-resident component consists of the QS-TX RAM buffer, the QS Filters, as well as the instrumentation embedded in the QP Framework. Additionally, the QS target-resident component contains the receive-channel (QS-RX) with its own RAM buffer, which can receive data from the QSPY host component.
  2. Host-resident component called QSPY. This component is not part of QP Framework and is described in the QTools collection .

Figure SRS-QS-STR: Structure of the QP/Spy Software Tracing system.
The QS tracing instrumentation is inactive by default. It becomes active only when explicitly enabled (by defining a special macro). This means that the QS instrumentation can be safely left in the code for future development, testing, and maintenance.

Data Protocol

The QS target-resident component inserts trace records into the QS-TX RAM buffer using a binary data protocol. The QP/Spy protocol must be lightweight, but must support clearly delimited frames, as well as provisions to check data continuity and integrity of the frames. These features are necessary to allow flexible removal of data from the RAM buffer in any chunks typically not aligned with the frame boundaries. Finally, the data transmitted from the target with the QS data protocol must also allow the host to instantaneously re-synchronize after any buffering or transmission error to minimize loss of useful data.

Run-time Filtering

To minimize the intrusiveness of tracing, the QS target-resident component must perform efficient, selective logging of trace records using as little processing and memory resources of the target as possible. Selective logging means that the tracing system provides user-definable, fine-granularity filters so that the QS target-resident component only collects trace records of interest, and you can filter out as many or as few instrumented trace records as you need.

Predefined Trace Records

QP Framework contains the tracing instrumentation for pre-defined trace records, such as state machine activity (dispatching events, entering/exiting a state, executing transitions, etc.), Active Object activity (allocating events, posting/publishing events, time events, etc.), and more. These QS records have predefined (hard-coded) structure both in the QS target-resident component and in the QSPY host-based application.

Application-Specific Trace Records

In addition to the predefined QS records, QP Application can add its own, flexible, application-specific trace records, whose structure is not known in advance to the QSPY host-resident component. Application-specific trace records carry the format information in them.

QS Dictionaries

Every Software Tracing system, just like every single-step debugger, needs the symbolic information, such as the names of various objects, names of functions, and symbolic names of event signals. This is because by the time source code is compiled and loaded into the target, the symbolic information is stripped. Therefore, the symbolic information must be somehow provided to the QSPY host-resident component, so that it can associate the symbolic names with binary addresses and other binary information received from the target and then display the symbolic names in the human-readable trace. Various Software Tracing systems approach this problem differently.

The QS target-resident component provides special dictionary trace records designed expressly for providing the symbolic information about the target code in the trace itself. These "dictionary records" provide mapping between the unique object or function addresses in the target memory and the corresponding symbolic names from the source code. The dictionary trace records are typically generated during the system initialization and this is the only time they are sent to the QSPY host component. Generating the "dictionaries" is the responsibility of the QP Application.

QS-RX Receive Channel

The QS target-resident component can also implement a receive-channel (QS-RX), which allows receiving, parsing and executing commands from the QSPY host application. Such a QS-RX channel can be the backbone for interacting with the target system and implementing such features as unit testing and monitoring of the target system.


Finally, the QS target-resident tracing component must allow consolidating data from all parts of the system, including concurrently executing Active Objects, "naked" threads (if used), and interrupts. This means that the QS API must be reentrant (i.e., both thread-safe and interrupt-safe).



QP Framework shall support Software Tracing.
Support for Software Tracing means that QP Framework shall implement the QS target-resident component. This also means that QP Framework shall provide the Software Tracing API for initializing the QS target-resident component, setting up the filters, encoding QS trace records, accessing the QS trace buffers, etc.


QS target-resident component shall be inactive by default and activated only when explicitly enabled.
The QS tracing API shall be inactive by default, meaning it should not produce any executable code. The QS instrumentation shall become activate only when explicitly enabled. This requirement does NOT mean that the QS tracing instrumentation gets removed or changed in the QP Framework or QP Application source code. Instead, this requirement means that the (inactive) QS instrumentation can be safely left in the source code (both QP Framework and QP Application) to help in future development, testing, and maintenance.
Use Case
The QS API can be implemented as preprocessor macros (in C or C++), which are defined to nothing by default resulting in no code generation by the compiler. Only when a special macro is defined (e.g., Q_SPY), the QS API can be defined such that it actually generates code. That way also avoids contaminating the source code with conditional compilation for every QS API, which could run the risk of inadvertently leaving some QS APIs active (should the developer forget to surround the QS API with conditional compilation).


QS target-resident component shall use the binary data protocol.
The QS target-resident component shall produce tracing data into the RAM buffer encoded by means of a binary protocol. The main feature required from the applied protocol is maintaining clearly delimited frames, each containing one trace record. The protocol "frames" shall contain the following elements:
  • sequence-number to enable checking data continuity (range 0..255, "wrapping around")
  • record-id to identify the type of the trace record (range 0..127)
  • optional time-stamp data element (configurable length integer of 1, 2, or 4 bytes)
  • optional data-payload (length 0..n bytes)
  • checksum (range 0..255) to enable checking integrity of the frame
Use Case
This requirement can be met, for example, by a High Level Data Link Control (HDLC) protocol, which is characterized by establishing a very easily identifiable frames in the serial data stream. Any receiver of such a protocol can instantaneously synchronize to the frame boundary by simply finding the Flag byte (typically 0x7E).


QS target-resident component shall allow flexible buffering schemes and decoupling trace generation from transmission to the host.
The QS data protocol is the main enabler for a flexible buffering policy. Specifically, the protocol inserts only complete trace records as clearly delimited "frames" into the RAM buffer, which has two important consequences:
  1. For each trace record the QS transmission protocol maintains both the continuity and the integrity checks (see SRS-QP-QS_10), which means that any data corruption caused by overrunning the old data with the new data can be always reliably detected on the host side. Therefore, the new trace data can be simply inserted into the trace RAM buffer, regardless if it perhaps overwrites the old data that hasn't been removed and sent out yet. The detection of any data corruption can be thus removed from the target system and deferred to the QSPY host component.
  2. The insertion of delimited "frames" in the trace buffers enables decoupling data insertion into the trace buffers from data removal out of the trace buffers. QP Application can remove the trace data in arbitrary chunks, without any consideration for frame boundaries. QP Application can employ any physical data link available in the target for transferring the trace data the host.


QS target-resident component shall support Global-Filter.
The QS Global-Filter is based on the record-id associated with each QS trace record (see SRS-QP-QS_10). The Global-Filter shall allow QP Application to disable or enabling each individual record-id or an arbitrary subset of record-ids. For example, QP Application might enable or disable only state-machine-entry records, or all state-machine group of records. This filter works globally for all trace records in the entire system.


QS target-resident component shall support Local-Filter.
The QS Local-Filter is based on the individual object-id associated with various objects in the target memory. The object-ids are small integer numbers in the range 0..127. The object-ids in the range 0..64 are reserved for the Active Objects, where object-id corresponds to the unique priority of the Active Object. QP Application can associate other remaining object-ids (65..127) with other objects. Then, QP Application can set up the QS Local-Filter to enable only a specific object-id or any subset of object-ids.
Use Case
The main use case for the Local-Filter is an application where certain objects (e.g., Active Objects) are very "noisy", and would overwhelm the trace. The Local-Filter allows QP Application to silence the "noisy" objects and let the others through. Please note that the Global-Filter cannot achieve such a selection and must be complemented by the Local-Filter.


QS target-resident component shall support predefined trace records.
Predefined trace records are trace records with a fixed format known upfront by both the QS target-resident component and the QSPY host-resident component. Every predefined trace record is uniquely identified by its record-id (see SRS-QP-QS_10), and the record-id range 0..100 is reserved for the predefined records. This one-to-one mapping between record-ids and predefined records allows the QSPY host-resident component to easily recognize and correctly parse all the "pre-defined" records. Most predefined trace records have the timestamp data element (see SRS-QP-QS_10), but some (e.g., dictionary trace records) do not.
Use Case
Predefined trace records are used for tracing instrumentation embedded in the QP Framework, such as state machine activity (dispatching events, entering/exiting a state, executing transitions, etc.), Active Object activity (allocating events, posting/publishing events, time events, etc.), and more.


QS target-resident component shall support application-specific trace records.
Application-specific trace records have flexible format not known in advance to the QSPY host-resident component. Instead, application-specific trace records carry the format information in them (which makes them somewhat less efficient than predefined trace records). The record-ids of application-specific trace records are in the range 101-127, and are used only for the Local-Filter (see SRS-QP-QS_20). However, the record-ids in this case do not determine any specific format of the application-specific record. In other words, many application-specific trace records can have the same record-id. All application-specific trace records have the timestamp data element (see SRS-QP-QS_10).
Use Case
Application-specific trace records are used for tracing instrumentation embedded in the QP Applications. Their flexible format allows the QP Application to add arbitrary information to the software trace.


QS target-resident component shall provide symbolic information in the trace by means of dictionary trace records.
The QS target-resident component provides special predefined dictionary trace records designed expressly for providing the symbolic information about the target code in the trace itself. These dictionary records are similar to the symbolic information embedded in the object files for the traditional single-step debugger. QS supports five types of dictionary trace records:
  1. object dictionary
  2. function dictionary
  3. signal dictionary
  4. enumeration dictionary
  5. user dictionary (for the application-specific trace records)


QS target-resident component shall provide the receive channel to allow interaction between the host and the target.
A QS-RX channel is required for interacting with the target system and implementing such features as testing, validation, verification, and monitoring of the target system. The QS-RX channel provides the following services:
  • Remotely reset the Target
  • Request target information (version, all sizes of objects, build time-stamp)
  • Execute a user-defined command inside the target with arguments supplied from the QSPY host component
  • Inject an arbitrary event to the target (dispatch, post or publish)
  • Changing the Global-Filter" inside the target (see @ref SRS-QP-QS_20) - Changing the Local-Filter" inside the target (see SRS-QP-QS_21)
  • Changing the Current-Object inside the target
  • Peek data inside the target memory
  • Poke data into the target memory
  • Fill a specified target memory area with the supplied bit pattern
  • Execute clock tick inside the target
  • Execute test setup inside the target
  • Execute test teardown inside the Target
  • Store a "Test Probe" inside the target

Time ManagementNon-Preemptive Kernel