Traceable Documentation with Doxygen & Spexygen
Hello and welcome to the "Modern Embedded Systems Programming" course. I'm Miro Samek, and in this lesson I'll show you how to document your software with Doxygen. However, I'll go several steps beyond just documenting the source code. You'll also see a Doxygen extension called Spexygen for creating traceable formal specifications, such as requirements, architecture, design, and many others, mandated by functional safety standards.

Let me start by giving you an idea of what such traceable documentation looks like because if you think of a career in embedded systems, you will be increasingly required not only to create documentation but to create traceable documentation compliant with international functional safety standards, such as IEC 61508.

Traceability is the cornerstone of any formal documentation system, especially those intended for managing functional safety. It is the explicit representation of the relationships among work artifacts. For example, a requirement can be connected to a code element that fulfills the requirement or a code element can be connected to a unit test that tests that code.

The IEC 61508 functional safety standard addresses the bi-directional nature of traceability:

Backward traceability begins at a specific work artifact and links it to the upstream artifact. For example, a code artifact can be linked with an original requirement or a test artifact with the upstream code artifact. Backward traceability is the most natural and efficient way of specifying hierarchical relationships, such as subclass-superclass in object-oriented programming (OOP). In most documentation systems, including Spexygen, the developer explicitly defines backward traceability links among work artifacts.

Forward traceability begins at the original artifact and links it with all the resulting downstream work items. For example, a requirement can be linked with the source code element that implements that requirement.

In contrast to backward traceability, forward traceability links can and should be generated automatically based on the existing backward traceability links.

Please note that forward traceability is typically recursive, meaning that if an artifact, such as a unit test, traces to a code artifact and that code artifact traces to a requirement, then the test artifact also traces to the requirement artifact.

Spexygen generates recursive forward traceability because it's needed to identify the potential consequences of changing a given artifact, which is called "impact analysis."

In the end, backward traceability and forward traceability result in bi-directional traceability, which is highly desired for functional safety certification.

Of course, commercial Application Lifecycle Management (ALM) tools can automate traceability management, but Spexygen offers similar capabilities as a free and open-source alternative.

Spexygen's most significant advantage is that it seamlessly incorporates source code in the bidirectional traceability management because it is based on Doxygen. In other words, Spexygen is a universal system that combines functional safety specifications with source code documentation.

Now, let me step back and show you how to create traceable documentation with Spexygen.

You need to start with Doxygen.

Type doxygen in the search box and go to the website. Download doxygen for your platform, which is Windows, in my case. The installation is straightforward.

Now, in the downloads for this lesson #51, you will find the examples for doxygen.

Let's open the folder for lesson-51 and look inside example0. This is just some code organized as a header.h file, source.c file, and a couple of tests. At this point, the project knows nothing about doxygen.

But launch the the freshly installed doxywizard application.
For the working directory, select lesson-51/example0.
For project name type: example
For project synopsis type: doxygen example
For project version type: 0.0.0
You can select the project logo
For the directory to scan for source code type dot, meaning the already selected working directory.
But here, you need to check the "Scan recursively" box because the code is in subdirectories inc, src, and test.
For the destination directory, type a dot as well.

Next, the desired extraction mode: All entries and include cross-reference
You can select "Optimize for C" because this is C code.

Next, you can leave the preselected HTML and LateX output formats.

And you don't need to change anything in the Diagrams tab.

Finally, you can run doxygen, after which you can click on Show HTML output. This opens your browser.

The main page doesn't show much, but you can view your files, such as the header.h and source.c.
Doxygen has generated some documentation boxes for functions and data structs, but they only show the signatures.
However, when you go to the source code for this file and hover your mouse over various code elements, you can see the balloon help for them. This is useful and might be a faster way to explore and learn a new codebase, although modern IDEs also have this capability.

Now, let's look at the project directory because Doxygen has produced some output there. You can see the new HTML directory, which contains the static website you just browsed, but you also see the new LaTeX directory because you allowed Doxygen to produce LateX output.

This directory does not contain any PDF yet, but it has a make.bat file, which you can run. Now, this batch file requires a LaTeX processor, and I've installed MikTex. But eventually, the make batch produces PDF output called refman.pdf, which looks like this.

Before moving on, save the doxywizard project, which creates the file Doxyfile in the project directory.

Now, let's move on to example1, which already has its Doxyfile, so let's open this one in doxywizard.

Note that the version number for example1 is 0.0.1.

Now, let's compare the whole example0 directory with example1 to see the differences.

The header.h file contains the same code, but now it has the special doxygen comments.

These doxygen comments are just comments, so the compiler ignores them. Actually, so does doxygen, unless the comment starts with a special character.

To be processed by doxygen, a multi-line comment must start with an exclamation point or a star.
A single-line, double-slash comment must start with, again, an exclamation point or a slash.

Most often, the doxygen comment must precede the documented element, but doxygen also supports comments after the element when they additionally start with the less-than character.

Inside the doxygen comments, you can see some special doxygen commands that start with an ampersand or a backslash. To use doxygen effectively, you need to learn about the available commands, but here you see some of the frequently used ones. The @brief command provides a brief description of the element, @par command starts a paragraph with a title, and @param command documents a function parameter.

Compared to example0, example1 adds a new file main.dox, which consists of one big doxygen comment. This file provides the main page with the description of the project. Here, you can see that doxygen allows you to create textual documentation of any kind, not just code comments.

Again, this textual documentation uses some doxygen commands, such as mainpage, section, verbaitm, or refernce. You can also see an example of using an image in the documentation.

Finally, let's look at the difference between Doxyfiles. The version number in example1 has changed to 0.0.1. The project logo has been changed to the relative path instead of the absolute path previously, the image path has been added as .., meaning relative directory one level up, and the HTML option GENERATE_TREEVIEW has been changed to YES.

You can access and change all these options from doxywizard, but now you should use the "Expert" tab.

So, finally, you can run doxygen and view the generated HTML output.

This time, you can see the main page generated from your main.dox file and the tree view along the left edge.

The generated links from the ref commands allow quick navigation, for example, to the header.h file, which now contains all the brief descriptions, detailed descriptions, and parameter descriptions.

The last example, number 2, shows how you can add a formal specification, such as a software requirements specification SRS. The goal is to represent the requirements so that doxygen can display, reference, and search the individual requirement work items.

There are a couple of ways to achieve this, but the simplest way, applied later in Spexygen, is to represent work items as doxygen subsections. You'll see in a minute how this looks and works.

However, once the work items are specified, you can reference them from other work items or code items. This is the explicitly provided backward traceability.

One more thing I wanted to point out with example2 is that the SRS can be incorporated as a subpage of the main page.

When it comes to generating the HTML and LateX outputs, you could do this from doxywizard. But you can also do this directly from a command prompt, where you change to the project directory and type doxygen. The Doxygen directory should be added to your path during installation. When you launch it, Doxygen will look for the Doxyfile and process it if found.

The generated HTML output is located in the html directory, where you can just double-click on the index.html file to open it in your browser.

So, here is the example2 in HTML. The main page now lists Software Requirements Specification, so let's click it and take a look.
The requirement artifacts are clearly visible in the SRS document and the tree view. You can also quickly navigate to the individual reuirement items from any view.

When you copy a requirement name and paste it into the search box, doxygen finds it.

So, these are all the properties you wanted, but they are enabled by the names assigned to the requirements. In the industry, such names are known as Unique Identifiers or UIDs.

Many naming conventions exist to ensure that the UIDs are indeed unique, but specifically for doxygen, the UIDs should follow the naming restrictions for identifiers in programming languages such as C or Python. You should use only underscores and avoid dashes or dots to separate the various fields in the UIDs.

Alright, so example2 shows how far you can push it in pure doxygen. You can write specifications, such as SRS, and you can create your own traceable work artifacts alongside already traceable code artifacts.

But there is undoubtedly a lot of room for improvement. First, the specifications can be better structured and better formatted.

But most importantly, the documentation contains only the explicitly created backward traceability links. Of course, you could manually add and maintain forward traceability links as well. However, this would quickly go out of sync with the backward traceability because it would violate the DRY principle (Don't Repeat Yourself).

As mentioned before, forward traceability can and should be generated automatically. Additionally, in any traceability view, the brief descriptions of work artifacts next to the cryptic UIDs would be very useful, so they should also be generated automatically. Which leads us to the Spexygen system that provides precisely the features just described.

First, you need to get Spexygen and put it on your computer. The easiest way is to search for "spexygen" and click on the GitHub link.

In the GitHub repo, you can just click on the code drop-down and download ZIP.

You can unzip spexygen into the lesson-51 directory.

Spexygen comes with an example that is very close to the examples you just saw for pure doxygen. So, again, let's examine the differences from the last example2 to see what changes were introduced in the example for Spexygen.

Let's start with the differences in the SRS (Software Requirements Specification).
On the Spexygen side, you can see that each work item specification starts with the custom Spexygen command @uid, which takes two parameters: the Unique Identifier and the brief description. The @uid command ends with the @enduid custom Spexygen command. Between those two, you can have various line items, such as Description, defined with the @uid_litem custom Spexygen command.

The backward traceability of the work item is specified by the @uid_bw_trace custom Spexygen command. This command takes a parameter "brief," which indicates that you want Spexygen to generate a brief description of the following traced UIDs. The traced UIDs are specified with the @tr custom Spexygen command.

Finally, the custom Spexygen command @uid_fw_trace indicates that you wish Spexygen to generate the forward trace in this precise place of the work item specification. In other words, the @uid_fw_trace command is a placeholder for the generated forward traceability section.

All these commands are documented in the Spexyen manual generated with Spexygen. You can access it from the Spexygen GitHub repo.

Besides being used by Spexygen to correctly parse the specification, the custom commands provide a formal structure for the work items. Of course, you can easily impose your own company's standards on top of those commands. For example, you can have templates of UID line items that must be present for requirements, design, functional safety docs, etc.

Spexygen also provides a separate set of custom commands for specifying code items. These are similar to commands for work items but are used inside the doxygen code comments, where doxygen already applies a specific formatting.

You can find examples of the custom code commands in the header.h file. The crucial aspect here is to properly define the UID of the code item, which Spexygen will later use for traceability. The code UID you provide must be recognized by doxygen.

Once you prepare your documentation by applying the custom Spexygen commands, the next step is to generate the traceability information. The Spexygen system provides a Python script, spexygen.py, that automates this step.
Before you can run this script, however, you need to tell Spexygen which files you'd like to generate and where to generate them. This is done in the configuration file spex.json located in your project directory.

In this json file, you specify the files you wish to trace, which means collecting the traceability information. You can also specify the output directory for the generated documentation, the name of the include file for Doxygen, and the set of files to generate.

At this point, you're ready to run the spexygen.py script and see what it generates.

At first, you'll run the script manually. For this copy the current project directory and open a command prompt. Change to the project directory and type "python", relative path to the Spexygen directory, and "spexygen.py."

In the console output, you can see that Spexygen reported that the requested forward traces for two UIDs were missing, which probably means that you still need to provide some tests.

But more interestingly, the spexygen.py script has generated a new folder spex in your project directory. This folder contains the code you told it to generate.

To see what Spexygen has done, say, to the SRS document, let's diff the generated srs.dox against the original.

As you can see, spexygen generated the forward traceability sections and also added the brief descriptions to the backward traceability sections.

The generated traces include both work items, such as requirements, and code items, such as Foo::x or Foo_ctor(). The forward traceability sections are also recursive, whereas the levels of dependency are indicated by increasing the indentation of the trace commands. You'll see in a minute how this looks in the doxygen HTML output.

Speaking of which, the final step is to run Doxygen. But before that, let's examine the Doxyfile because now it needs to contain some Spexygen stuff.
First, the Doxyfile includes the Spexyfile from the Spexygen installation directory stored in the SPEXYGEN environment variable. This Spexyfile defines all the custom Spexygen commands you've seen before.

Second, the Doxyfile input comes from two sources: files that were not processed by Spexygen, such as main.dox, and files that were generated by Spexgyen, conveniently listed in the generated Spexyinc file in the spex directory.

Alright, so now let's run Doxygen manually, remembering to define the SPEXYGEN environment variable.

As usual, Doxygen has generated the HTML output in the html folder inside your project directory.

Please note the more attractive and modern styling of the doxygen output. This styling is coming from Spexygen, which, in turn, customized it from the doxygen-awesome project.

As you can see, the work items, such as requirements, are consistently formatted with clearly delimited line items and now contain all the backward and forward traceability links.

You can easily navigate to any work item or code item.
You can also search for them in the doxygen search box.

You've arrived at this final output by running spexygen and doxygen manually, but of course, these two steps can be merged and automated. In fact, the Spexugen example comes with a make.bat batch file, which runs the whole process in one step.

By default, this batch file produces HTML, but it can also generate PDF when invoked with the -PDF argument.

This conculdes this quick introduction to Doxygen and Spexygen. Together, these two tools provide a powerful, automated documentation system that is a free alternative to costly commercial solutions.

The toy example presented here doesn't really do justice to Spexygen's power. However, you can view real-life documentation created with Spexygen by visiting state-machine.com/qpc.

Happy documenting, and thank you for watching!