Hello and welcome to the Modern Embedded Systems Programming course. My name is Miro Samek and in this seventh lesson on state machines I would like to show you can automatically generate code from your state machines. Graphical modeling and automatic code generation do not seem to be mainstream yet, but they are very much parts of the *modern* approach and I believe they are the future of embedded software development.

Today, you'll see you how code generation actually works and in the process, I will try to convince you about the benefits of graphical modeling as opposed to writing your code manually. You'll then see what the *real* issues with modeling and code generation are, because chances are that they are quite different from what might think right now. So, let's get to it! 

As usual, let's get started with copying the previous lesson-40 directory and renaming it to lesson-41. 

Get inside the new lesson-41 directory, but instead of opening the project "lesson" in the Micro-Vision IDE right away, let's start today with the QM model, also provided in that directory.

Now, you've seen the QM modeling tool before, but it was used just as a drawing tool for creating state machine diagrams. The way those diagrams were created didn't really matter and any other tool would do, including just pen and paper or a whiteboard.

But today, QM will be an essential tool, because you will finally use its code generation capabilities. Therefore, I first need to show you how you can get QM and install it on your machine.

To download QM, you can visit the companion web-page to this video course at state-machine.com/video-course and look for the project downloads. The software that you need for each lesson is listed in the table.

So, for today's lesson 41 you will need qp-bundle, which you can download for Windows, Linux, and MacOS.

Assuming that you work on Windows, you should download QP-bundle for Windows.

Now, as the name suggest, the QP-bundle contains much more than just QM. For example, you also get the QP/C framework that you also already used for the past several lessons, but separately.

If you're interested exactly what's inside QP-bundle, you could watch the "Getting Started" video.

But for today, let's just run the digitally signed Windows installer.

As far as the installation is concerned, after accepting the license agreement, I recommend to install the QP-bundle in the default location C-backslash-qp. 

Next, you can de-select the individual components of the bundle, but I recommend that you keep all of them. Among others, you're getting here the QP/C and QP/C++ frameworks, the QM modeling tool, and the GNU C-and-C++ compilers for the desktop Windows as well as the cross-compiler for ARM Cortex-M. These command-line compilers are quite useful for testing embedded software both on the desktop and on the embedded targets. You will see how to do this in the future lessons of this course.

I also recommend that you keep the option to add the QP-bundle installation folder to your path.

So, finally, you can run the installation, which takes a minute or two.

Once the installation completes, the dot-QM files are recognized in the Windows Explorer and shown with a red-ball icon. You can open them in QM by double-clicking.

Alternatively, you can simply open the QM modeler via a QM shortcut on your desktop and: either open the model through the file menu or by dragging and dropping the model file on the QM modeler.

Either way, you should now have the model of your TimeBomb from the last lesson open in QM. If you don't see the state machine, you can double-click on the TimeBomb class in the Model Explorer.

Of course, QM offers also the Help menu, where, among others, you can learn how to get started with QM. Here, I'd recommend that you check out the QM Tutorial, which is available as a video, or as a step-by-step guide.

But going back to your TimeBomb model, let's start with correcting the model documentation.

Now, the state diagram is done, but the actions taken in the state machine are only provided as pseudocode. This was good enough to avoid clutter in your diagram, but for code generation, you need to provide the actual actions in C.

For that, alongside QM, you can open the Micro-Vision project, where you have your manually written state machine code. You can then copy the actions in C back into them model.

This is actually my general game plan for today. I will to go backwards: from the final code that you already know, because you wrote it manually. You will then see what you need to do, back in the model, in order to generate essentially the same code, but this time do it automatically.

For every action code, every time you see some pseudocode in the model, you need to copy over the actual, corresponding code in C for that action.

OK, so now the state machine is fully defined, but you still need to make sure that your TimeBomb class has all the attributes that you accessed through the "me" pointer. Specifically, in your actions you used the "te" attribute to be the QP time event of the type QTimeEvt. You also used the blink_ctr attribute of the type uint32_t.

You can also provide operations of your TimeBomb class, for example, the constructor. Please remember that QM will automatically pre-pend the class name to all class operations, because it knows the conventions for object-oriented programming in C, which you've learned back in lessons 29 through 32. Here, you can also specify both the return type as well as any parameters, if needed, again remembering that QM will automatically supply the "me" pointer for class operations. Finally, you can provide the actual code of the operation.

So, now all aspects of your TimeBomb class are fully designed. This is called "logical design", because it specifies all the items, such as classes, and their internal structure, such as attributes, operations, and state machines.

But such logical design says nothing about the code organization. The code structure, that is, the directories and files, are the subject of the so called *physical design*.

The majority of modeling and code generating tools don't really support the "physical design". The generated code goes simply to a predetermined directory and has a rather rigid structure: Typically, tor each class a tool generates a header file named "class-name-dot-H" and an implementation file named "class-name-dot-C".

But QM is a unique modeling tool that fully supports the physical design and allows *you* to organize the generated code whichever way you like.

Fully supporting physical design means that directories and files become explicit, first-class model items, just like classes, attributes, operations and state machines are.

So, for example, in QM you *can* choose to generate the code exactly as you have organized your manually-written code.

Specifically, your current code consists of main-dot-c, bsp-dot-h and bsp-dot-c. Out of these files, only the source file main-dot-c contains the TimeBomb active object class and its state machine.

So, let's generate only the main-dot-c file in the same directory as your TimeBomb-dot-qm model file. This will illustrate a non-standard physical design, as well as flexibility to mix and match generated code with any other code located in the same file, like main-dot-c, and in other files, such as bsp-dot-h and bsp-dot-c.

The physical design starts with a directory item. Your current model already contains a directory, but it was for the original Blinky model template, so let's just delete it and start today from scratch.

To add a directory to your model, right-click on the model item and choose "Add Directory" from the popup menu.

Next in the Property Editor, change the generic directory path to "." (dot), meaning that your code directory will be the same as the model directory. The directory path is always relative to the model file and can include "dot-dot" for going up levels, as well as slashes and directory names for going down levels. But today, you just want the current model directory, so you leave it as "." (dot).

Once you have a directory you can add files into it by right-clicking on the directory and choosing "Add File" popup menu.

Your physical design for today is to generate the file named main-dot-c. As soon as you provide the extension, QM will recognize it as a C file and will apply C syntax coloring to it.

Now, you need to edit the body of the file. For this you open the file by double-clicking on it.

Your main-dot-c file starts obviously empty, but today you want it to look like your manually-created code. So, continuing to work backwards, you first copy the entire content of your existing main-dot-c file and paste all of this into your main-dot-c model item.

Now, if you would leave it at this, QM would literally copy the content of the file item from the model to the specified directory and file on disk.

But the twist here is that you can also instruct QM to generate specific parts of the file.

For example, this part of your manually created code corresponds to the *declaration* of your TimeBomb class.

So, you can replace it with the directive to QM to generate such a declaration for you.

You specify this by typing dollar-declare and dragging the desired model element, such as your TimeBomb class, from the Model Explorer like this. Of course, you could also simply type the name of the model item.

The next big segment of code contains the definitions of the TimeBomb class state machine and other operations. So, you can again replace all of this with the directive to QM to generate the *definition* of the whole TimeBomb class. As you probably expect, you specify this by typing dollar-define and dragging the TimeBomb class item from the Model Explorer, like this.

And this is it. Now you can let QM generate the code by clicking on the toolbar button or pressing the F7 shortcut key.

This causes QM to generate the main-dot-c file on disk, and in the process to verify your model items that you asked QM to generate.

QM reports all this in the Log Console, whereas only the files that actually have changed get overwritten on disk.

This time around the file main-dot-c has changed, which the Micro-Vision IDE notices and asks you to reload the file.

At the top of the generated file, you can see the comment added by QM, which tells you not to edit the file manually, because all your changes will be lost when the file is re-generated. This means that from now on, you should only modify the file in QM and let QM re-generate the file after each change.

But below that you can see the section copied verbatim from your file item in the model.

The dollar-declare directive from the model is expanded into the declaration of your TimeBomb class, essentially identical to how you declared the class before.

Below this, you have the expanded dollar-define directive.

And below that, again a snippet of code copied verbatim from your file item.

Again, the generated code is essentially identical to your hand-written code before, and in particular it follows exactly the same rules for coding states, transitions, entry/exit actions and superstates.

This must be so, because the implementation strategy for state machines is determined by the underlying framework, which is QP/C in this case. In fact, the framework is the whole enabler of code generation, exactly because it provides the well-established rules for codifying various model elements.


The generated code contains also some comments, which are designed to help you navigate between the model and the code, but I'll discuss all this in a minute.

Because right now perhaps the most interesting for you is whether the generated code still compiles and runs...

Well, the code builds error- and warning-free right off the bat. That's great.

So let's load it to your TivaC LaunchPad board and test how it works...

- Reset: Green-LED
- SW1 button: blinking Red-LED
- SW2 button: Blue-LED, meaning "defused" state

- SW2 button again: combination of Blue- and Green-LEDs, meaning back to "wait4button" substate
- Now, SW1 button: solid Blue-LED and blinking Red-LED and finally White-LED, meaning back to the "boom" state

This is exactly the same behavior as at the end of the previous lesson-40, which means that the auto-generated code works exactly as your hand-written code before.


Alright, so this was just a simple example of code generation from a visual model, whereas I glossed over many details that turn out to be important in everyday programming practice.

But before getting into some of these details, I'd like to step back for a minute and address some of the more fundamental questions.

The most important such question is: Why should you even bother with modeling and code generation? I mean, isn't modeling just a costly overhead that distracts from the real work of writing code?

Well, some of the best arguments to such a criticism I found in the article "The Pragmatics of Model-Driven Development" by Bran Selic. Please see the video description for a link to this article.

The first argument for modeling comes from the more mature engineering disciplines, where it is inconceivable to construct a car or a house without first preparing conceptual models and detailed drawings.

This popularity of visual representations goes back to the human psychology. Our brain has at least an order of magnitude more hardware (more neurons) devoted to visual processing than all other senses combined. Consequently, we process visual information at a much higher bandwidth, and retain it better than any other form of information, especially complex textual representation.

As they say: "A picture is worth a thousand words", or perhaps in our field, "a thousand lines of code". But not any picture will do.

In order to take advantage of our visual super-intelligence, we need to choose a visual representation that is intuitive, yet precise enough to convey the information unambiguously. For example, mechanical or architectural drawings apply quite specific projections and cross-sections. Similarly, electrical schematics contain carefully chosen, and standardized symbols.

Unfortunately in software, when visual representations are used at all, more often than not these are informal, nebulous "block diagrams" of some kind. These are often missed opportunities for applying visual intelligence, because of ambiguities and overall confusion. For example, such diagrams often mix notations: Does this arrow represent inheritance? Do these multiplicities indicate aggregation? Do the other arrows represent communication? What do the different shadings of boxes mean? How about the meaning of the different fonts? And so on...

For these reasons, the efforts like the Unified Modeling Language (UML), have standardized the diagrams and their semantics. But even in the UML, not all diagrams are equally expressive. Out of all the UML, state machine diagrams stand out as intuitive, yet precise and truly constructive, meaning really useful for effective code generation.

Regarding the intuitive nature of state diagrams, here is one of the stories described by David Harel - the inventor of statecharts - in his article "Statecharts in the Making: A Personal Account". (Again, a link to this article is provided in the video description). 

Here is what David Harel writes: "I recall an anecdote from late 1983 in which in the midst of one session an Air Force pilot entered the room. I remember him staring at our intricate statechart diagram on the blackboard... and asking, "What's that?" One of the engineers said, "Oh, that's the behavior of the so-and-so part of the system, and, by the way, these rounded rectangles are states, and the arrows are transitions between states." The pilot studied the blackboard for a couple of minutes, then said, "I think you have a mistake down here; this arrow should go over here and not over there." He was right!

So there are certainly some compelling reasons for modeling, not just coding. Basically, you don't want to miss out on the extra visual-brainpower and the chance to engage more stakeholders in the design process, not necessarily just software developers. If you don't take advantage of these opportunities, your competition will.

But going back the subject of today's lesson. Code generation turns out to be critical to the success of modeling. Modeling without code generation has been tried and failed. Every time. Models that don't contribute to the final production code are like cars without wheels: they cannot go too far, become outdated and get abandoned sooner or later. 

Alright, so assuming that you are intrigued about code generation, the next set of concerns expressed by developers first exposed to this method is about the correctness, quality, and efficiency of the generated code. These are, of course, the very same questions that were asked when compilers were first introduced more than 50 years ago. But today nobody questions anymore the efficiency of the machine code generated by the compilers.

Same thing applies to modeling and code generation, because in practice correctness and efficiency are really non-issues. In fact, the reality is quite the opposite. The automatically generated code tends to be the most solid part of the project, while the manually written parts, are typically more problematic, due to human errors.

So, what are the *real* problems of code generation when it is actually applied in the everyday programming practice?

Well, here is a list that I'd like to go over with you and illustrate with the QM tool.

The biggest real problem is what the developers call the need to constantly "fight the tool". This feeling arises when things that used to be simple without the tool, suddenly become complex with the tool.

The QM modeling tool has been very carefully designed to minimize the need to "fight the tool". For example, any code that you enter into the tool is directly in C (or C++ if you chose the QP/C++ framework) and the tool itself doesn't attempt to change any of your code.

Other modeling tools handle things like that differently, and often put various restrictions on what you can or cannot do in your actions. To appreciate the simplicity of QM you'd really need to see some other tools, especially the big, "high ceremony" modeling tools. 

But to give you an idea of how "fighting the tool" might look like, back in lesson-39 of this video course you saw the State-Machine Compiler (SMC). Even though this was not a visual tool, it's job was to generate code from a textual model in this case.

The problem is that SMC tries to be smart about your action code. For example, SMC will not let you use simple assignment

You cannot access attributes of your state machine either. 

All you can really do is to call operations on the context class. So, instead of writing your actions in the simple way, you need to invent workarounds, such as adding a bunch of class operations in this case.

But moving on, the next big issue is that the generated code must be complete, meaning that no changes of any kind should be necessary for the code to cleanly compile and link with the rest of the application.

Generating only code skeletons with the "TODO" comments in places that later need to be coded manually are just not good enough.

To that end, QM generates complete code only. If fact, QM assumes that the generated code won't be modified and to prevent any accidental changes, QM marks the generated files as read-only. What that means is that any changes must be performed in the model, and the code must be re-generated, as opposed to being modified manually. 


The next issue, critically important in practice, is the ease of integration between the generated code and the existing code. 

As you saw in the TimeBomb model, in QM *you* write the body of each file, and then you direct QM to generate specified model elements for you. In this arrangement it is trivial to surround the generated code with any include directives, macro definitions, or anything else you need.

This is actually unique to QM, because in most other tools such simple things require going through dialog boxes or special "deployment models".

Also, the TimeBomb example showed you how to generate code for a class. But QM allows you to generate code of both a finer and a coarser granularity than that. For example, you can generate definitions of individual operations, like, say just the constructor.

Or you can generate the whole package.

As you noticed, I keep creating here new files to show how easy that is and not to disturb the working code.

For state machines, you can generate definitions of individual states, whereas the dollar-define directive generates a recursive definition of a state and all its substates.

While the dollar-define-1 directive generates a non-recursive definition of just a given state. You can then use these directives to split the definition of a larger state machine among multiple files.

The next pragmatic consideration is the traceability from the generated code back to the model.

For example, suppose that your model has an error and the generated code fails to compile.

In that case you want to quickly correct the problem in your model, because you are not supposed to change the generated code.

And here is where "traceability" comes in. QM generates special model-link comments right before each element. This allows you to select a piece of the code, including the comment and copy it to the Clipboard. This works for any code editor.

You can then paste the model-link into QM, which causes QM to highlight the corresponding model element, where you can conveniently correct the problem.

Traceability is also important in the opposite direction: from the model to code.

For instance, suppose that you want to set a breakpoint upon the entry to the "boom" state. Your problem is to quickly and reliably locate the corresponding place in the generated code.

In QM you can do this easily by highlighting the desired state in the model and copying the model-link to the Clipboard.

You can then simply search your code for the model-link comment. Again, any code editor supports simple text search.

After locating the place in the code, you can set your breakpoint and proceed with the standard debugger.


Please note that the bi-directional traceability from the model to code and vice versa is the property of the particular "optimal" state machine implementation that you learned back in lesson-39, and which is used in the QP/C framework.

Such traceability is not a given, and many other code generators don't produce bi-directionally traceable code. For example, some code generators transform hierarchical state machines into classical "flat" state machines for implementation. This, unfortunately, brings back the repetitions of transitions, which hierarchical state machines were exactly designed to remove. In that case the traceability from the model to code is destroyed, because a single transition in the model can map to several transitions in the code.

Yet, the bi-directional traceability is a very desirable property and is actually required by various functional safety standards.


When you start modeling, your models will become a new kind of your source code. That means that you will gradually produce different version of your models, which then you'd need to difference and merge, like any other source files.

For example, let's save the current version of your TimeBomb-dot-qm model file as TimeBomb-zero-dot-qm.

And next, let's add the exit action to the "defused" state to undo the entry action, that is, to turn the Blue-LED off.

Please note that his creates two differences: one is the new exit action code and the other is the graphical rendering of the exit action box, which has been resized.

Now, let's change some views in the model and save it as the new version.

At this point, let's say that you would like to see the differences between your original and the new model.

QM does not support such model differencing directly, but to see the difference between your two models you can use any standard code differencing tool, like WinMerge. This is possible, because QM stores the models in XML. 

So, here you can see your two differences: the first being the exit code and the second being the size of the exit box. The QM model format in XML is specifically designed to be quite readable.

And also here again, in the model file itself, QM generates the model-link comments, which you can copy and paste into QM to visually locate the item that has changed.

The saving of the model-link comments in the XML is controlled by the checkbox in the model's property sheet.

It's important to note that, just like the generated code, the XML is read-only, because you are not supposed to make any manual changes there. All changes to the model should be done through the QM modeling tool.


And finally, your models should scale up to efficiently support really big projects and teams. This is were modeling really shows it's biggest advantages.

Here I would like to mention only two QM features that support large-scale projects.

First, is the unique support for the physical design in QM, which becomes critically important for larger-scale projects. Physical design is vastly underappreciated, and there are very few resources that cover it well. One notable exception is John Lakos'es book "Large Scale C++ Software Design", which I highly recommend and list in the video description.

The other feature of QM for bigger projects is the ability to break up models into "external packages" that are saved into separate files.

These external packages can be then managed separately by sub-teams and can be imported into multiple models.

This concludes this quick introduction to automatic code generation.

The three main points I'd like you to remember are:

 that to succeed, modeling must be accompanied by code generation 
 that besides the big, heavy, and expensive tools of the past, there are now lightweight modeling tools that you no longer need to "fight"
 and that success of modeling and code generation in everyday practice requires many little accommodations that collectively bring up the paradigm shift.



If you like this channel, please give this video a like and subscribe to stay tuned. You can also visit state-machine.com/video-course for the class notes and project file downloads.

Finally, all the projects are also available on GitHub in the Quantum Leaps repository "modern embedded programming course". Tanks for watching!