Hello and welcome to the Modern Embedded Systems Programming course. I'm Miro Samek, and in this lesson, I continue exploring the different ways to execute event-driven Active Objects. Today's focus is on extending modern event-driven programming--with Active Objects and hierarchical state machines--to conventional real-time operating systems. Specifically, you'll see how to integrate the QP Active Object framework with the Zephyr RTOS.

In many ways, this lesson ties together the major modern embedded programming themes of this course: event-driven programming, object-oriented programming, and the Active Object model of computation, along with the various execution strategies for event-driven Active Objects covered in lessons 52 through 55.

However, a particularly important reference for today is lesson 34, where the Active Object design pattern was introduced as a collection of best practices for concurrent programming. In that lesson, you also saw how to implement Active Objects using a rudimentary application framework called MicroC/AO, built on the conventional MicroC/OS-II RTOS.

If you haven't watched lesson 34 yet, I recommend doing that first. I won't repeat the fundamentals of what an event-driven framework is, why it necessarily relies on inversion of control, or how it differs from a conventional RTOS.

I will, however, revisit the goals of lesson 34--and even more so, the objectives of this lesson--which are to dispel several deeply rooted misconceptions about event-driven programming, state machines, and Active Objects.

The first misconception is the belief that state machines are only for bare-metal programming, serving as a low-level alternative to a full RTOS. That perception may hold for "input-driven," or "polled," state machines, which were discussed in lesson 37.

But *event-driven* state machines, introduced in lesson 35, not only can work, but actually require a real-time kernel of some sort. This complementary relationship between event-driven programming and an RTOS was already demonstrated in the MicroC/AO event framework used in lessons 34 and 35.

Today's lesson takes that idea much further by explaining that it takes more than RTOS message queues, threads structured as event loops, and switch statements for state machines to apply event-driven programming in practice.

In particular, you'll see how the mature and battle-tested QP real-time event framework integrates with a mainstream, conventional RTOS such as Zephyr.

The second misconception about Active Objects and state machines is that the run-to-completion (RTC) event-processing semantics somehow prevent Active Objects from being truly real-time, because "real" real-time is supposedly possible only with a real RTOS. 

This misunderstanding comes from the notion that an RTC step must necessarily monopolize the CPU for its entire duration, which is the case only for the simplest "superloop"-like schedulers discussed in lessons 53 and 54.

However, Active Objects can also run on top of preemptive kernels, as you saw in lesson 55. In that case, RTC steps of Active Objects at different priority levels *can* preempt one another. This means the hard real-time assumptions behind methods such as Rate-Monotonic Scheduling (RMS) are not violated.

You will see this behavior of Active Objects again today because most conventional RTOS kernels, including Zephyr, are preemptive and compatible with RMS.

The third major misconception concerns the non-blocking requirement for Active Objects, which seems at odds with the way conventional RTOS kernels operate. As explained in lesson 25 on efficient thread blocking, every RTOS thread must block somewhere in its endless loop; otherwise, it consumes all CPU cycles and starves lower-priority threads. 

The key to resolving this apparent contradiction is the event-loop structure of Active Object threads. They do block--but only at the top of the loop when the event queue is empty. The rest of the thread code, the part that executes the Active Objects state machine, must *not* block. You saw this event-loop structure in lesson 34, and you'll see it again today.

And the final confusion I'll try to clarify by the end of this lesson concerns mixing non-blocking event-driven programming with traditional sequential programming based on blocking. This issue becomes critical when an Active Object framework is built on top of a conventional, blocking RTOS.

With this preamble, the plan for today is to first demonstrate how to build and run the Zephyr examples included in the QP/C Active Object framework. Then we'll look at the integration between the QP framework and Zephyr.

Since all code examples for today are already included in the QP/C framework, the download for this lesson 56, which will be posted in the usual companion webpage, will contain QP/C 8.1.2, the latest version at the time of this recording. To avoid bloating the download, the framework has been stripped down, in that parts irrelevant to today's lesson have been deleted.

Now, to quickly explain the code organization for Zephyr inside the qpc folder, first, you see the Zephyr sub-directory. This is the QP/C port to Zephyr. All other QP ports are inside the ports sub-directory, but for the QPC framework to be considered also a Zephyr module, the directory structure must follow the particular Zephyr module requirements. 

We'll take a closer look at the zephyr port directory later because right now, I'd like to show you the examples for Zephyr that are provided, as usual, in the examples directory. Here, in the zephyr sub-directory, you can find 3 examples, the simplest being the blinky example, which blinks an LED on the board. This example is designed generically according to the Zephyr principles and should work unchanged with most boards supported by Zephyr.

Now, usually the examples for this course try to be self-contained, and most of the 3rd-party software is included, typically in the 3rd_party directory. However, the Zephyr project, together with the Zephyr SDK, is humongous, almost 20 Gigabytes, so it is assumed that you've installed all this separately, following the instructions on the Zephyr getting-started web page.

Also, regarding the development host computer, Zephyr, being a Linux Foundation project, is predominantly intended for Linux hosts. And the supplied examples will build on Linux. But the Zephyr getting-started guide also shows how to install Zephyr on Windows, and I will use that installation.

Speaking of the Zephyr development process, you must have noticed that I don't use the usual KEIL uVision development environment, but rather just an editor (VS Code in this case) and a simple terminal. This is because Zephyr tooling is command-line oriented and uses a special tool called 'west', which is a meta-tool on top of the CMake meat-tool on top of make.

In order for all this to work, you need to prepare the terminal by executing special scripts. The commands for Linux and Windows hosts are provided in the blinky example README file, so you just need to copy and paste them into your terminal.

The first script activates the virtual Python environment for west, and the second configures the environment variables for Zephyr.

Once this is done, you change directory to: lesson-56, qpc, examples, zephyr, blinky.

And next, you initiate the Zephyr build by invoking west with the board parameter specified in the -b option. As I mentioned, this example should work with most Zephyr-supported boards. For this lesson, I use the NUCLEO-C031C6.

Zephyr builds take a while, especially the first one, so in the meantime, let's take a look at the blinky source code--by the Zephyr convention located in the src sub-directory.

The main function is identical to all other QP applications. So is the blinky active object and its internal state machine. This would work with any real-time kernel.

The Zephyr-specifics are limited to the Zephyr configuration in prj.conf file and the board-support package (bsp.c). Here, the noteworthy detail is the implementation of the QP time event processing, which is called from the Zephyr timer callback, started to tick every hardware clock tick.

Additionally, you can see that the BSP functions for turning the LED on and off also call the Zephyr print instrumentation, which you will see in a minute.

When the build finally finishes, you can plug in the NUCLEO board and flash the binary. You can try doing this with the west flash command, but this invokes a board-specific flash programming utility, which I haven't configured for Zephyr yet.

But this is a NUCLEO board that enumerates as a USB drive, to which you can copy the binary. You just need to know that the binary is located in build\zephyr\zephyr.bin. Also, on my machine NUCLEO enumerates as drive f:, but of course you need to adjust that to your machine.

After the board is programmed, it should start to blink the LED once per second. You can also open a serial terminal and watch the printouts from the instrumented BSP on and off functions.

The next example for Zephyr is the Dining Philosopher Problem (DPP), which you build identically to Blinky before.

This example demonstrates multiple communicating Active Objects. It consists of 5 active objects emulating the philosophers and 1 active object called Table that manages the forks for hungry philosophers.

As before, the main function and the active object code are generic and independent on any real-time kernel.

You program the board as before, by copying the binary.

This time, the LED should also blink, but less regularly, and you can also watch the printouts from the changing Philosopher status on the serial terminal.

The DPP example demonstrates one more QP feature, which is the software tracing instrumentation. To enable that feature, you must rebuild the DPP example with the option provided in the README file. 

You saw the concept of minimally intrusive software tracing in lesson 46.

When you flash the board now, the ASCII serial terminal shows garbage because the tracing protocol is binary.

To really see the output, you need to launch the special QSPY host utility, which is provided in the QP-bundle. Assuming that you have it installed, you launch QSPY and attach it to the COM port of the NUCLEO board, which you can find from the PC Device Manager.

If QSPY reports errors connecting to the board, check if you have the ASCII serial terminal still running because it uses the same serial port as QSPY.

Now, you can also test the bi-directional connection to the board, for example, by pressing the 'r' key to send the reset command.

The QP/Spy software tracing is implemented in the bsp.c Board Support Package by means of the generic Zephyr serial port driver, and therefore it should work on any board supported by Zephyr.

The last provided example for Zephyr is the real-time testing application used already in lessons 54 and 55.

You build it with the additional option to enable the lowest-priority idle thread, which is added in the QP port to Zephyr, but needs to be enabled.

This application is specific only to the NUCLEO-C031C6 board in that its Board Support Package uses the CMSIS-based register-level access to the hardware, bypassing the Zephyr high-level device tree and generic drivers. This might be frowned upon by the Zephyr purists, but it is much simpler and more efficient, especially when real-time performance is of interest.

You can flash this example to the board as before, but to really see what it does, you need a logic analyzer. This is all explained in the previous lesson 55, so I won't repeat that, but just for context, this application has 4 active objects that post each other events, which leads to interesting preemption scenarios, especially after you press the user button on the NUCLEO board.

The system tick occurs in this example every 200 microseconds, which corresponds to a very high ticking rate of 5kHz, with the 48MHz CPU clock. This high interrupt rate is set up to cause more interesting preemptions. And indeed, you can see many thread preemptions in this logic analyzer trace.

We'll return to this trace again to compare the performance of Zephyr to FreeRTOS and the preemptive QK kernel presented in lesson 55. 

After you've seen the examples, let's talk about the adaptation of the QP framework necessary to run it on top of the Zephyr RTOS. Such adaptation is called a *port*.

Portability of the QP framework is one of its non-functional requirements and is reflected in the layered architecture of the framework. This means that the QP framework defines the interface to the underlying kernel, called the Operating System Abstraction Layer (OSAL).

On one side, the OSAL defines an interface between QP and an abstract kernel. On the other side, the OSAL interfaces with the kernel through its specific API. The big benefit of all this is that once the OSAL layer is ported to the RTOS, it can work without any changes with any CPU or board supported by the RTOS.

The QP OSAL layer is implemented in two files located in the port directory, qpc/zephyr, in this case.

First, the header file qp_port.h provides specific definitions of the kernel-dependent elements and services required by QP.

For example, the QP active object class, QActive, needs an event queue and a thread of execution. The types of these members of the QActive class can't be hard-coded in the framework. Rather, they are OSAL macros, whose definition depends on the specific kernel. For Zephyr, the event queue is defined as struct k_msgq and the thread as struct k_thread. The best way to choose the right representation is to study the application examples accompanying a given kernel. 

Next, an important choice is the critical section for the QP framework to protect its internal integrity against race conditions. Ideally, this should be the same mechanism as used internally by the RTOS to protect its own integrity.

So, when you study the RTOS source code, you find that Zephyr uses the k_spin_lock() and k_spin_unlock() services for this. Aadditionally Zephyr saves the previous status of the lock in a stack variable and then restores the status upon the critical section exit. The QP OSAL critical section abstraction is flexible and can accommodate such behavior.

Please also note that QP critical sections don't nest, so you can use just one spinlock object QF_spinlock for all critical sections in QP.

The next service required by QP is scheduler locking during event publishing to multiple subscribers. Ideally, the mechanism should allow selective scheduler locking only to the priority ceiling of the highest-priority subscriber. However, Zephyr provides only a crude global scheduler lock.

Since the event publishing can also occur in the ISR context, the scheduler locking mechanism must also properly work inside ISRs. In the case of Zephyr, the protection is implemented by explicitly checking the ISR context.

And finally, the QP framework needs deterministic memory management to implement the event pools for mutable events. Some RTOSes provide fixed-size heaps, also known as memory pools, which could be adequate. However, most QP ports, including this port to Zephyr, apply the QP-native QMPool class for event pools. Here you can see the OSAL abstract event-pool operations defined in terms of the QMPool functions. 

The next Operating System Abstraction Layer file, qf_port.c, provides the implementation of the kernel-dependent behavior in the QP framework.

At the top of the file, you see the thread function structured as an event-loop, which I mentioned at the beginning of this lesson, and here you can see a concrete example for Zephyr.

Since this is a critical aspect of executing Active Objects with a conventional RTOS, let me reiterate the main points:

First: *all* Active Object threads run exactly this *same* event-loop function defined inside the framework and even made static, to hide it completely in this module. This is a drastic departure from the traditional way of creating RTOS applications, where each thread runs its own *custom* thread function defined in the application-level code, outside the RTOS.

This means that the control over the thread function resides in the Active Object framework rather than the application. I hope you remember from lessons 33 and 34 that such *inversion of control* is one of the main characteristics of event-driven programming.

However, while all Active Objects run this exact same thread function, each such function operates on a *different* Active Object instance, which is passed as the 'p1' void pointer parameter, and immediately cast to the QActive pointer 'act'.

Now, the body of the event-loop in QP always consists of the following three steps:

Step one: receiving an event from the active object's event queue. In an RTOS port like this one, this call blocks when the event queue is empty, but this is the only blocking call in the whole loop.

Step two: dispatching the event to the Active Object's hierarchical state machine. This is the run-to-completion (RTC) processing and can be complex. However, this code should never internally block or poll for events because it would clog the event-loop and make it unresponsive to events delivered through the event queue. In QP/C, event dispatching is a "virtual" call, emulated in C as explained in lesson 32 about Object-Oriented Programming and polymorphism. In QP/C++, this is obviously a native virtual call.

And step three: the processed event is passed to the QF garbage collector for recycling. This is part of the automatic event management feature provided in QP, which takes advantage of the inherent control inversion. Dynamically allocated, mutable events in QP are reference-counted and *automatically* recycled when no longer referenced.

The reference counting of mutable events is visible in the next function QActive_post_(), which must be customized for Zephyr because it ultimately relies on the Zephyr k_msgq_put() facility.

Before that, however, the event is checked for being mutable, and if so, its reference count is incremented.

You can also see here the software tracing instrumentation, which is active only when software tracing is enabled.

Also, at the end of the function, in the case of posting being unsuccessful, the event is recycled to avoid an event leak.

I'll skip the other QActive operations related to the event queue because they are similar to QActive_post().

However, I will explain starting Active Objects, which involves initializing the event queue, assigning the Active Object priority, and creating the execution thread.

Regarding the priority, the problem is that QP framework uses a direct priority numbering scheme, where the lowest priority is 1, and the highest is QF_MAX_ACTIVE. Zephyr uses the reverse scheme, where priority 0 is the highest-priority and larger numbers correspond to the lower-priority threads.

The solution implemented in this QP port to Zephyr supports two ways of assigning the Active Object priority.

First, the user can specify a separate QP priority and a separate Zephyr priority, as illustrated in the bsp.c file for the real-time application example. Please note that it is the developer's responsibility to specify the priorities consistently; that is, a higher QP priority should correspond to the higher-priority Zephyr thread.

Second, the user can specify only the QP priority and leave the Zephyr priority at zero. In that case, the priority for the Zephyr thread is calculated as the reversal of the QP priority according to the shown formula.


And the last function in this QP port I'd like to explain is QF_run(), which transfers the control to the framework to run the Active Objects.

An interesting aspect here is the optional implementation of the lowest-priority idle thread because Zephyr does not provide any access, such as a callback, to its own idle loop.

When the idle feature is configured, QF_run() lowers the priority of the main thread and enters an endless loop that continuously calls idle processing. Otherwise, QF_run() simply returns. The idle processing is useful for implementing software tracing output or other idle behavior.


Alright, so at this point, I hope you have a general idea of how a conventional, *blocking* RTOS, like Zephyr, can be used to execute event-driven, *non-blocking* state machines encapsulated inside Active Objects.

The key to resolving the apparent contradiction between the blocking and non-blocking paradigms is the event-loop, which blocks only at the top and then executes a non-blocking run-to-completion code segment.

But using a conventional RTOS for running event loops is wasteful. I mean, an RTOS is a complex machinery capable of blocking at any number of points inside a thread function, including blocking inside deeply nested function calls. But this comes at a hefty price of a separate stack for each thread and complex context switching.

So, while a conventional RTOS can definitely do it, it is just not the most efficient tool for the job--like rolling out a cannon to kill a fly.

But, as you saw in the previous lesson 55, there are other types of lightweight kernels, such as the preemptive, non-blocking QK built into the QP framework, which do away with the wasteful event-loop and execute only the run-to-completion code segment as a one-shot task.

Please note that such a task is called only when the kernel knows that the event queue has some events, so the queue get-operation does not block.

The result is the same, preemptive real-time behavior, but at a much lower cost of just a single stack and simpler context switch.

A Kernel of this type is a more suitable tool for the job of executing non-blocking event-driven systems in real-time.

To illustrate the point, let me compare the logic analyzer traces from the real-time example with Zephyr discussed today with the same real-time example with the QK kernel from the last lesson 55.

As you can see, both traces show the same preemptions, but in Zephyr, everything is slower because of its higher overhead.

Also, on the right, you can see a comparison of ROM and RAM footprints of the same real-time application with Zephyr and QK.

Which all leads to the question: why should you use a conventional RTOS for event-driven systems?

Well, you don't need to use it to achieve preemptive multitasking and hard-real time performance. In fact, a conventional Real-Time Operating System kernel is *less* suitable for hard real-time than the lightweight non-blocking kernel, because an RTOS causes more overhead and blocking calls scattered inside the threads are more difficult to analyze.

But there are reasons when you might need a conventional RTOS:

First, you might be using a CPU type that the non-blocking kernel does not yet support, but your RTOS does. Such use is possible for properly layered QP ports, where all CPU dependencies are handled by the RTOS.

Second, your software libraries, such as TCP/IP and USB communication stacks, file systems, and various device drivers, require blocking, for example, inside APIs, such as sockets.

Third, a lot of in-house legacy code relies on blocking, which mandates the use of a conventional RTOS capable of blocking.

Fourth, the development team is familiar with the sequential blocking paradigm and uncomfortable with the event-driven non-blocking paradigm.

So, if you must use a conventional RTOS for any of the reasons, here are a couple of the most important guidelines: 

You should never mix the blocking and non-blocking paradigms within a single thread. In other words, you should never block inside Active Objects' state machines. If you need to block, create a dedicated "naked" RTOS thread for it.

Please remember that you *can* post or publish an event to Active Objects from any code, including from your blocking threads.

To communicate in the opposite direction: from Active Objects to the blocking threads, you can also use the same mailbox mechanism as your QP port (k_msgq in case of the Zephyr port).

However, after you receive a QP event, you must remember to explicitly recycle the event after processing in your blocking thread (by calling the QF_gc() function)


This concludes this lesson about extending the modern event-driven paradigm to conventional RTOS.

As always, the project for this lesson is available for download from the companion webpage to this video course and from the GitHub repository.

If you enjoy this channel, please consider subscribing to help support the ongoing production of new videos. Thank you for watching!