RTOS Benchmark Program

As announced in the preceding issues of Real-Time Magazine, we present the RTOS Benchmark Project.

By Martin Timmerman,
Editor-In-Chief of Real-Time Magazine,
and Jean-Christophe Monfret,
Benchmark Program Manager,
Real-Time Consult.

Through our consultancy and seminar activities, we have discovered that most engineers and project managers having to choose a Real-Time Operating System (RTOS) are facing unsolvable problems. They can hardly find a fair view of the RTOS market (except in Real-Time Magazine of course) Each manufacturer declares that its RTOS is the most scaleable, the fastest and the easiest to use. Indeed everybody adapted by now the micro or pico-kernel technology. Who to believe? This is why we have decided to launch a RTOS benchmark program to help these engineers and project managers. We will give them the opportunity to get an independent view of the capabilities of all the RTOS. To solve this vast problem, let us present how we conceive this benchmark.

Why is a RTOS benchmark at all needed?
In the ever growing real-time market it has become clear that the best way to develop a good real-time application is to use adequate and powerful tools. Among these tools, the RTOS plays the most important role in the application. But the way to find THE good RTOS for THE application to develop is a difficult one even if ever greater numbers of providers claim their RTOS can do everything. But what is a good RTOS? It is certainly not only a very fast micro, nano or pico kernel (whatever word the manufacturers are using). Using a RTOS should help developers, not entail their "fighting" to get the best out of it. It is also of the utmost importance that both design and implementation of the application be facilitated by a well-designed environment.
As the choice of a good RTOS can make the difference and could become critical for the developer to meet his deadline to the market, real-time engineers should have the opportunity to get an independent point of view on the RTOS market. This is the solution to help them choose the RTOS that best suits their needs. To do so, solid useful and independent benchmarks of RTOS present on the market are needed.

What kind of benchmark?
Some people think that the figures given by most RTOS providers are already worthy benchmarks. It is indeed useful to have figures (such as execution time of each object, intertasks switching time, interrupt latency, etc.) to evaluate the time needed before a response can be given to an event. However, such figures are not amenable to a benchmark as they are at best given without processor or interrupt load. And even so, a very fast RTOS would be useless if it is a nightmare to work with. So we have decided to benchmark the RTOS in a real development environment as well. Doing so will enable us to evaluate the product in actual conditions of working, that is, evaluate the development tools, the product finish, the OS capabilities, the assistance service and documentation quality. To test the RTOS in a real environment, we have to classify and categorise the different fields involved in real-time applications because their needs are very different. For example, a small control system for a factory has requirements that are very far from those of a complex air-traffic control system. We therefore categorise application types in order to assess which particular RTOS suits each of them best. This classification is based on the PO- SIX 1003.13 standard as well as on the reality of the market.

Intended audience
As said before, the aim of the RTOS benchmark program is to help people in their choice of the RTOS that best suits their needs. Therefore this bench- mark is first aimed at decision-makers. Through it, they will have an independent view of the capabilities, the ease of use and the performance of all RTOS available on the market and will thus be in a position to make a well-advised expert choice. Any real-time engineer should also have a great interest in the benchmark program because they will have the opportunity to see if they are using the right RTOS (in the right way) for the application they have to build. They will have an independent view of the state of the art in the RTOS field so that they can advise their managers in the choice of a new RTOS. The RTOS manufacturer should also be very interested in this benchmark program because it is meaningful to be judged by someone taking a truly independent stance. So the quality of the product may be increased. This benchmark should contribute to enhance the quality of the RTOS by stimulating the competition between RTOS manufacturers. Real-time engineers will thus be in a position to develop bigger and better quality applications within shorter time frames, according to present requirements. Finally, as the quality of the RT application will in- crease, more and more RTOS will be sold following the demand of good real-time applications. Never forget that more than 60% of real-time applications still do not use any RTOS off-the-shelf but their own or no RTOS at all!

What to benchmark, why and how?
RTOS object
We have stressed that this benchmark would not just consist of a set of figures measuring the time of execution of a particular system call. Yet, as a real- time system should react to an event within a predictable delay, such figures ought not to be ignored. Most of the time they are given in the best condition of execution, which is absolutely not interesting if we have to calculate the maximum response time to an event. On the contrary we must calculate or measure it in the worst case. So we benchmark the response time of the RTOS object (system calls like posting a message to a queue) under different conditions of load (processor load, memory and task load, interrupt load, etc.). So doing, we will measure the performance degradation, which is in fact the most important point to analyse when one is interested in real-time performance.

Total Quality of the product
The previous point is often considered as the only important one by RTOS makers (probably because it is rather easy to measure). We consider that other elements are of even more importance. What is actually the point of using the fastest RTOS (intertasks switching time, interrupt latency) if it is a nightmare to configure it to reach these performances or if the user interfaces to the tools are absolutely not friendly? This would simply imply a delayed project and resulting in extra costs at the end of the day. As the total quality of the product appears to be very important we have defined some criteria to test it:

• Installation procedure. This particular point tells you a lot about the product's total quality. Most often the installation procedure is created at the very last moment and we consider that an installation done by the provider as a service is not a good solution. If the product is easy to install, it simply shows that the manufacturer took the time to build an easy installation procedure instead of rushing to market the product.

• User-friendliness of the product. User inter- faces nowadays should be of graphical nature. Designing a good graphical user interface is time consuming. So when you are facing a good one, that is very "user-friendly", which you hardly need documentation to work with, you can be sure that the product is of good shape. Most of the time however in the real-time field, people think that putting a few graphics and windows above a former text-based interface will do the job. In such cases though developers simply do not use them ...

• Documentation. Documentation tells you a hundred time more than anything else to see if a product is well finished, or has been delivered paying attention to details that change the developers' life. With good documentation, no training course is needed and the application developer hardly requires assistance service or only a very small one to help in solving very tricky problems. Different kinds of documentation have to be looked at: commercial, user, technical, etc.

Product usability
This is the crux of the matter. Today developers have to build systems that are more and more complex and difficult to handle. To help in this task, the RTOS should be either poor or rich depending on the target application and enable very flexible powerful solutions. The development environment should al- ways be as rich as possible to enable the programmer to develop, debug and tune its system in an easy way. With the previous remarks in mind we will test each RTOS as follows:

• Development tools (profiler debugger, simulator, emulator, ...). A rich development environment is nowadays the key to building rapidly powerful reliable applications.
• OS scaleability. Nowadays every manufacturer claims that his RTOS is very scaleable and good for any type of RT applications. We will verify this and qualify the scaleability.
• OS capacities (variety of system call, presence of time-out, ...). As we said, the richness of the RTOS is very important to design today's RT applications. We will thus evaluate this richness.
• BSP (Board Support Package) richness (number of supported boards, number of device drivers available, etc.). The number of available BSP is also important, as a company buying a RTOS has generally more than a single project on the agenda. The BSP should therefore be rich enough to enable the company to develop products on several platforms, with several options.

Assistance
Even for the best product an assistance service is always needed, for solving complex problems requires very good knowledge of the product. So we qualify the assistance services by simulating real application development circumstances.

RT mechanisms
To test more accurately the powerfulness of a RTOS, we will implement some of the most used mechanisms in real-time applications. Here again we will classify these mechanisms according to the field in which they are mostly used. We will accordingly implement the following RT mechanism:

• Queue with multiple receivers - multiple senders.
• Control shared variables.
• Monitors.
• Simple device drivers.
• Etc.

RT applications
Finally we will develop some actual applications (or parts thereof) representing typical systems of different fields and types, based on the POSIX 1003.13 standard and on the reality of the market. This is the only way to determine for which kind of application the use of a particular RTOS is to be recommended. It is generally considered that the different industries, in which real-time is involved, are the following ones: Communications, Computers and their peripherals, Consumer products, Defence/aerospace, Manufacturing, Scientific and medical instrumentation, Transportation and automotive. Even if the previous application fields are some- how different, they often share the same kind of applications. To group the applications in different classes we have to choose among several possible criteria:

• the time constraint requirement is a criterion. However, by using this as the only criterion, the applications would be much too different in the same category. For example a data acquisition application can have the same kind of time requirement as an industrial control application, yet these two applications remain very different. The first one will use a network and local disk (with a real-time file system) to store the data while the second one will not need any network or real-time file system but will need very special I/O devices. This criterion is only useful to deter- mine which time requirements a RTOS can handle.

• Another criterion is the POSIX 1003.13 standard profile. However, the Application Environment Profile (AEP) model seems to be too restrictive or unclear and does not represent the reality of the market. For instance a real-time application using the network is classified in the multi profile even if it is a small controller system with only one process (but composed of many threads) and using only special I/O. The multi-purpose class is in fact dedicated to applications running on a multi-purposed OS with real-time capabilities. However most real-time applications, ex- cept embedded ones (classified in the minimal profile), will be classified in the multi profile be- cause of a few details that are neither present nor optional in the other profiles. We thus chose to use a functional and environmental criterion to achieve our classification, based on the POSIX classification. We added or clarified some points to take into account the previous remarks. In fact we split the profiles Control and Dedicated into three sub-classes depending on the use of a network or of a local disk (with or without standard or real-time file system).

To clarify things, here are some examples and their classification. We choose an application con- trolling an assembly line. The control system is connected to some sensors through specials IO.

1. In the case of an isolated embedded system (controlled by a very simple panel), it is naturally classified in the minimal profile.
2. Now the supervisor can control the system via a console connected to the system on a serial line. In the POSIX terminology it is a Control profile system. We call this class Embedded Controller.
3. Imagine now that the control is done on a workstation by a client application connected to the system via a network. In the POSIX terminology, because of the network, it should be a multi profile application. We call this class Connected Controller.
4. To conclude this example, imagine the system has to store locally a great amount of data concerning the assembly line. We then need a local hard disk to do this. These data could be printed or stored on tape or sent to the client application when the operator requires them. This Control class with the use of a local disk that can be organised with a standard or real-time file system is called Data Acquisition Controller.

Table 1. It is our aim to qualify which type of application can be handled by a particular OS depending on performance and functional availability.
Of course the last two classes can be mixed to get the Connected Data Acquisition Controller. The same idea and name is used for the Dedicated pro- file (Embedded Dedicated, Connected Dedicated, Data Acquisition Dedicated .. )Table 1 is describing this classification.

RTOS benchmark program road map and deliveries
We are now working on the detail design of the RTOS benchmark program. The result of the first benchmark will be published in one of the following magazines. Afterwards, we will publish one bench- mark every three months. Real-Time Magazine will give you the opportunity to find each trimester an executive summary of the last benchmark. It will be a synthesis of the complete detailed benchmark giving a survey of the product advantages and drawbacks. By the end of the year we intend to deliver the first RTOS benchmark folder that will contain at least two benchmarks. Then the updates of the folder will be quarterly based. These folders will be provided on a subscription basis. Each trimester the subscribers will receive the new detailed benchmark that has been performed and also some updates of the previous benchmarks in case of new release of RTOS. In this way the subscriber will always be in a position to have an up-to-date view of the RTOS market.

Conclusion
So much for the reason why a RTOS benchmark program is needed and the form this benchmark will take: not only a set of figures but also tests in real working conditions, based on standard (POSIX) and also on the reality of the market.
By now you should hopefully have a better idea of how we intend to benchmark RTOS.