How a small company makes a VMEbus Multiprocessor Real-Time System IN TIME and WITHIN BUDGET: a case study.

Real-Time User Support International (RTUSI) is a company specialized in the development of parallel bus based (VMEbus, PCI, etc..) Multiprocessor Real-Time Systems. Created in 1990, it took some years and a hard learning process to apply PARETO's principle and focus on the important factors to assure success when producing new Systems IN TIME and WITHIN BUDGET. Among these factors we find: a rapid sophisticated personnel selection process, communication with the customer in the context of a methodological approach, prototyping and simulation to gain customer confidence, the use of an adaptive low cost CASE tool in all cycles of the lifecycle and defensive programming techniques including embedded debugging and testing techniques.

THE CASE STUDY

General Description

COLUMBUS is the spacecraft project involving a lot of participating nations. The space platform consists of approximately 120 modules to be launched by the USA and Russia starting end of 1997. All these modules pass extensive tests before use. RTUSI participates to the development of VMEbus based test and simulation equipment. The subsystem described here is an ETHERNET to Mil1553 Bus Data Interface Front-end Assembly (DIFA) (Figure 1). It creates the bridge between a group of workstations and servers where the tests and configurations are prepared, and the spacecraft modules that are interconnected with the Mil1553 bus. The equipment is software reconfigurable depending on the number of modules to be tested, the behavior of each tested module, the number of Mil1553 busses present, etc... When the complete spacecraft is present, the DIFA is used as monitoring/test equipment. When some modules are absent, then the DIFA will simulate these.
The actual delivered subsystem is approximately a four man-years project. It is executed in one calendar year.

Figure 1 System Overview

Equipment and software involved

It is not our intention here to discuss the advantages and drawbacks of the hardware and software choices. We only describe here the actual system used to be able to understand the complexity (or simplicity) of the project.
Figure 2 shows the system architecture based on a 68040 board from Motorola and a number of dual processor (29000 and DSP) boards from SBS. The SBS boards are expensive (+/- 35.000 DM) because they have to fulfill simultaneously the MilBus test functions, BC (Bus Controller) and RT (Remote Terminal) Simulation functions.
The Motorola board runs the VRTX Real-Time Operating System from Microtec Research. The SBS board has proprietary software on board included in the board supply.
All the software is written in C using Microtec Compilers. The device-drivers for the SBS boards for VRTX were not available and are adapted from a library delivered by SBS. In a first instance the TCP/IP package delivered with VRTX is used inclusive NFS. However due to extremely poor performance (80% of all the 68040 processor power was used for the protocol) we decided to use another TCP/IP protocol stack delivered from Pacific Softworks together with the NFS layer from Microtec.
This all shows that RTUSI deals with a very non homogeneous hardware and software environment delivered from different manufacturers. This involves a great deal of responsibility going to the RTUSI team. They should be capable of dealing with problems going from the detailed hardware machine level (interrupt handling) via network protocol stacks towards complex functional programming. Therefore great care is taken in the personnel selection process.

Figure 2 DIFA architecture

A PERSONNEL SELECTION PROCESS

Do we need people with experience?

A first reaction when you have to execute such a project is to think you need people with an extensive experience in Real-Time System analysis and design. However, TODAY you need the experience to work in a methodological framework, this is not the experience most of the older Real-Time engineers have. They may be catalogued as genius electronic hobbyists. Today you need more and less than that.
Ten years ago most projects had almost no time and budget constraints. This is not true anymore. Therefore it is necessary to carefully go through analysis, design, implementation, testing and documenting. This creates the need for trained (and convinced) people for these new approaches.

Hire in only young engineers.

We therefore decided to use young engineers only, having at least an university degree (4 or 5 years) in computer science. The selection process takes into account the extensive communication and abstraction skills needed by these people in this kind of job.

Selection procedure

The selection process we elaborated during the last 3 years consists of different phases we want to go through in a one month time span involving as less personnel effort from our side. (We are a small company). Figure 3 explains this procedure.
The problem with the procedure is that finally only 3 % of the initial candidates for a specific selection procedure become RTUSI employee. We prefer not to accept a project rather than to use people not corresponding to our framework.
The selection process consists of technical and psychological test. The psychological test consist in a personality test only. We are using MMPI2 (ref. 1) for having a complete personal profile. We are most interested in the stress and energetic level, this together with the ability to work as well individually as in a team.

Figure 3 Personnel selection procedure.

The technical tests have the intention to clarify the broad spectrum of knowledge of the candidate in order to observe how complementary or supplementary he or she is compared to the actual team. As we have to cover a lot of skills in a Real-Time environment, to be complementary is essential.
All personnel is participating in a way or another in the selection process. During the interview, almost all personnel is present. This has two purposes. Making more solid the existing team and having no problems introducing a new member in the team.
The whole procedure is organized in a way to distribute the selection process overhead over all company employees.

COMMUNICATE WITH THE CUSTOMER USING A METHODOLOGICAL FRAMEWORK

Framework and nothing else.

All work you do for ESA imposes special standard procedures. The COLUMBUS Software Development Standard STD 1213 800 is imposed. However, this standard is very rigorous for ADA based development used for the Control supervision and Management level. For the "control level" the designer is free to use what he wants. NO software development technique is imposed. We had only to stick to the classical framework of audits and reviews, including producing the right documents in time.

Figure 4 Industrial Control Hierarchy

For the subsystem of our concern, the C language is used together with the VRTX Real-Time Operating System as stated earlier.

Structured analysis and design.

We did not fall in the trap NOT to use any technique. Being free we decided to adopt a structured analysis and design methodology for the purpose of creating a good communication link between the customer and ourselves. We considered that the analysis phase is the most important one. In this phase the "user needs" are converted to requirements using data flow diagrams and state charts. We then forecast the needed work and budget for the design, implementation, testing and documentation part of the project based on this analysis phase.

Project in 2 phases.

We do the first phase together with the customer paying on a day base. The second phase is executed for a fixed price. This is well accepted by the customer because we take part in the risk for the project and he may budget correctly. We observe that by working that way, the first (analysis phase) takes between 10 and 20% of the project (in this case 130 days) The second phase took 650 working days. System architect is used during both phases.

PROTOTYPING & SIMULATION TO GAIN CUSTOMER CONFIDENCE

Progress control is essential.

The biggest problem to produce a new system IN TIME and WITHIN BUDGET is to have progress control and to be as less as possible influenced by the progress of other teams from other companies working on related equipment (other subsystems). Therefore we use a technique of prototyping. The prototype here is the system working with partial functionality. During the analysis phase, the parts making a testable partial functioning system are discovered.
This partial system is delivered and tested as a prototype. It helps refining the requirements and permits for clear milestones with deliverables so that partial payments are no problem. This is essential for a small company.

Simulation software

Simulator software is defined for 2 reasons: give to other parties a simulation of our equipment and simulate missing equipment from other parties so we may test our subsystem. By building these testable prototypes and simulators, all parties involved may easily measure project progress and are themselves independent of the progress of the others. As both simulators may work with each other (simulating together the complete system) it is guaranteed that the integration will be easy. In the mean time the interface between the 2 systems is clearly defined (in an Interface Control Document or ICD).

AN ADAPTIVE CASE TOOL USED IN ALL CYCLES OF THE LIFECYCLE

The Real-Time system life cycle

CASE tools are necessary to formalize the different steps in the methodology. As no techniques were imposed, we decided for the easy approach of structured analysis and design, knowing that this approach is easily understood by the older Real-Time engineers (our customer) who were not used to any technique on analysis level.

Figure 5. System level analysis and design.

The Ward and Mellor (Ref. 2) technique with the Hatley Pirbhai extensions (Ref. 3) is used for (sub)system analysis including state charts. The use of mini-specifications and diagrams are most important to fulfill all the requirements and specially to find the missing ones. The methodological approach is inspired from Shumate & Keller (Ref. 4).

Figure 6 Subsystem analysis and design

"Misuse" of System Architect

One of the less trivial things in Real-Time design is the architectural software design were you decide the multitasking structure. Here we decided to "misuse" the available data flow diagramming technique from Gane and Sarson for multitasking design. Structured design was done using the standard structured charts and flow charts. System Architect seemed to be very effective and flexible in the analysis phase of the project and in the system and software architectural design.

Figure 7 Gane & Sarson Data Flow Diagrams misused for multitasking modelling

DEFENSIVE PROGRAMMING INCLUDING EMBEDDED DEBUGGING TECHNIQUES

How to make robust software?

Real-Time applications need quality robust software. The silly approach is to first make the software work and then make it robust. We prefer the initial longer approach of making it robust, debuggable and easy readable from the very beginning. Error-handling, tracing, statistics of resource usage, logging of time aspects, etc. All these aspects are included in the software at the very beginning even if the customer does not want it. It needs courage to start the project the hard way but the reward is tremendously high. We reduce the debugging and integration phase to a minimum. All found bugs during this phase are easily traceable and repairable. In the end our system works so well that it is used as debugging aid for the surrounding systems.

R & D in debugging and testing.

As we understand the importance of this issue, RTUSI is carrying out a research project sponsored by the Brussels Capital Regional Government on Testing and Debugging Complex Hybrid Real-Time Systems. This projects aims at defining new tools and principles to help the engineer more in what we call the integration phase. This phase is taking up to 80% of the project design phase due to the fact more and more people go for the buy instead of make solutions. Figure 6 shows the benefits of not making the RTOS yourself.

Figure 8 Comparison using an RTOS or not

But more and more people use more than just a commercial RTOS. You may get of the shelf TCP/IP protocol stacks and Graphical User Interfaces. As a consequence, the integration, debugging and testing phase takes even more of the design phase.

Figure 9 System development today and tomorrow.

This debug and test tools should be well integrated with the CASE tools used, but this is another story.

CONCLUSIONS

A fool with a tool is even a bigger fool! You should not introduce CASE tools as the ultimate salvation of how to avoid a project disaster. Project disaster is avoidable by using personnel with communication and abstraction skills. The customer should be continuously involved in the project. Good analysis, prototyping and simulation shows to be the key elements.
Finally, you cannot make a quality product out of spaghetti software. The software, debugging and testing should be designed and implemented with quality and robustness from the very beginning of the project.

Dr. Ir. Martin Timmerman is graduated in Tele-communications Engineering at the Royal Military Academy (RMA) Brussels and is Doctor in Applied Science at the Gent State University (1982). He became the director of the System Development Centre (SDC) at RMA, which he created in 1983, and converted himself to a Computer Science Engineer. Actually he is giving general courses on Computer Platforms and more specific courses on System Development Methodologies at RMA. Outside RMA, Martin is known for his audits, reviews and seminars for his two companies Real-Time Consult and Real-Time User's Support International. RTUSI provides hardware and software support services and is involved in project engineering for Real-Time Systems.

REFERENCES

1. R.L. GREENE, MMPI2/MMPI An Interpretive Manual, 1991 Texas Tech University, by Allyn and Bacon, A Division of Simon & Schuster, Inc. Massachusetts 02194, ISBN 0-205-12525-5
2. P.T. WARD & S. J. MELLOR Structured Development for Real-Time Systems, 1985 Volumes 1-3, New York: Yourdon Press, ISBN 0-13-854787-4, 0-13-854795-5, 0-13-854803-X
3. D.J. HATLEY & I.A. PIRBHAI Strategies for Real-Time System Specification. 1987 Dorset House Publishing New York. ISBN 0-932633-11-0
4. K. SHUMATE & M. KELLER Software Specification and design - A disciplined Approach for Real-Time Systems - 1992 by John Wiley & Sons Inc. ISBN 0-471-53296-7

Go to RTUSI
Go back to index Real-Time Magazine 96Q1

Interesting links on Alpha Space Station:
http://station.nasa.gov/core.html
http://www.estec.esa.nl/spaceflight/index.html