Building Quality

Intelligent Transportation Systems

Through Systems Engineering

April 2002

Prepared for

Intelligent Transportation Systems
Joint Program Office
US Department of Transportation

By Mitretek Systems, Inc.

Technical Report Documentation Page

1. Report No. FHWA-OP-02-046	2. Government Accession No.			3. Recipient’s Catalog No.
4. Title and Subtitle Building Quality Intelligent Transportation Systems Through Systems Engineering				5. Report Date April 2002
				6.  Performing Organization Code
7. Author(s) Paul J. Gonzalez				8. Performing Organization Report No.
9. Performing Organization Name and Address Mitretek Systems, Inc. 600 Maryland Avenue, SW, Suite 755 Washington, DC  20024				10. Work Unit No. (TRAIS)
				11. Contract or Grant No.   DTFH61-00-C-00001
12. Sponsoring Agency Name and Address Department of Transportation Intelligent Transportation Systems Joint Program Office 400 Seventh Street, SW – Room 3416 Washington, DC  20590				13. Type of Report and Period Covered
				14. Sponsoring Agency Code   HOIT
15. Supplementary Notes William S. Jones – Task Manager
16. Abstract This monograph is intended to introduce the topic of systems engineering to managers and staff working on transportation systems projects, with particular emphasis on Intelligent Transportation Systems (ITS) projects.  Systems engineering is a discipline that has been used for over 50 years and has its roots in the building of large, complex systems for the Department of Defense.  Systems engineering is an approach to building systems that enhances the quality of the end result and the expectation is that its application to transportation systems projects will make those projects more effective in developing and implementing the systems they are intended to build.  Although applying systems engineering techniques on a project doesn’t guarantee success, not following a systems engineering approach is a strong recipe for failure. This monograph presents a common approach to systems engineering, one that is followed in many industries and domains, not just by Defense Department contractors.  This approach emphasizes the combination of technical and management activities that produce a disciplined approach to building systems.  And, although anyone from a technical or scientific discipline can learn to practice good systems engineering techniques, the most effective systems engineers are those who also bring domain knowledge about the system being built.  Since the domain knowledge in transportation systems involves knowledge of transportation systems, it is important to being familiarizing transportation engineers about this discipline. This monograph is intended for use in conjunction with the systems engineering courses being offered by the National Highway Institute.
17. Key Word Systems Engineering, Intelligent Transportation Systems, ITS, Transportation System Projects			18. Distribution Statement No Restrictions This document is available to the public.
19. Security Classif. (of this report)  Unclassified		20. Security Classif. (of this page)  Unclassified			21. No. of Pages  81	22. Price  NA

Form DOT F 1700.7 (8-72)          Reproduction of completed page authorized

 Table of Contents

List of Figures and Tables 

Executive Summary

Chapter 1 - Introduction

Purpose of This Document 
Some Conventions Used in the Document 
Intended Audience
What is Systems Engineering?
Why is Systems Engineering So Difficult to Define?
Why Use Systems Engineering?
The Challenges in System Engineering
Identifying and Evaluating System Alternatives
Managing Uncertainty and Risk in Our Systems 
Designing Quality into Our Systems
Handling Project Management Issues that Arise
Our Approach to Illustrating Systems Engineering Concepts

Chapter 2 - The System Life Cycle and Systems Engineering

Definitions
Where in the Systems Life Cycle Does System Engineering Fit?
Deciding on the Acquisition Method
How Does Systems Engineering Relate to Project Management?

Chapter 3 - Elements of Systems Engineering

System Engineering Standards
Systems Engineering Capability Maturity Model (SE-CMM)
Organizational Process Areas
Project Process Areas
Engineering Process Areas
Minimum Requirements for Systems Engineering

Chapter 4 - Addressing Uncertainty and Risk With Systems Engineering

Types of Uncertainty and Risk Addressed
Decision Making Under Uncertainty
Systems Engineering Tools for Addressing Uncertainty and Risk
Project Scheduling and Tracking Tools
Trade-off Studies
Reviews and Audits
Modeling and Simulation
Prototyping
Benchmarks
Technical Performance Measures

Chapter 5 - Where to Go to Get More Help

Department of Transportation Resources
National Highway Institute
Standards

References

Other

Appendix

Bibliography

List of Tables and Figures

Tables

No.      Name

 1         Key Questions 

 2         AVI Tag Trade-Off Study Matrix 

3         Comparison of Relative Costs by System Life Cycle Stage

4         SE-CMM Process Area Groupings 

5         Alternative Assessment Matrix

6         List of Representative Standards

Figures

No.      Name                                                                           

1         Project Resolutions

2         Assessment of Importance vs. Uncertainty

3         Statement of the Problem 

4         Concept of Operations Outline 

5         Sample Operation Scenario

6         “V” Diagram

7         Sample Work Breakdown Structure

8         Requirements Traceability Matrix

9         Sample Gantt Chart 

10         Activity Diagram 

Executive Summary

Purpose and Intended Audience

Systems engineering is an approach to building systems that enhances the quality of the end result.  It has its roots in the development of large, complex systems for the Department of Defense.  Systems engineering has been around for over 50 years and there are many books and articles that cover it in great detail.  Our goal is to introduce systems engineering to managers and staff working on Intelligent Transportation Systems (ITS) projects, if they aren’t already familiar with its practice.  We believe that they will become more effective in acquiring, developing, and implementing ITS systems by following systems engineering practices.  We prepared this monograph to introduce transportation professionals to established systems engineering practices that have proven successful in other domains.

Our intended audience includes:

• ITS project managers
• ITS project staff
• Contractors and their staff working on ITS projects
• Anyone else interested in systems engineering, particularly on software-intensive systems

Although our primary focus is on ITS project managers, we hope our coverage of this topic encourages others to learn more about this important area.  Applying systems engineering techniques on a project doesn’t guarantee success; not following a systems engineering approach, however, is a strong recipe for failure.

What is Systems Engineering?

If you ask a group of systems engineers to define “systems engineering,” you might get more definitions than there are members of that group.  In fact, when the International Council on Systems Engineering (INCOSE) was formed, the “entrance fee” to the first meeting was a definition of “systems engineering” from anyone who wanted to attend.  No two definitions were exactly alike, but they tended to fall into one of four categories:

• Those who considered it a function solely of “systems engineers,” and consisting only of technical activities
• Those who saw it as a function solely of systems engineers, but consisting of both technical and management activities
• Those who believed that it consisted only of technical activities that anyone from a technical or scientific discipline could do
• Those who saw it as set of technical and management activities that anyone from a technical or scientific discipline could do

Our position on the subject falls into the last category.  As we discuss throughout this paper, systems engineering combines technical activities and management activities to produce a disciplined approach to building systems.

Although anyone from a technical or scientific discipline can learn to practice good systems engineering techniques, to be most effective as a systems engineer, a person must have domain knowledge about the system being built.  Domain knowledge is a fundamental understanding of the technology and functions involved in the system being built.  In an ITS system, for example, domain knowledge includes transportation engineering or transit system management.  Without domain knowledge, a systems engineer is not as effective.  However, if you have to choose between using systems engineers who don’t have domain knowledge and not using any systems engineers, use the systems engineers and complement them with staff who do have domain knowledge.

Why Does Systems Engineering Matter to ITS Projects and Project Managers?

What every ITS project manager wants is a successful system at the end of the project, with “success” measured by how well the system satisfies the requirements of the people who use it.  So a goal-oriented ITS project manager wants to use any tools or techniques that help achieve success.  Systems engineering provides those tools and techniques.

Systems engineering helps accomplish four key activities that impact a project’s success.  These are:

• Identify and evaluate alternatives
• Manage uncertainty and risk in our systems
• Design quality into our systems
• Handle program management issues that arise

Identifying and evaluating alternatives is important as we attempt to determine which alternative system design and implementation offers us the best chance to succeed.  We needed to measure the feasibility of each alternative from three different points of view: technical feasibility, cost feasibility, and schedule feasibility.  Technical feasibility addresses whether we can build, maintain, and operate a system alternative, given the technology and people available to us.  Cost feasibility looks at whether we can build, maintain, and operate a system alternative with the funds available for it.  Schedule feasibility considers whether we can build a system alternative within the time frame allotted for its development.  Usually we have to make trade-offs; one alternative may cost less than another, but we may be able to build a second alternative faster than the first.  We have to decide which offers the better value.  If several alternatives fit within the technical, cost, and schedule parameters that we’ve set, it may come down to which alternative offers the least implementation risk.

Managing risk and uncertainty in our systems development efforts is important because we want to avoid mistakes or potential problems that threaten the success of our work.  Systems engineering focuses on three aspects of risk management: identification, analysis, and mitigation.

Designing quality into our systems is done by addressing those factors that can negatively affect quality.  Paraphrasing the International Organization for Standardization (ISO), we can define quality as “the totality of features of a system that bear on its ability to satisfy stated or implied needs.”  Among the factors that can negatively affect the quality of a system are its complexity, its inflexibility, its lack of standardized components, and its reliability and availability.  A complex system is hard to use and maintain.  While it may be necessary for a system to be complex, our goal should be to keep it as simple as possible.  An inflexible system doesn’t adapt well to change; when its environment changes or when we must add to it or modify it to deal with changes in our needs, it may fail us.  Systems that lack standardized components are difficult to maintain.  When we must replace some part of the system, we may not be able to find a replacement part.  This can increase overall system maintenance costs or cause the system to fail while operating.  We can’t use systems effectively if they are not reliable and available.  An unreliable system breaks down while we’re trying to use it; an unavailable system isn’t there to be used when we need it.

Handling project management issues that arise is easier to do if we have a good project plan to start with.  A good project plan should be complete, comprehensive, and communicated.  We can ensure the completeness and comprehensiveness of the plan by:

• Including all tasks that we must perform
• Accurately estimating the resources required to accomplish each task
• Assigning the appropriate resources to each task
• Defining all dependencies among tasks
• Identifying all products or other criteria whose completion signifies that a task is done
• Determining how to measure progress against plan when managing our project

We must also communicate the plan to everyone that it affects.  We must tell those people assigned to work on project tasks:

• What work they are responsible for
• When they should begin a task
• When they should complete a task and how they will know when a task is done
• Who else will be working on that task
• What tasks are dependent on the one they are working on and what tasks their task depends on
• What non-people resources they will need to do their job

Having done that, we must then track each task, measure its progress, revise the overall plan if needed, and identify and address any obstacles that impede our progress.  These are standard project management activities, but project management is an important element in a good systems engineering program.

Systems Engineering Impacts on System Implementation

Systems engineering is a process, not just a set of tools.  As such, systems engineering activities occur throughout the system development life cycle.  A common set of stages in a system development life cycle and the systems engineering technical activities that accompany them include the following:

• Conception.  The stage in which the need for a system (or major system enhancement) is first identified.  The principal systems engineering activities in this stage revolve around feasibility assessment.  Is the system feasible?  Can a feasible system be built in a reasonable time and at a reasonable cost?  How much time will we need to build this system?  How much should it cost?  These are the types of questions that a systems engineer focuses on during this stage.
  
• Requirements Analysis.  During the requirements analysis stage, the systems engineer focuses on ensuring that the requirements defined for the system state what needs to be done rather than how it should be done.  The systems engineer also works at ensuring that the requirements defined are clear, complete, and correct.  The systems engineer schedules reviews with the stakeholders in the system to ensure that all parties involved in building the system have the same understanding of what each requirement means.  Stakeholders include the contractor selected to build the system.
  
• Design.  During this stage, the systems engineer helps flesh out the details of the system, helping make decisions about the best way to satisfy the system’s requirements.  To help overcome technical uncertainty, the systems engineer may conduct trade-off studies, use modeling and simulation to analyze potential system performance, build prototypes to assess the technical feasibility of a proposed solution, or perform in-depth analyses of different technologies to assess their applicability to the system under development.  The systems engineer also conducts design reviews with project stakeholders, to ensure that the design approach selected is consistent with their needs and expectations.
  
• Implementation.  Systems engineering activities during this stage (sometimes called Development) focus on ensuring that what gets built matches the agreed-upon design.  One specific subset of systems engineering activities during this stage is software engineering, designed to ensure the quality of the software in the system.  These activities include walk-throughs of developed programs (where programmers review the work of another programmer, to determine whether any errors exist).  In addition, software engineers define standards for code development and ensure that these standards are followed.  But the systems engineering activities aren’t solely directed at software.  It is also important to ensure that hardware developed (or modified) during this stage also matches the agreed-upon design.  Hardware inspections are one technique for quality control during this stage.
  
• Testing.  Systems engineers create complete and comprehensive test sets that effectively exercise the system and its components.  They also analyze test results and assess the degree to which the system satisfies its requirements.

Testing is important to software development, because it uncovers errors in the logic of the programs.  Although it is unlikely that all execution paths through a program can be covered, systems engineers focus on identifying test coverage to ensure that mission-critical functions are thoroughly tested.

Hardware components need to be exercised as part of the testing process to ensure that they perform as expected and as the system requires them to.  And you should recognize that hardware testing needs to take place under realistic conditions of use, not just a “laboratory” environment.  Some hardware components, e.g., global positioning system receivers, may work quite well under controlled climatic conditions, yet fail when exposed to realistic weather conditions, such as below-freezing temperatures.

• System Acceptance.  Some of the key systems engineering activities during this stage look at the training of system users and the approach for fielding the system.  New systems, or enhanced versions of existing systems, need to be fielded in a manner that does not disrupt normal business operations.  Systems engineers plan these “rollout” activities to effect a smooth transition to the new system.  This stage also involves the final testing of the system by its users, to ensure that it meets their requirements.
  
• Operation and Maintenance.  After the system is fielded, systems engineers focus on ensuring that the system continues to meet its performance goals.  Should problems arise, they will diagnose the causes and determine how to solve the problem with minimal delay.  In addition to fixing problems that occur, systems engineers also assess potential enhancements and upgrades to system capabilities.  This is particularly critical, considering the normal incremental, evolutionary approach to fielding modern transportation systems.

In addition to the technical activities that systems engineers perform, systems engineering also plays an important role in project and program management.  Systems engineering activities in project and program management span multiple system development life cycle stages.  Management activities associated with systems engineering fall into four major categories:

• Project planning and control. Planning is an essential element of systems engineering.  Without good plans, it’s difficult to know what to do and when to do it.  But control is also important.  Control involves two key areas: 1) monitoring project activities to determine task status, and 2) initiating corrective action when problems arise.  To monitor project tasks, the systems engineer sets up feedback mechanisms that provide accurate and timely status information.  These feedback mechanisms also identify problems that impede progress.  When a problem arises, the systems engineer shifts into analysis, to assess the impact of the problem and to determine ways to eliminate it or to minimize its effect.
  
• Technical planning.  Technical planning primarily involves determining how your systems engineering organization is structured and what activities it will perform.  For each project, the systems engineer creates the Systems Engineering Master Plan (SEMP), which describes how systems engineering is organized on the project and how systems engineering activities are carried out.  You must decide how you will involve your contractors, if at all, in conducting systems engineering activities during your project.
  
• Technical audits and reviews.  At each project milestone, you conduct a technical audit and review to determine how complete the technical work on the project is, before continuing.  The SEMP spells out which technical audits and reviews you will conduct, when you will conduct them, who will conduct them, and what effect the outcome of the technical review and/or audit will have on the project’s next steps.  All of these reviews and audits should be formal and documented as part of the project’s audit trail.
  
• Cost estimating.  Estimating project costs is a management activity.  The systems engineer takes a global view of the project and considers all costs, the “life cycle” costs of the project.  Thus, the systems engineer will estimate not just development costs, but also operation and maintenance costs (e.g., repair costs, replacement costs – including upgrades), the costs involved in fielding a system implementation (e.g., supply costs), and training costs.

Introducing Systems Engineering Practices on ITS Projects

Every ITS project has “success” as a goal.  Success is delivering a sound, reliable system that meets user needs and gets done within schedule and within budget.  Your best chance at achieving success comes from introducing good systems engineering practices on your project from the start and continuing to use them until the project ends (and beyond).  Let’s consider what some of these practices are.

Planning

Good systems engineering starts with planning.  Your plan is your roadmap for getting to success.  Systems engineering looks at the plan from the broadest possible view, considering all aspects of the project – from onset to operation and maintenance.  Systems engineering plans each step along the way and plans how you’re going to address the potential road blocks that you may find as you move from beginning to final delivery.  While it isn’t always possible to plan long projects in detail at the beginning, you can usually plan the early stages of a project in detail and the rest at a higher level.  Then, as you accomplish the tasks planned in detail, you learn enough about how the rest of the work should be done and can increase the level of detail in subsequent months.  This is a continual planning approach that lets you enhance your chances of planning the right work and planning the work right.

Your plans should synchronize with your cost estimates for the work to be done.  As you refine your plans, you must refine your cost estimates.  Your initial cost estimate for your project sets a ceiling that likely is difficult to raise.  Your re-planning and re-costing efforts must aim at staying within that cost ceiling.  However, if it appears that the original target was underestimated, your continual re-planning and re-costing gives you an early indication of cost growth while you can still control the growth.

Reviews and Audits

In a manufacturing process, quality is maintained through inspection of the finished goods and of the goods in process.  If quality inspectors detect flaws early enough in the manufacturing process, the manufacturer can correct the cause of the flaws and might salvage the product.  If the quality inspectors don’t detect the flaws until the product is finished, the flawed product is kept from the market place, protecting the manufacturer’s quality reputation.

In systems development, the quality inspection process consists of the reviews and audits that you perform.  You review documents, requirements, designs, program code, test results, and anything else that might give you an indication of the quality of the system that you’re building.  Audits ensure that you know, at any time, what your system consists of and where all of its parts reside.  Many texts on systems engineering, and even the systems engineering standards that exist, describe the types of reviews and audits that you can conduct.  As you introduce the practice of systems engineering, these should be your references.

Uncertainty and Risk Management

We can never be certain about the future.  However, we must make decisions when implementing a system that assume some confidence in what the future holds.  When uncertainty is cause by lack of knowledge or lack of information, we can apply systems engineering practices to help reduce that lack.  Systems engineers use mathematical and software tools and engineering techniques to increase our knowledge and information.  Prototyping, for example, is frequently used to gain a better understanding of what human-computer interfaces best suit the needs of our system’s users.  We can also use it as a “proof of concept” technique, where we integrate components (hardware and software) that have not been integrated before.  In integrating these components, we learn what works and what doesn’t work as an integrated unit.  We also learn how to integrate the components effectively (or at least, how not to).

Another tool that systems engineers use is simulation modeling.  A simulation can test different assumptions about how a system will work, under different conditions of load, before that system is built.  That helps identify the bottlenecks in the system that negatively affect performance and provides us clues about how to eliminate those bottlenecks.  Simulations can also help us identify when and where processes interfere with one another.  This allows us to revise the processes and improve workflow.

Any tool or technique that reduces our uncertainty helps reduce the risk involved in building a system.  The more we know about the possible outcomes of a decision or approach we plan to take, the more easily we can avoid the undesirable outcomes.  This is one aspect of risk management.  But there are other aspects as well.  Risks can come from “threats,” i.e., outside events that can have a negative impact on our efforts.  “Threat analysis” identifies those potential events that may harm us and allows us to estimate the potential cost to our project of each event’s occurrence.  Once we have identified “threats,” we can plan our strategy for mitigating their ability to harm us.

A Final Word

Systems engineering is hard work.  It brings as a reward the increased probability that we will successfully implement a useful, reliable system.  This paper won’t make you a systems engineer, but it does introduce the topic and provides you with a framework for understanding its value.  We hope you find it useful.

1.0 Introduction

Purpose of This Document

This document introduces the concept and practice of systems engineering, and discusses why it applies to the acquisition, development, and fielding of Intelligent Transportation Systems (ITS).  Systems engineering is a vast topic area and you can find many books and articles that cover it in detail.  You’ll find some of them listed in the References at the end of this document.  Reading this document (or those books and articles) won’t turn you into a systems engineer; that takes experience as well as more in-depth knowledge of systems engineering topics.  But our goal is to provide you with an overview of the subject and, we hope, spark your interest in pursuing it in more detail.

Systems engineering has proved beneficial in building systems in a number of different areas.  We believe that its use in implementing ITS systems will lead to better systems and more successful projects.  By the end of this document, we hope you agree.

Some Conventions Used in the Document

Throughout this document, we’ll mark “key” ideas by using a key symbol and some text that summarizes the idea we want to emphasize.  This marker looks like the following example:

We’ll float the key idea box in the margin next to the text that we are emphasizing.  This should help you extract the important concepts you’ll use when implementing systems engineering in your projects.

We’ll also use some examples to get our points across.  We’ll use examples that are related to transportation and ITS projects whenever possible, but may bring in ones from other areas when they help make the point.  We’ll mark our examples with simple graphics as well.  When we want to talk about schedules, costs, and technical areas, we’ll use the following graphics:

Intended Audience

The audience for this document is any or all of the following:

• Project managers of ITS projects
• Project team members on ITS projects
• Contractors and staff on ITS projects
• Anyone else interested in quality results in systems implementation

It is primarily geared, however, to ITS project managers, to help them succeed in their efforts.  Applying systems engineering in a project effort doesn’t guarantee success; external factors can always cause a project to fail.  However, experience shows that not applying a systems engineering approach when implementing a system is a good way to fail.  So, let’s look at what systems engineering is all about.

What is Systems Engineering?

A web page entitled “Systems Engineering Definitions” at a major Virginia university[1] has the following definitions (unchanged from how they appear on that page) from the university’s systems engineering faculty:

• Systems Engineering is the design and analysis of large-scale systems, where these activities add value to the enterprise or operation.
• Systems Engineering is a philosophy, grounded on the arts and sciences, supported by modeling and optimization methodologies, for the purpose of improving our understanding of the uncertain nature of the system and improving the decisionmaking process.
• Systems Engineering is the application of scientific principles to the design, development, implementation, and control of complex large-scale compositions of resources and processes which interact to achieve a common objective.
• Systems Engineering is just good problem solving.
• Systems Engineering is a structured approach to solving problems--a creative, iterative, decision-making process by which our understanding of logic, mathematics, and science is joined with our understanding of human and institutional needs, wants and limitations to conceive and refine alternatives that serve specific purposes within specific problem contexts.
• “A system is a set of elements so integrated as to aid in driving toward a defined goal.”  “Systems Engineering = Systems Analysis,” and “Systems Analysis equals operations research plus policy analysis.”

Our intent in providing such varying definitions is not to confuse you (or to poke fun at academia), but to illustrate that there is no one accepted, “perfect” definition of the term.

Why is Systems Engineering So Difficult to Define?

The term, “systems engineering,” was coined in the 1950s to describe a process used by engineers working on large and complex systems, usually incorporating computers and software.  A significant body of work in systems engineering derives from the US military’s efforts to build large command, control, and communications defense systems and the need to ensure that those systems worked when fielded.  Since the Department of Defense (DoD) hired contractors to build most of those systems, the DoD wanted those contractors to follow an approach that gave the highest possible likelihood of overall success, regardless of whom the DoD selected to build a particular system.  The DoD developed systems engineering standards and required contractors to conform to them.  Over time, these standards have evolved and matured and become accepted by more than just defense contractors.

But systems engineering applies to the building of any large system, regardless of type.  As practitioners began to write textbooks for others to use, some followed the DoD model and others looked at the basic principles involved, without necessarily using the formalism of the DoD model.  Thus, there are diverse descriptions and definitions throughout the literature of systems engineering.  If you look at any text among the references in this paper, you will find several with at least two different definitions.

People working in this field don’t always agree on what constitutes systems engineering.  When the International Council on Systems Engineering (INCOSE) was formed in 1990, all who attended the founding session had to contribute a definition of systems engineering.  No two were the same, but they could all be grouped into one of the following categories:

• Those who considered it a function solely of “systems engineers,” and consisting only of technical activities
• Those who saw it as a function solely of systems engineers, but consisting of both technical and management activities
• Those who believed that it consisted only of technical activities that anyone from a technical or scientific discipline could do
• Those who saw it as set of technical and management activities that anyone from a technical or scientific discipline could do

The technical activities in systems engineering involve an understanding of such tools as queuing theory, modeling and simulation, and linear programming.  It also helps to have a sound grounding in probability and statistics.  But, to be most effective, you also need domain knowledge.  Domain knowledge is a fundamental understanding of the technology and functions involved in the system under discussion.  For example, in an ITS system, domain knowledge may involve transportation engineering.  In an air traffic control system, domain knowledge includes understanding what an air traffic controller does.

Why Use Systems Engineering?

A lot of what’s involved in systems engineering is common sense and forethought – and good engineering principles that you learn in any engineering program.  Systems engineering is practiced along with many other engineering disciplines, whenever complex systems are involved.

One area where systems engineering is particularly important is in building software-intensive system, i.e., systems where software is a critical component for overall system success.  Software-intensive systems are notorious for being difficult to implement successfully.  The Standish Group, a national research firm, found, in a study they conducted in 1994, that only 16% of all software-intensive system development projects were “successful. [2]“  By “successful,” they meant that the project was completed on time, within budget, and with all of the initially planned functionality delivered.  In the same study, they found that 31% of all such projects were “failures,” i.e., were canceled prior to completion.  This is a failure to success ratio of nearly 2 to 1.  The remaining projects were completed with one or more of the following conditions: behind schedule, over budget, or with less functionality in the systems than originally planned.  Four years later, the Standish Group repeated its study and found improvement.  The percentage of “successful” projects had gone up to 28% and the number of “failures” had dropped to 28%.  The failure to success ratio was now 1 to 1.  Figure 1 below illustrates the Standish Group results from their two studies.

Figure 1

Project Resolutions

The use of systems engineering techniques wasn’t the sole cause for the improvement, but their use did play a key role.

The Challenges in System Engineering

When we apply systems engineering on any project, there are at least four major things that we want to accomplish.  These are:

• Identify and evaluate system alternatives
• Manage uncertainty and risk in our systems
• Design quality into our systems
• Handle any program management issues that arise

Identifying and Evaluating System Alternatives

There is no one right way to build a system.

When building a system, we make choices about how to accomplish our goal.   Systems engineering helps us make better choices by providing a framework within which to make those choices, one that helps us analyze and understand the consequences of our choices.  If we can compare the consequences of our choices on an even par, we improve the likelihood that we will make a good choice.  Again, we’re not dealing with a crystal ball that predicts the future accurately.  We’re going to use analysis and judgment and make estimates about what will happen.

When we build a system, we use systems engineering to do several things.  First, systems engineering techniques help us define a set of alternative ways of building it, that is, to define a set of “system design trade-offs.”  Second, system engineering provides techniques for evaluating each system design trade-off and determining which one gives us the best chance of successfully building the system we want.  We do this by assessing three things:

• Technical feasibility: This is our judgment on whether the system we want can and should be built using the technology in our design alternative.  Let’s take the following example:  We want to be able to process the input from 300 different sensors and have the information available at our Transportation Management Center (TMC).  One design alternative is to use existing communications lines to transmit the feed from all 300 sensors.  A second alternative is to use upgraded communications lines, with increased capacity, to transmit the feed from all 300 sensors.  A third alternative is to use a combination of existing communications lines and new lines, with part of the sensor array on each set of lines.

We conduct a technical trade-off analysis to assess each alternative and compare it to the others.  First we determine if the alternative is feasible.  If, for example, the existing communications lines are incapable of handling all 300 sensors, we can reject that alternative immediately.  That alternative can’t solve our problem.  If all three alternatives are feasible, then we consider other factors to determine which gives us the best solution.  From a technical viewpoint, the best alternative is the one that gives us the most capacity, because that provides the greatest flexibility and potential to handle growth, is the most reliable, because we avoid transmission errors and interruptions, and the most available (which may require building in some redundancy), since that provides the most consistent service.  However, the best technical solution isn’t necessarily the best solution.  To determine which one is best overall, we also need to consider cost and schedule factors.

• Cost feasibility: This is our judgment on whether the system we want can be built and operated for the money we intend to spend, using a specific design alternative.  We must consider each design alternative’s life-cycle costs.  We have to look at the initial system development costs, the costs to operate the system (including people costs), and the costs to maintain it (both hardware and software), including the cost of supplies associated with the system during its operation.  We evaluate these costs over the expected life of the system, i.e., the amount of time we expect the system to be in use.  If a system can be built for the development money that we have, but costs too much to operate and maintain, it does not have cost feasibility.  Take the alternative that we assessed as the best technical solution above, the one with the most capacity.  If the life-cycle costs of that alternative are beyond what we believe we can afford to invest in this solution, we can’t consider it the “best” solution.  Instead, we have to go with the “better” solution, if that solution is affordable.
  
• Schedule feasibility: This is our judgment on whether the system we want can be built, using a particular design alternative, within our implementation time frame.  There may be external factors to consider, such as whether a supplier can provide one or more necessary components on time.  There are always internal factors, such as staff availability for our project and staff skills, that we must consider when assessing schedule feasibility.  If a particular design alternative can’t be used successfully by our deadline, we have two choices: eliminate it or change the deadline.

What we’re after in assessing the technical, cost, and schedule aspects of our alternatives is to choose the alternative that is the best combination of the three.  It must be technically feasible, it must have a reasonable (affordable) cost, and it must be one that we can build in the required time frame.  That’s the solution that our trade-off study should lead us to.

Managing Uncertainty and Risk in Our Systems

If we could accurately predict the future, it would be easy to avoid mistakes and problems.  However, in real life, we need to deal with uncertainty and risk.  Systems engineering focuses on three aspects of uncertainty and risk management:

• Identification
• Analysis
• Mitigation

Identification deals with ferreting out the issues and potential problems that can affect our project.  Table 1 contains a set of questions that you can ask yourself, to help identify issues and potential problems that you might have.  The questions are at a high level.  As you answer them, other, more detailed questions will arise that you must also address.

Analysis assesses the importance of each issue and problem and determines what we know about it.  One way to analyze the issues and problems is to rank-order them according to their importance and the degree of uncertainty in our knowledge about them. This organizes them into a 3x3 matrix, as shown in Figure 2.

Figure 2
Assessment of Importance vs. Uncertainty

The upper right-hand cell of our matrix is the one of greatest concern.  These are highly important issues and problems about which we have a high degree of uncertainty, i.e., the inverse of knowledge.  The lower left-hand cell is of least significance.  The seven other cells are all addressed according to whatever scheme we deem best, but generally we address the issues in each cell, moving from left to right, top to bottom, as we move between cells.

Table 1
Key Questions

Area	Key Questions
Needs Analysis	What is wrong with the current situation? What needs does the ITS project fill? Have we clearly articulated the need? Do all ITS project stakeholders have a common understanding of the project’s goals and objectives?
Concept of Operations	Is our concept consistent with any Architecture(s) with which it must interact? Have we identified all intended users of the ITS system? How will each intended user interact with the ITS system? How is this different from the current situation, if at all? Do the intended users understand their role in the ITS system? Have we coordinated with all other agencies affected by this ITS system?
Requirements	What specific functions will this ITS project perform? Have we defined each function in detail? Have we identified all system interfaces? Are all system interfaces well defined? Have we defined our required system performance in quantifiable terms? Have we reviewed all requirements with stakeholders? Have we considered system availability requirements? Have we assessed our reliability and maintainability requirements? What derived requirements must we validate with our customer(s)? Have we considered what security our system needs?
System Architecture	What are the components of the ITS (e.g., TMC, ATIS)? How does this ITS system fit in with other ITS systems in the region? Is there an existing regional or project architecture based on the National ITS Architecture?
Allocated Requirements	Which components address which requirements? Is this allocation appropriate? Is this allocation complete? Are there any unaddressed requirements?
Detailed Design	Do the details meet the requirements? Is each component buildable? Are the interfaces satisfied? Are the details well documented? Do the details of the design map to all allocated requirements? Have we built sufficient redundancy into all mission-critical components?
Implementation (includes system integration)	What is the overall plan for building this ITS system? If capability will be phased in, what is our overall schedule? Can the ITS project be completed within cost and schedule? Have we considered human factors? Have we assessed the maintainability of the final system? Can we re-use/integrate existing components or capabilities?
Test	How will we know when a test is successful? Are all mission-critical functions thoroughly tested? What areas will we not test and why? Have we scheduled a full end-to-end test, integrating all interfaced systems?
System Acceptance	Do we have clear criteria for system completion? Have all users agreed with our completion criteria? Will our customers be satisfied with the system? Will we have adequate system documentation for all users and maintainers?
Operation and Maintenance	Have we assessed the full life-cycle costs of the system, including training, operation, and maintenance? Have we identified who will maintain the system? Do we have the system maintainer on-board? Do we have a schedule for upgrades and/or enhancements to this system? What growth in demand have we planned for?

Analysis continues with an evaluation of what it will cost us, in terms of time, money, and effort, to increase our knowledge about each of these items so that we are comfortable with the degree of certainty that we have.  Once we know that, we can decide how we want to invest our resources (time, money, and effort) in mitigation.

Mitigation involves performing some action to gain knowledge and reduce the risk related to an item.  The types of actions that we can perform include:

• Prototyping
• Modeling and simulation
• Experimentation
• Trade-off studies
• Market assessments
• Surveys and questionnaires

While that list is not comprehensive, it illustrates ways we can reduce uncertainty and mitigate risk.

Designing Quality into Our Systems

A US automobile manufacturer once had as its slogan, “Quality is Our Most Important Product.”  For systems engineers, that’s a motto to take to heart.  Systems engineering is all about building quality into systems.  Consider the following definition from one of the world’s most important standards-making bodies, the International Organization for Standardization (ISO), which defines quality as:

“…the totality of features of a product or service that bear on its ability to satisfy stated or implied needs.”

If we replace the phrase “product or service” with the word “system,” the definition fits our needs well.

Systems engineering helps us design quality into our systems by giving us tools to address factors that can affect quality negatively.  Among these are:

• Complexity:  If a system is complex, it consists of many different, dissimilar parts, each of which has to interoperate with many other parts for the system to work successfully.  If the design of a system is complex, there are many different parts, each of which has to be well understood to build the system successfully.  It may not be possible to reduce the complexity of the system, but it serves us well to reduce the complexity of its design.  The simpler the design of the system, the easier it will be to build.
  
• Inflexibility:  An inflexible system is one that cannot be easily adapted to changes in its environment.  An inflexible system design is one that cannot be easily adapted to changes in its requirements.  Either type of inflexibility affects quality negatively.  Without adding so much flexibility that we make the system or its design overly complex, we need to build in the ability to deal with changes.  Change is inevitable but we must accommodate it.  We must control how and when it is done.
  
• Lack of Standardized Components: When a system is built with one-of-a-kind components or with components built around a vendor’s proprietary products, it may lack maintainability.  Maintainability is a measure of how difficult it is to keep a system performing correctly as time passes and the system ages or requirements change.  With one-of-a-kind components, if a component ceases to function, it must be re-fabricated.  This can be anything from building a new hardware item that fails or reconstructing software that no longer meets requirements.  If the system depends on a vendor’s proprietary products, one must go back to that vendor if a component must be replaced.  If the vendor is no longer in business, a new product must be integrated into the system, leading to considerable effort to ensure that the new products works with all other system components in the same way that the original product did.  Ideally, one would build a system using commercially available products built using an industry standard.  Then, if a component fails, a compatible product should be available from multiple sources.  While it may be necessary to build some ITS systems with unique or proprietary components, more standardized system components are becoming available.  A good systems engineering approach would identify these components and adapt the system design to use them.

Another key point related to the lack of standardized components relates to the issue of system expansion or growth.  If you are dealing with standard components and system growth requires you to add more of them, it’s a simpler acquisition.  You’re buying more of a product that you can purchase from multiple sources, so there’s competition and the likelihood that you’ll get a reasonable price on those items.  On the other hand, if system growth requires you to add more nonstandard components, you’re faced with a more difficult acquisition.  The components are probably only available from the original supplier.  There’s no competition, so there’s no price break; you’re at the mercy of the original supplier.  Or worse, the original supplier may have gone out of business.  This may force you to re-engineer these components and hope that the re-engineered ones work the same as, and in conjunction with, the remaining original components.

• Reliability and Availability:  Reliability is a measure of a system’s ability to function without error.  In software, one increases reliability by finding and fixing errors in the system’s programs.  One technique for doing this is testing.  However, it is impossible to test all possible paths that a complex program can take during its execution.  Therefore, even extensive testing does not guarantee 100% reliability.  You should have contingency plans in place to handle situations where the system turns out to be less reliable than you hoped.  A good systems engineering approach will identify the risk areas that exist and propose reasonable contingency plans to address them.  Availability is a measure of how much outage, i.e., times when the system (or some component) is not functioning, a system has.  Availability requirements are frequently stated in terms of the number of “9s” you are trying to achieve.  For example, two “9s” means 99% availability, three “9s” means 99.9% availability, and four “9s” means 99.99% availability.  Since all components of a system are liable to fail at some point, the greater the availability we define as a requirement, the more backup systems or backup components we will need.  This comes at a cost, which can be very high.  Take a system that must operate 7 days a week, 24 hours a day.  There are 8,760 hours in a 365-day year.  If we call for 99% availability, the system can have only 87.6 hours of downtime during the year.  With 99.9% availability, only 8.76 hours of downtime are permitted.  With 99.99% availability, only .876 hours of downtime (about 53 minutes of downtime) are permitted.  The other issue in determining what the requirement for system availability is lies in deciding what constitutes system availability?  Do all components of the system have to be available all of the time?  Can some be down without affecting the measurement of overall system availability?  These issues are not trivial, particularly when there are contractual conditions involved.  Systems engineering considers and answers these types of questions.

Again, the list above is not all-inclusive, just illustrative.

One place where we can begin building quality into our system is in the Concept of Operations (see next section).  The Concept of Operations represents the “vision” of the system that is communicated to all “stakeholders,” i.e., individuals, organizations, and groups that have an interest in seeing the ITS project succeed.

Handling Project Management Issues that Arise

One key to a successful project is a sound project plan.  This means that a plan exists that has:

• Complete and comprehensive list of tasks that must be performed
• Adequate resources (labor, time, money) to accomplish all tasks
• Defined dependencies, where they exist, among tasks to show the order and precedence of task accomplishment
• Defined products or completion criteria for each task
• A method of measuring progress against plan

Having a good plan is not enough.  The plan must be communicated to and understood by all of the people working on the project.  Everyone who is scheduled to work on a task needs to know:

• The work product(s) of each task for which they are responsible
• When the task is scheduled to begin
• What the relationship of the task is to other tasks (i.e., on what completed effort does the task depend, what depends on the successful completion of the task)
• Who else will be working on that task
• When must the task be completed
• What resources other than people are required to perform the task

How one prepares a realistic schedule is beyond the scope of this paper.  However, once the project is underway, there are several important project management functions that systems engineering can support.  These include:

• Tracking
• Measuring progress
• Revising schedules when required
• Addressing obstacles to progress

Tracking involves ensuring that tasks begin and end on schedule and, if they don’t, determining what caused the variance from plan.  If the variance is unfavorable, i.e., the task begins or ends behind schedule, you must assess what prevents the scheduled start or completion of a task.  It could be the lack of an important resource or the failure of a related task to start or end on time.  Determining the root cause of the problem, however, is important if you intend to solve it.  If the variance is favorable, i.e., the task starts or completes ahead of schedule, you must still assess the impact of the early start or finish.  It may be possible to shift resources from a task completed early to work on efforts scheduled later in the project.  This may have the overall effect of shortening the project’s duration.  But it is also possible that the early start or finish of a task only means that some resource is underutilized for a period, since there is nowhere else on the project where you can use the resource.

Measuring progress means monitoring the consumption of resources on each task to determine whether the product(s) being delivered is(are) consistent with the time, effort, and money being expended.  It helps to have quantifiable, objective means of measuring, i.e., “metrics,” available so that you can be objective rather than subjective in determining how much progress has taken place.  “Percent complete” is a subjective measure unless you have a quantitative value to measure.  For example, if you know you have 50 different signals to install at intersections, you can say you’re 30% complete when 15 are installed.  It’s harder to determine when you’re “30% complete” in coding a program.

Revising schedules is a re-planning effort that should be considered periodically on projects that last at least a year.  Feedback on the accuracy of schedule estimates is necessary to help planners improve their planning efforts.  A quarterly project review should assess whether the overall schedule for the project is still realistic, given the progress made in the previous quarter.  If not, you should revise your estimates, given that you better understand what each of the uncompleted tasks requires, to reflect what work each task entails.  However, make it clear that you have revised the project plan if you lengthen it.  You want all of the project’s stakeholders to recognize the schedule change explicitly.

Addressing obstacles to progress is important when there is a problem with starting or completing one or more tasks on schedule.  Identifying the problem is not enough; you must also come up with a solution that removes the obstacle or reduces its impact on the overall project effort.  The goal is to complete the project as close to the original schedule as is feasible and still deliver a quality system.

Our Approach to Illustrating Systems Engineering Concepts

In the rest of this paper, we’ll cover many key topics in systems engineering, although at the “30,000 foot” level.  To help make these topics clearer, we’ll use the following artificial example of an ITS project to illustrate what we mean.

Metropolis and Gotham City, two cities less than 50 miles apart in the State of Bliss, are linked by Interstate Highway 105 (I-105).  There are a number of ITS elements deployed between the two cities, and there is a regional Transportation Management Center (TMC) that manages these elements.  The MetGo County regional planning commission envisions an upgrade to the ITS capabilities along the Metropolis-Gotham City corridor, to provide an enhanced infrastructure with the ability to handle both growth and additional functionality.  The regional planning commission is working with the State of Bliss Department of Transportation (BDOT) and its Chief Traffic Engineer for the MetGo region, M. D. Mandrake.  At the latest planning meeting, Chief Traffic Engineer Mandrake provided the assessment of existing capabilities and needs shown in Figure 3.

After the Chief Traffic Engineer gives this initial report, she and her staff begin work on preparing an ITS project to deal with the situation that they’ve assessed.  We’ll see how this works out in the rest of this document.

Well, that’s the approach that we’re going to use.  So, let’s get started.

Description of the Problem

“One of the major problems we have in managing our ITS elements in this region is the multiplicity of communications networks that we are using, along with the limited capacity of some of those networks.  We have the following communications networks currently in use: Our Dynamic Message Signs (DMS) are controlled by voice grade telephone lines We have separate voice grade telephone lines to use for entering data into our two Highway Advisory Radio (HAR) sites A third set of voice grade telephone lines links five existing video detection sites along I-105 We have an additional nine video surveillance sites along I-105 that are linked to our TMC by analog microwave This gives us four relatively low-speed communications networks that operate independently, with a large number of connections required at our regional TMC.  In addition, three of the networks are saturated and cannot handle additional communications traffic.  The fourth is close to its rated capacity. We have requests from police and fire agencies asking to share the surveillance data from our cameras.  In addition, those agencies have suggested the installation of additional surveillance cameras on secondary routes feeding to and from I-105, to assist them in responding to incidents in the area.  Agencies that have requested data sharing or additional camera installations are: Bliss State Highway Patrol Metropolis Police Department Metropolis Fire Department Gotham City Police Gotham City Fire and Rescue MetGo County Sheriff’s Department MetGo County Volunteer Fire and Rescue In addition to the upgrades for our communications networks, we need to upgrade our computers and consolidate existing software.  At present, each separate system (VMS, HAR, the two video surveillance groups) is on a separate computer.  There is no backup computer available for any of these systems. We would like to put all four systems onto a single large server and have a second computer, of similar capacity, available as a backup to the main processor.  If we do this, the software for each of the existing systems will require some modification to run under the operating system used by the server. We intend to initiate this upgrade by installing a single high-capacity telecommunication network along I-105.  This network will take the place of the existing four networks and will give us additional capacity.  After this network is installed, we can run branch networks to each of the police and fire agencies that have asked for data sharing.  We will require some additional network software to control the distribution of data among the agencies and our TMC, but we believe we can do this with commercial off-the-shelf software. Once we have our communications upgrade completed, we will upgrade our computer facility, replacing our existing processors with newer, more powerful machines.”

Figure 3
Description of the Problem

2.0 The System Life Cycle and Systems Engineering

So when and how does systems engineering fit in as you build a system.  Before getting too far along, we’ll define some terms.  Definitions help ensure that the author clearly communicates what’s meant by key terms.  There are three key terms in this discussion:

• System
• System life cycle
• Systems engineering

Definitions

A system is something whose parts work together to achieve a common goal.  That’s a basic definition of the term, but it has all the components of a more formal definition[3].  No definition of “system” states what size a system has to be.  In fact, the parts of a system can themselves be systems.  When we describe an ITS system, we call it a “system of systems,” i.e., each item in it is (at a high level) a system itself.  For example, you can find an Advanced Traveler Information System, an Advanced Traffic Management System, an Emergency Management System, a Transit Fleet Management System, an Automated Transit Information System, and a Traffic Control System in an ITS system.  Where you draw the boundaries of a system determines what its components are.  However, the wider you draw the boundaries, the more complex the interactions of the system’s parts become.

A system life cycle starts when the need for the system is conceived and ends when the system is discarded.  The duration of that life cycle varies.  Also, the way we break the life cycle into its component parts varies.  Although there are many different ways of breaking a system life cycle into its component parts, we’re going to use the following stages or phases in our discussion:

• Conception: This begins when you first recognize the need for a system (or new system version).  During this stage, you do two things: 1) perform a needs analysis, and 2) develop a concept of operations.  The needs analysis determines what problem you are trying to solve; the concept of operations is a user-oriented view or “vision” of the system that you want to implement.  You develop a concept of operations to help communicate your vision of the system to the other “stakeholders,” i.e., the other agencies, organizations, and individuals whose interests the system affects.  It also highlights the interfaces that the system has and ensures, through the publication and discussion of the concept of operations with others, that you have identified all interfaces.  During this stage, you establish the high-level requirements for the system.

In some depictions of the system life cycle, this stage is known by the name of one of its major products, the concept of operations.  When we discuss a graphical depiction of the systems life cycle later on, we’ll use the Concept of Operations title for this stage.

• Requirements analysis: This stage allows you to determine, in a more detailed manner than you did in the Conception stage, what you want the system to do.  If the system will replace an existing one, many requirements may be embodied in what the existing system does.  However, you must abstract from “how” the existing system satisfies those requirements to a description of “what” the new system must do.  It’s also possible that the new system will have requirements not found in the existing one; after all, that may be a reason you want to replace it.  This stage can run through several cycles itself, as you define, review, and refine your requirements.  A key point related to this phase is that the end product must be a set of requirements on whose meaning everyone agrees.

Frequently, as you’ll see later on, this stage is divided into two major parts, each identified by the detail level of the requirements developed during each part.  The two major parts are: High-level requirements and Detailed requirements.

• Design: This stage involves deciding “how” each requirement in the system is satisfied.  It heavily involves systems engineering, since there are many trade-off decisions to make.  It is also important to ensure, as the design evolves and becomes more detailed, that the design retains its internal consistency.  If groups working independently on parts of the system design fail to communicate effectively, it may be difficult to make the system’s pieces mesh during implementation.

It’s also common to view this stage in two major parts, High-level design and Detailed design.

• Implementation: This stage transforms the design into a product.  It may involve building parts of the system from scratch or integrating the different pieces that you buy.  However it is done, it makes the system real.
  
• Integration and Testing: The testing stage overlaps implementation.  It starts with unit testing, i.e., the testing of individual pieces of the system, as soon as any pieces are available.  It continues with string or integration testing, which ensures that individual pieces work together as intended and which tests interfaces at the lowest level within the system.  Last is system testing.  This involves a full end-to-end test of the complete system and comes in two major “flavors.”  The first is the final integration test by the system developer (whether that’s you or a contractor you hired).  The second is the “acceptance” test, where the end user(s) of the system make sure that it does what it’s supposed to do.  The acceptance test, however, is conducted in the next stage.
  
• System Acceptance: During this stage, you test the completed system prior to putting it into production use.  Users and operators should participate in the system acceptance process.  During this stage, in preparation for the system acceptance process, you must train the users and operators of the system so they understand how to use it to do their jobs.
  
• Operation and maintenance: This should be the longest stage, the one in which you use and maintain the system for the remainder of its existence.  Maintenance involves all processes that keep the system performing satisfactorily, including upgrades of equipment and software to later versions to enhance the system’s performance as its volume grows.  When you deal with an upgrade or major change to a system, the life cycle starts over, for that piece of the system that is being modified.  It’s also important to recognize that you have to keep monitoring the system’s performance against its original performance requirements.  This is particularly important as demand on the system increases.  You may have planned for some growth in demand; you need to know when you reach that limit.

At a relatively early point in the system life cycle, you make your “build versus buy” decision.  When exactly you make that decision depends on whether you intend to conduct the implementation with internal staff or with external staff, i.e., contractors.  In most ITS projects, public sector agencies hire contractors to work with them on the development of the ITS system.  If you plan to use contractors, you should bring them on board before you complete the requirements analysis stage.  That way, the contractors can help develop and refine the requirements and there’s a better chance that everyone will develop a common understanding of what the requirements mean.

The decision on whether to build or to buy is not simple.  Even if you decide to “build” the system, you may buy equipment and services during its building.  If you decide to “buy” the system, you still must devote resources from your organization to working with the vendor(s) who sell you the system.  “Buy” decisions also involve determining what type of contract to use.  For guidance on acquisitions, refer to The Road to Successful ITS Software Acquisition, Vols. I and II as well as other DOT guidance (see Where to Go to Get More Help, at the end of this document).

Systems engineering is the process by which we build quality into complex systems.  It uses a set of management and technical tools to analyze problems and provide structure to projects involving system development.  It focuses on ensuring that requirements are adequately defined early in the process and that the system built satisfies all defined requirements.  It ensures that systems are robust yet sufficiently flexible to meet a reasonable set of changing needs during the system’s life.  It helps manage projects to their cost and schedule constraints and keeps realism in project cost and schedule estimates.

The key to the above definition is that systems engineering is a process, not just a set of tools.  Let’s look at how this process relates to the system life cycle we discussed earlier.

Where in the Systems Life Cycle Does System Engineering Fit?

The easy answer to the above question is “everywhere.”  Systems engineering should be an integral part of any system project.  After all, it’s the process by which you build quality into your systems.  Let’s go back to the Metropolis-Gotham City example to illustrate this idea.

Chief Traffic Engineer Mandrake has her marching orders.  She kicks off her system effort with the first stage, Conception.

Conception.  This stage begins with a needs analysis.  That there is a need is clear but she must figure out what the system should do.  Asking one of the questions that we listed in Table 1, “What is the problem we’re trying to solve?” is the best way to start.

Chief Traffic Engineer Mandrake plans to develop a Concept of Operations for a system that her project will build.  The Concept of Operations is a formal document that provides a user-oriented view of the proposed new system.  The content of a Concept of Operations document is spelled out in a standard[4] from the Institute of Electronic and Electrical Engineers (IEEE), one the major U.S. standards bodies.  Figure 4 illustrates the outline of a Concept of Operations document, as specified in that standard.  Let’s look at some of the sections of that document and see what Mandrake needs to include.

The first few sections, Scope and Referenced documents, are mostly “boilerplate,” necessary, but not exciting.  The third section, Current system or situation, deals with what currently exists.  We described that briefly on pages 13 and 14, although not as thoroughly as the Concept of Operations document might need.  However, let’s assume that the information on those pages is adequate and go on to the fourth section, Justification for and nature of changes.

This section addresses the shortcomings of the current system that motivate either the development of a new one or the modification of the existing one.  Let’s review what these shortcomings are, as indicated in Figure 3 on pages 13 and 14:

• Three of the four existing networks are saturated and cannot handle additional communications traffic.  That’s important because local public agencies want additional surveillance cameras that will increase the communications workload.
  
• All four networks operate independently, thus coordinating them requires a complex switching system at the TMC.  When new capabilities are added at the TMC, the system will grow more complex unless redesigned or replaced.
  
• There are no backup computers for any of the existing systems.  If any one computer goes down, the system that runs on it is unavailable.


• Each system runs independently on its own computer.  This can be viewed as a shortcoming because it means that having a backup system (as of now) for each system would require four separate backup computers.
  

Those shortcomings are taken from the original problem statement Mandrake provided.  But she knows that there are some other operational issues that need to be addressed.  The seven public safety agencies asking to share data want to manage their incident response better.  To help them do this, the TMC needs to understand what data they need and when they need to get it.  The TMC also needs to know which agencies to notify and transmit surveillance data to and what criteria to use in notifying them and sending the data.

To address these operational issues, Mandrake must interact with representatives of those agencies, gather data, and use that data in compiling the Concept of Operations.  These interactions provide the basis for justifying the changes, determining exactly what they should be, and establishing the priority in which the changes should be implemented.  During this process, Mandrake may consider some changes that she ultimately decides to forego.  If so, the Concept of Operations provides a section in which to document these rejected options.  And this is one of the several benefits of a Concept of Operations document.  By writing down the options that she considered but rejected, along with the reasons she rejected them, Mandrake creates a tool for use in later stages of the project.  Should someone resurface one of these options, or a similar approach, later on in the project, Mandrake can review the original reasons for rejecting the option and get to closure on the discussion more easily and more quickly.  This is true even if additional data leads to a change in the decision.

The last subsection in the Justification section documents assumptions and constraints.  Putting assumptions and constraints in writing makes it easier to communicate them to all interested parties.  And, if any assumptions or constraints change during the project, Mandrake can go back and see what effect they had and what impact a changed assumption or constraint would have on how the system evolves.

The fifth Concept of Operations section, Concepts for the proposed system, gives Mandrake the opportunity to lay out the system and explain how she thinks it will work, once it’s in operation.  This is a key section, because it shows the intended users of the system how the new system will affect them.  It’s a good tool for setting or managing user expectation for the new system.  Note that one subsection specifically deals with operational policies and constraints related to the new system.  These might include items such as:

• Limits on the hours of operation of the system, or any part of the system
• Policies related to system access or access to secure areas of the system
• Constraints related to specific hardware that must be used

It’s in this section that Mandrake addresses the operational protocols that must exist for determining which agencies get data and when and how they get it.

The sixth Concept of Operations section, Operational scenarios, allows Mandrake to describe specific cases of system use.  Some like to use flow diagrams to illustrate operational scenarios; others like to use a narrative approach, sort of a “Day in the Life of …” approach.  We’ve provided a narrative example operational scenario in Figure 5 below.

Perspective – MetGo Regional TMC Operator

Joe, an operator at the MetGo Regional TMC, sits at his console.  The console includes a sloping display, raised/curved keyboard, proper distance from display to eye, and adjustable height for maximum comfort, all sound human ergonomics.  He logs onto the operating system and runs a software application that gives him “operational” access to the systems in the TMC.  Using his keyboard, pointing device, predefined operating instructions, and the TMC video wall, Joe monitors and operates the system.  Throughout the day, he monitors and controls functions and systems within the TMC, which include: Switch any surveillance camera images available on the network to any monitor(s) or display screen(s) in the TMC Modify or update special event data on any MetGo County, BDOT, Metropolis, or Gotham City websites Retrieve Preventive Maintenance schedules for system fine tuning Access the TMC information database for update purposes Change any HAR message within the MetGo County area, selecting from an approved message list Access any DMS in the TMC’s operating area and change messages to be displayed, selecting from an approved message list View traffic congestion and incident data fro both surface traffic street and freeways View automatic vehicle locator (AVL) data form the Metropolis and Gotham City Police Departments and the MetGo County Sheriff’s Department Alerted by the sudden decrease in vehicle speeds along Palm avenue, Joe selects the video images from Palm and Date Streets in Gotham City.  He notices that the northbound lanes on Palm are backing up due to a stalled car on the outside lane.  The problem is compounded by the fact that a construction crew is working next to the car, on the inside lane.  Looking more closely, he sees the construction crew assisting the driver out of his car and onto the street.  He switches the image to the large screen display – it looks like a medical emergency. Joe logs the incident using an Incident Management Report window.  He records information about the incident – reporting agency, contact name, phone, incident location, lanes affected, place on map, description, notification list, traffic management plan, and alternate routes.  The incident report is immediately and automatically routed to the Gotham City Police Department’s computer-aided dispatching (CAD) system, alerting operators who can react by dispatching police and emergency services.  Additionally, the report automatically updates local DMSs (already in use for the construction activity) to warn oncoming traffic of the incident and suggest alternate routes. Quick action addresses the medical emergency, possibly saving a life.  Travelers receive information about alternates for navigating more efficiently through or around the problem area.

Figure 5

Sample Operational Scenario

The seventh section of the Concept of Operations, Summary of inputs, and the eighth section, Analysis of proposed system, recap much of the material covered in earlier sections, but in a more condensed form.  The ninth section, Notes, is where you’d put additional material, things that don’t fit anywhere else, but help the reader understand what’s in the Concept of Operations.

Requirements Analysis.  In developing a Concept of Operations, Mandrake looked at user needs and desires and the Concept of Operations document reflects these.  However, those needs are usually stated in broad terms and need to be refined before they can guide the design of the desired system.  We deal with the subject of requirements in more detail in another paper in this series, Developing Functional Requirements for ITS Projects.  We’ll just cover a few points here.

For requirements to be most useful, they should be statements of what is desired, not descriptions of how the need should be satisfied.  They also have to be clear, complete, and correct.  It is the job of the systems engineer to ensure that these last three attributes are satisfied in the statement of requirements.  The systems engineer must review the statement of each requirement and look for ambiguities or possible misunderstandings that can arise from the way it is described.  One way to eliminate ambiguity is to conduct a System Requirements Review.  A System Requirements Review brings in representatives from each stakeholder and walks them through your understanding of each requirement.  The stakeholders have to agree: 1) with your interpretation of the written requirement, and 2) that this interpretation meets their needs.  Once everyone agrees on the meaning of the requirements, the next stage can begin.  (Note: Once the builder/integrator of the system is selected, the same kind of system requirements review should take place, only now with representatives of the builder/integrator present.  Otherwise, the builder/integrator may not have the same understanding as the other stakeholders of each requirement’s meaning.)

Once you have completed the development of your system’s requirements, the natural tendency is to “freeze” the requirements, i.e., assume that they are fixed and cannot be changed.  In reality, freezing requirements is counter-productive.  Since systems don’t happen overnight, it is likely there will be at least one legitimate major change in the system’s requirements before the system goes into production use.  Instead of freezing requirements, the best systems engineering practice controls changes to requirements through Configuration Management.  We’ll discuss Configuration Management later.

Design.  During this stage, you will make some high-level decisions about how you expect to implement the system.  For example, you may decide (or at least narrow the decision on) where to place the Traffic Management Center you plan as part of this system.  Some key systems engineering activities in this stage are to identify what trade-off decisions are needed and to conduct the trade-off studies to provide information on which to base these decisions.  As an example of a trade-off study, let’s say Chief Traffic Engineer Mandrake’s ITS plan includes Automated Vehicle Identification (AVI) tags to measure and report travel times at congested sites among the major arteries between Metropolis and Gotham City.  Since the type of tag you use has an effect on the specifics of system design, she must assess the types available to her against her selection criteria.  Table 2 is an example of the results that might occur from a trade-off study, with criteria, weights assigned to criteria, and relative ratings assigned to each factor assessed.  The trade-off studies clarify the Concept of Operations for the system.

Table 2
AVI Tag Trade-Off Study Matrix

					Type of Tag
		IVRT[5]				Toll				Beam
Criteria	Weight	Characteristic	Ranking	Score		Characteristic	Ranking		Score	Characteristic	Ranking	Score
Shelf Life (Recurring cost is most important	0.3	Battery powered, 2 to 5 years	2	0.6		Battery powered, 5 years	5		1.5	Beam-powered, unlimited	10	3.0
Minimum Quantity (Don’t want to order more tags than can be distributed)	0.2	Large quantities required to initiate an order	2	0.4		Low	10		2.0	Low	10	2.0
Cost (Initial cost is less important than recurring cost)	0.2	Low, tags contain no intelligence or protective packaging	10	2.0		High, tags contain intelligence to support toll-collection transactions	2		0.4	Medium, tags are passive and contain no intelligence	5	1.0
Adhesion Method	0.1	Sticker with adhesive backing attached to windshield	5	0-.5		Velcro backing attached to Velcro on windshield	8		0.8	Velcro backing attached to Velcro on windshield	8	0.8
Distribution Method (Has to consider that two major jurisdictions are involved)	0.2	Combined with state vehicle registration sticker that is renewed each year	5	1.0		Voluntary	10		2.0	Voluntary	10	2.0
Total	1.0			4.5					6.7			8.8

As we flesh out the details of the system in our design, we use system engineering activities to guide decisions on how to satisfy our requirements.  For example, we use modeling and simulations to test assumptions about a design choice’s ability to meet our performance requirements.  These performance requirements may be either an ability to handle a specified range of data volume over a specified time or an ability to process input and return a response in a specific window of time.

In this stage, we’ll also again make use of reviews.  Design reviews generally fall into two major categories: Preliminary Design Reviews and Critical Design Reviews.  You conduct the Preliminary Design Review (PDR) on each component of the system.  The PDR: 1) assesses the progress, technical adequacy, and risk mitigation involved in the selected design approach; 2) determines whether the item being reviewed is compatible with defined performance requirements; 3) estimates the degree of technical risk remaining; and 4) verifies the existence and compatibility of all interfaces involved.  You conduct the Critical Design Review (CDR) when the design for each component is complete.  The CDR: 1) ensures that the item under review satisfies all requirements allocated to it; 2) ensures that the item is compatible with all other items in the system; and 3) assesses whether there is any remaining risk associated with the item.  You also conduct trade-off studies here as part of your risk mitigation approach, although the trade-offs are more detailed.

Implementation.  This is the stage where all components of the system are either built or brought together into the overall system.  Your systems engineering activities focus on ensuring that what gets built matches what you designed.  In the implementation stage, systems engineering is synonymous with quality engineering.  When you are applying standards as part of your design approach, systems engineering activities focus on ensuring that the developers adhere to the standard (or justify a waiver from the standard, if necessary).

Hardware development is done on some ITS projects, at least in some States, but hardware development is not always done.  When your project is developing hardware, you may want to consider some of the issues associated with configuration management for hardware.  One place to start is another monograph in this series, The ITS Guide to Configuration Management.  That monograph covers configuration management topics for both hardware and software.

Integration and Testing.  You design your systems engineering activities to ensure that all relevant tests are included in the test effort.  Functional testing looks at each of the functions or operations that the system performs and validates that each function or operation satisfies its requirement(s).  Coverage testing looks at the different logical or physical paths through which data travels and ensures that at least the mission-critical ones are tested.  (It is usually impossible to test all paths in a complex system.)  Volume or stress testing looks at the system’s ability to handle the peak load expected during its operation.  The system may operate in “degraded” mode at peak conditions; this is only acceptable if that’s what you planned for it.  Integration or “end-to-end” testing looks at the system’s ability to interface all of its components with each other and to interface itself with all other systems with which it shares data.  For each type of test that you plan to conduct here, you devote your systems engineering activities to creating complete and comprehensive test sets that effectively exercise the system and its components.  In addition, your systems engineers analyze the test results and assess the degree to which the system satisfies its requirements.

This stage also deals with the actual installation of the system in pre-production use.  It includes preparing all of the instruction manuals for the system and all training materials, and training all users of the system.  The training conducted here is the initial system training.  Additional training is needed during the next stage as new users come on board and the system is enhanced with new features.  Your systems engineering activities during this stage focus on ensuring that the system is fielded so it does not disrupt the organization’s critical business functions.  For example, if you’re installing new sensors throughout an existing road network, you recognize it may be necessary to close certain travel lanes while installing them.  However, these closures should take place during low usage times on the affected roadways.

System Acceptance.  This is the final step before putting the system into operation and maintenance.  System acceptance implies that the operators and users take ownership of the system.  System engineering activities here focus on ensuring that all of the needs of the users and operators have been met.  Many of the systems engineering activities relate to developing a sound system acceptance test process, one that is both fair to the developer and ensures the needs of the system’s new owner(s) are met.

Operation and Maintenance.  The primary focus of systems engineering activities in this stage is performance.  If problems arise with the system during operation, you want to detect the problem, diagnose its cause, and deliver a solution with minimal delay.  It is difficult to predict the system engineering activities here, unless they deal with analysis for possible enhancements.  However, one key activity that does normally occur here is performance measurement.  The systems engineer wants to ensure that the system is meeting its performance goals, established during the initial project stages.  Of course, the system’s performance goals have to be quantifiable or you can’t measure them.

Before we go too much further, let’s distinguish between different types of performance measures one can apply.  One type of performance measure that’s been used in discussions of ITS systems relates to the performance of the transportation system itself, as influenced by the use of ITS systems to improve its efficiency.  One performance measure frequently discussed here is travel time/delay.  For example, a state DOT could measure total person-delay time (delay time per incident times number of persons delayed by the incident) or per-person delay time (average amount of delay experienced by all persons affected by an incident).  You can have two incidents with the same total person-delay time (e.g., 100 people delayed for two hours versus 2,000 people delayed an average of 6 minutes), but with a very different impact on the people affected.  Another type of performance measure relates to the performance of the ITS system itself, or its components.  In this case, you might measure actual network carrying capacity against planned capacity or actual system throughput against planned system throughput.  You could perform both types of performance measurement during the operations and maintenance stage.  But, since collecting and analyzing the data required for either type of performance measurement is expensive, you should only do this if you seriously plan to use the information you get as the basis for decision-making.

A good way to see how systems engineering and the system life cycle interact is to look at the life-cycle as what’s known as a “V” model or diagram of the system life cycle.  The “V” (or “VEE”) model is a way of relating the different stages in the system life cycle to one another.  Figure 6 illustrates the “V” model.

One reason for using the “V” model is to emphasize the importance of evaluation in all stages of a system project.  In the early stages of the system life cycle (the left leg of the “V” model), you are using mostly inspection and analysis as your evaluation tools.  In the later stages of the system life cycle (the right leg of the “V” model), the primary evaluation tool is testing.  Regardless of which leg of the “V” model you’re on, you’re interleaving evaluation efforts with system development activities.

Another reason for using the “V” model is to show an explicit relationship between work done on each side of the “V.”  Let’s look at the relationships between the two legs of the “V” model to show what this means.

Concept of Operations vs. Operations and Maintenance.  During the Conception or Concept of Operations stage of the system life cycle, you create a Concept of Operations for the system.  Concept of Operations is the term used in the “V” model for what we earlier called the Conception stage.  The Concept of Operations describes how the system will work once it’s built.  Therefore, it relates directly to the Operation and Maintenance stage, during which the system’s Concept of Operations is realized.

Looking at the explicit relationship between the two stages helps the systems engineer focus, at the beginning of the project, on issues associated with keeping the system in operation and effectively maintained once it’s built.  This long-range perspective is important in systems engineering.

Requirements Analysis vs. System Verification.  The Requirements Analysis stage isn’t explicitly shown in the “V” diagram, but it covers what the “V” shows as “High-Level Requirements” and “Detailed Requirements,” since those are the products of the Requirements Analysis stage.  In the Requirements Analysis stage, you determine what the system has to do, when it has to do it, and how well it has to perform.  In addition, the “V” diagram calls out the System Acceptance process as “System Verification.”  They mean the same thing.  In System Verification, you determine whether the system you built satisfies all system requirements.  Satisfying those requirements involves two different approaches:

• Verification – ensuring that all functions implemented in the system have been implemented correctly
• Validation – ensuring that the desired functions have been implemented in the delivered system

Another way of describing verification and validation is that verification is “building the system right” and validation is “building the right system.”

It’s important to recognize that you should be able to trace every requirement in the system to the system component that satisfies it.  One way to do this is through a requirements traceability matrix, which we discuss in Chapter 3.

System Design vs. Integration and Testing.  The “V” diagram in Figure 6 actually breaks this down into two components.  System Design is subdivided into “High-Level Design” and “Detailed Design.”  Directly across from “High-Level Design” is “Subsystem Verification,” which is a reasonable correlation.  Part of what’s done in the High-Level Design activity is to break the system down into its subsystems, each of which is assigned a major functional area of the overall system.  In Subsystem Verification, you’re integrating all of the components of the subsystem and testing the subsystem(s) as units.  It’s the appropriate way to look at how the two stages are correlated.

“Detailed Design” is directly across from the “Integration and Test” component.  Detailed Design relates to the definition of the complete system, the final design of all of the elements, at the level necessary to build it bottom-up.  The Integration and Test portion of the “V” is the start up on the leg where you bring all of the pieces together.  During this stage, you’ll test individual components as “units” and begin testing the individual units that interact with one another, preparing to fit them all together into individual subsystems.

The design of a system, in part, allocates functions to be performed by specific system components.  In addition, during design, we decide how we’re going to implement those functions.  That may involve developing algorithms (i.e., specific, detailed instructions on how to perform a function) or developing interfaces that allow devices in our system to interoperate.  The integration and testing of a system involves ensuring, at all levels of the system, that each piece works as it should and that all pieces successfully and accurately interact.

The design stage is the final decomposition stage of the decomposition leg of system development.  It’s the stage where you establish, at the most detailed level, your plan for how you will accomplish the work of the system.  The integration and testing stage is the first of the major re-composition stages, where you begin assembling the pieces of the system into an integrated whole.

Implementation.  This stage transforms the design of a system into actual products.  It transitions the work of system development from the conceptual level to the physical level.  It is an extremely critical stage because, once the system concepts are embodied in the system products, making changes becomes much more expensive.  Table 3 illustrates the experience software development organizations have had with the relative cost of making changes in software-intensive systems.  By their nature, ITS systems are software-intensive.

One activity associated with systems projects that doesn’t fall neatly into only one stage is deciding when and how you’re going to acquire the system or its components.  This is often categorized as the “make vs. buy” decision, although it’s rarely that simple.  Let’s look at how systems engineering fits in with making the acquisition decision.

Table 3
Comparison of Relative Costs By System Life Cycle Stage

System Stage	Minimum Cost	Maximum Cost
Requirements Analysis	$1	$1
Design	$3	$6
Implementation	$10	$40
Integration and Testing	$30	$70
Operation and Maintenance	$40	$1,000

Deciding on the Acquisition Method.

When you’re deciding whether to “make” or to “buy” the system, the decision is rarely straightforward.  But there are several key systems engineering activities that take place here.

One is developing alternative ways, at a high level, of satisfying the system’s requirements.  You need to define what alternatives you’ll consider in estimating a life cycle cost estimate for the system.  You may have defined your alternatives in an earlier stage.  If so, you can now look at the life cycle costs of each alternative as a method of deciding which one is the most feasible.  However, if you haven’t yet defined alternative ways of addressing system requirements, this is the point at which you should do it.  The life cycle of a system lasts from the time you conceive of a need for it to when the system is discarded.  Some of the types of alternatives that you may consider include:

• What types of communication systems you will use
• What types of data you will store and how
• What backup systems you will employ and where you will house them

The manner in which you answer each question will have an impact on the overall costs of the system.

In the early stages of the system life cycle, it’s relatively easy to recognize some costs, namely those involved in getting to a fielded system.  However, it is also easy to overlook what are probably the more significant costs of the system -- its operational and maintenance costs once it is fielded.  It is particularly easy to ov