Computer-Aided Dispatch - Traffic Management Center
Field Operational Test Final Evaluation Plan: State of Utah

Contract #: DTFH61-02-C-00061
ITS Program Assessment Support Contract
Task SA61004

Submitted to:

United States Department of Transportation
Mr. David Helman
Dr. Joseph Peters

Submitted by:

Science Applications International Corporation
8031 Greensboro Drive
M/S E-7-6
McLean, VA 22102

September 22, 2003

TABLE OF CONTENTS

1.0     Introduction

2.0     Project Overview

2.1     Background

2.2     Project Agencies

2.2.1     UDOT ITS Division

2.2.2     Utah Department of Public Safety

2.2.3     Valley Emergency Communications Center (VECC)

2.2.4     Salt Lake City (SLC)

2.2.5     Utah Transit Authority (UTA)

2.2.6     Federal Highway Administration (FHWA)

2.3    Existing Systems

2.3.1     Computer-Aided Dispatch Systems

2.3.2     CommuterLink Advanced Traffic Management System

2.3.3     511 Traveler Information System

2.3.4     Automated Vehicle Location and Geographic Information Systems

2.3.5     Signal Priority Systems

2.3.6     Mobile Computer Systems

2.3.7     UTA Systems

2.4     FOT Summary

3.0     FOT and Evaluation Goals and Objectives

3.1     Overview

3.2     FHWA Goals and Objectives

3.3     Utah Goals and Objectives

3.4     Evaluation Goals and Objectives

3.5     Combined Evaluation Goals and Objectives

4.0     Evaluation Approach

4.1     System Component Performance Study

4.1.1     Describing the FOT and Its Operating Environment

4.1.2     Identifying Key Performance Measures and the Design Basis

4.1.3     Documenting Performance Measures

4.1.4     Identifying Other Factors Affecting System Performance

4.1.5     Evaluating the Degree to which ITS Standards Were Incorporated

4.1.6     Identifying the Approach Used to Share Geographic Data

4.2   System Impact Assessment

4.2.1     Baseline System Performance

4.2.2     Documenting the---- Learning Curve

4.2.3     Systems Impact Evaluation

4.2.4     Deployment Specific Evaluation Components

4.3   Institutional Challenges Evaluation Approach

4.4   Technical Issues Evaluation Approach

4.5   Lessons Learned Assessment

4.6   Benefits Summary

5.0     Detailed Test Plans – Outline and Level of Effort

5.1   High-Level Outlines

5.1.1     System Performance Assessment

5.1.2     System Impact Study

5.1.3     UTA Interface

5.1.4     Institutional Challenges Assessment

5.1.5     Technical Issues Assessment

5.1.6     Lessons Learned Summary

5.1.7     Benefits Summary

5.2   Work Breakdown Structure

5.3   Estimated Resource Requirements

6.0     Evaluation Management Plan

7.0     SCHEDULES AND MILESTONES

7.1   Evaluation Schedule

7.2   Evaluation Milestones and Deliverables

7.3   Data Management Plan

7.3.1     Overview

7.3.2     Data Storage

7.3.3     Downloading and Processing Data

7.3.4     Analysis Tools and Data Queries

7.3.5     Security Management, Safeguards, and Controls

7.4     Quality Control Review for all Deliverables

LIST OF FIGURES

Figure 2-1. Proposed System Architecture
Figure 2-2. Utah CAD – TMC FOT – Existing Infrastructure
Figure 4-1. Expected System Impacts.........................Error! Bookmark not defined
Figure 5-1. Test Plan Work Breakdown Structure.
Figure 7-1. Draft and Final Deliverables Quality Control Review Process.

LIST OF TABLES

Table 2-1. Existing ATMS-EMS Integration Status
Table 2-2. CAD – TMC FOT: Participating Agencies
Table 3-1. Utah-Expected FOT Benefits
Table 3-2. Evaluation Goals and Objectives
Table 3-3. Combined Evaluation Goals and Objectives
Table 4-1. System Performance Measures
Table 4-2. System Performance Measures.
Table 4-3. System Impact Experimental Design for Mobility
Table 4-4. System Impact Experimental Design for Capacity/Throughput
Table 4-6. Traveler Information Assessment
Table 6-1. Evaluation Staffing Plan.
Table 7-1. Evaluation Schedule.
Table 7-2. Evaluation Milestones and Deliverables.

1.0     Introduction

Reducing traffic related fatalities and improving emergency response capabilities are two primary goals of the U.S. Department of Transportation’s (USDOT’s) Intelligent Transportation Systems (ITS) Public Safety Program. To help achieve these goals, the ITS Public Safety Program is committed to:

• Improving incident detection and notification.
• Reducing emergency response times.
• Improving information flows between emergency response agencies (real-time wireless communications links, integration of systems).[1]

The current Federal Highway Administration (FHWA)- funded Computer-Aided Dispatch – Traffic Management Center (CAD-TMC) integration and data exchange Field Operational Test (FOT), is one of many initiatives intended to meet the program goals. Most major metropolitan areas in the United States rely on some type of advanced traffic management system(s) (ATMS) to help manage mobility, congestion, and incident response. Many states have installed an extensive infrastructure of remote cameras, loop detectors, and other ITS applications that provide traffic management services. These systems are operated from centralized TMCs, where traffic-related information is received and processed and appropriate remedial actions are deployed and coordinated. However, to date, many of these systems are not integrated with the CAD systems used by public safety and law enforcement agencies.[2]
To demonstrate how the integration of CAD and TMC systems can improve incident response capabilities and technical and how institutional barriers can be overcome, the USDOT is sponsoring two FOTs that will integrate CAD-TMC systems in Utah and Washington State, respectively. As stated in the Request for Proposals (RFP) for the CAD-TMC Integration FOT evaluation:

Transportation, law enforcement, fire, and emergency medical personnel are discovering significant improvements in public safety operations can be made when information is shared across organizations and jurisdictions. Equipment and personnel can be more efficiently deployed, incidents can be cleared faster, and incident scenes can be made safer for the responders and the traveling public. To date there has been little effort to integrate highway traffic management with public safety systems. Nor have systems supporting public safety operations been developed in the context of a regional ITS architecture or ITS standards. Most existing CAD systems are proprietary and not equipped to easily share information with systems with dissimilar interfaces. Further complicating integration are various data, message formats and standards used by public safety agencies and transportation agencies. Nevertheless, CAD and ATMS systems can be integrated and data can be shared, provided that a number of related institutional and technical issues are addressed. New procedures and methods of response that capitalize on the availability of the shared information must also be developed.[3]

This document presents the Evaluation Team’s plan for conducting the evaluation of the FOT in the state of Utah. A companion document exists for the evaluation of the Washington deployment. Each Evaluation Plan includes the experimental design for testing hypotheses at one of the FOT sites. Each plan also includes a detailed discussion of goals and objectives; a work break down structure (WBS); description of the evaluation management structure and schedule; and high-level outlines of deliverables’ content. The Evaluation Plans also contains a discussion of the target audiences for the report and includes a proposed report distribution list. The remaining sections of the Utah Evaluation Plan address the particular issues identified in the RFP:       

• Section 2: Project Overview. This section includes a description of: all relevant Utah agency systems included in the FOT: operational roles and responsibilities of each participant in the FOT; and proposed project plans and activities.
• Section 3: FOT and Evaluation Goals and Objectives. This section discusses the goals and objectives for the FOT, as identified by the State, and the evaluation goals and objectives identified by the Evaluation Team, including:
  • System Performance Study – Did the system perform as expected?
  • System Impact Study – How did the FOT help improve incident response capabilities at the state level and between state agencies and local, county, and municipal agencies?
  • Traveler Information – Was the additional information provided to the traveling public and the news media through the Utah DOT’s (UDOT) Web page, 511 systems beneficial?
  • Institutional and Technical Issues – what were these and how were they resolved?
  • Lessons Learned – what were these and how are they useful?
  • Benefits Summary – quantitative and qualitative benefits.
• Section 4: Evaluation Approach. This section includes the high-level outlines of the contents of each test plan deliverables, and an overall summary of evaluation approach, including:
  • The identification of the individual test plans to be developed.
  • The level of effort allocated for carrying out each of the individual test plans.
  • The Work Breakdown Structure (WBS).
• Section 5: Detailed Test Plans – Outline and Level of Effort. This section includes descriptions of the specific test plan outlines, level of effort, institutional challenges, and technical issues to be addressed in each FOT.
• Section 6: Evaluation Management Structure. This section includes a description of the evaluation management structure.
• Section 7: Schedule and Milestones. This section includes a summary of the evaluation schedule, including milestones for data collection, evaluations, and deliverables, and the data management approach. This section also includes a high-level outline of the final report and the evaluation findings briefing. The discussion of the target audiences for the report and includes a proposed report distribution list is also included in this section of the report.

2.0    Project Overview

2.1    Background

The Utah Department of Transportation – Intelligent Transportation System (UDOT - ITS) program, otherwise known as CommuterLink, was officially launched on April 27, 1999 with the opening of the UDOT Traffic Operations Center (TOC). The UDOT TOC, through the CommuterLink program, provides comprehensive traffic management and operational services and is currently fully integrated with all city and county transportation departments located within the Salt Lake Valley. CommuterLink has rapidly become a leading force in all aspects of traffic management and ITS in the State. Through CommuterLink, a consortium of State, county, and city transportation agencies working together with emergency management and private media organizations to manage, coordinate, and disseminate information about transportation- related events such as incidents, emergencies, accidents, planned roadway closures, special events, homeland security, and disasters. Communication links are also established with the University of Utah traffic lab; Salt Lake City police and fire dispatch centers; County Sheriff EOC; and the Valley Emergency Communication Center (VECC) for distribution and sharing of CommuterLink Closed Circuit Television (CCTV) images and traffic flow map displays. As part of the CommuterLink program, UDOT and the Utah Highway Patrol (UHP) have made significant strides in integrating emergency response operations. UDOT’s integrated the Department’s Incident Management Teams (IMT), which participate in police training and staff briefings, and are dispatched directly through the Department of Public Safety (DPS) Dispatch center. DPS, on the other hand, is provided a rent-free space within the TOC in exchange for after-hours support of DOT functions. This relationship and sharing of resources has evolved to such an extent, that all four entities involved (DOT operations, DPS, UHP, and IMT) have become inseparable, with each fully cognizant and appreciative of the others’ roles and agendas.A summary of the current ATMS-EMS integration in Utah is shown in Table 2-1.[4]

Table 2-1. Existing ATMS-EMS Integration Status

Goal	Objective	Current Status
TOC – DPS Integration	To provide a fully integrated communication system between the TOC and DPS with respect to telephone, radio, CAD, and ATMS for purposes of managing incidents, dispatches, and other routine activities in response to traffic and public safety-generated events.	Telephone – hot lines established. Radio – 800-MHz completed with talk channels assignments established for UDOT, DPS, VECC, SLC police and fire. CAD/ATMS – Shared CAD completed. ATMS integration scheduled for summer 2003. Real-time ATMS and CAD information completed to Media (dedicated fiber) and public (www.commuterlink.utah.gov. pager alert system and 511).
CommuterLink interties for sharing video and data	The CommuterLink system shall support an electronic interface with other CAD systems to support interagency dispatching (electronic data interface, video).	Integration completed between UDOT/DPS CAD vendor (CIS). Interties pending available funding: - VECC (Spillman) - SLC PD (Versaterm) - SLC Fire (FDM)
Application enhancements	Enhance systems to interface with a geographic information system (GIS) and third party map display to show real-time status of incidents (incident ID, type, status, etc.); vehicle location and status; geo-referenced location-specific conditions; and other relevant geo-coded information.	GIS map and integration with ATMS is complete in the TOC. Distribution to other sites/media is in early stages of deployment.
Field unit support	Provide support for routing planned and en-route guidance to dispatched vehicles. Ability to generate selected routes based on position information, incident location, and real-time roadway network status information. Ability of calculated estimated time of arrival information.	Field unit support pending completion of other tasks and identification of funding source.
In-vehicle navigation systems	Integrate communications and data transfers from the ATMS/CAD to field units via Mobile Data Terminals (MDTs) and Automated Vehicle Locations (AVLs) as follows. Utilize MDTs to support automatic central dispatching, self-dispatching, resource request, and State database access. Provide pre-arrival information on hazardous materials and other on-scene conditions, including video. Support for messaging between vehicles, on-scene reporting, and access to records management systems. Use of UPS-based AVL receivers integrated with in-vehicle MDTs to report vehicle positioning information for centralized resource tracking, selection, and monitoring.	MDTs deployed UHP, IMT, and some UDOT vehicles. Deployment to secondary units (i.e., snowplows, service patrols, structural engineers, etc.), pending additional funding. System enhancements to complete stated objectives pending additional funding.

2.2   Project Agencies

The State and local government agencies participating in the CAD-TMC Integration FOT are listed in Table 2-2. The table also provides a summary of each agency’s programmatic responsibilities and the vendor or contractor supporting the respective FOT activities.

Table 2-2. CAD – TMC FOT: Participating Agencies

2.2.1    UDOT ITS Division

The UDOT ITS Division is the lead agency for the CAD-TMC FOT. The Division was created less than 3 years ago to manage the Utah ITS program. Since 1999, the Division has deployed an ITS infrastructure that now manages over 100 miles of freeway, arterial, and transit routes within the Salt Lake Valley. This infrastructure was designed, implemented, and is now shared with local transportation departments. Information and device access at varying levels of control is also provided to other partners, including local police, fire, and dispatch centers.

2.2.2        Utah Department of Public Safety

As the primary dispatch center for all highway patrol units throughout the state, and co-located at the TOC, DPS has been one of Utah’s best advocates for the sharing of information not only between other public safety agencies, but to the media and general public as well. DPS was the first agency within the State to provide relatively unfettered access to CAD information. With development and implementation of the CommuterLink Website (www.commuterlink.utah.gov), DPS is furthering this sharing of data to the public by providing filtered access to records on the Web in real time.

DPS is also a key participant, since this agency represents the local interests of the police, highway patrol, and other dispatch centers. DPS hosts quarterly coordination and management meetings with these agencies at the TOC, and is largely responsible for encouraging and demonstrating the usefulness of coordinating with UDOT.

2.2.3        Valley Emergency Communications Center (VECC)

VECC serves as the Primary Safety Answering Point (PSAP) for all of Salt Lake County, and coordinates closely with the Salt Lake City police, fire, and DPS dispatchers at the TOC. VECC’s 25,000 square-foot center handles dispatching for 15 fire and eight law enforcement agencies, with agents fielding about 3,500 telephone calls, and dispatching nearly 2,000 incidents per average day.

As a CommuterLink partner, UDOT has active agreements in place with VECC to provide ATMS support from the TOC, including real-time video feeds and access to the video switch for camera selection control.

2.2.4        Salt Lake City (SLC)

Through independent departments, the City provides police, fire, transportation, and airport security services to all areas within its 111-square mile jurisdiction. The City transportation department has its own Traffic Control Center (TCC) that operates as an extension of the CommuterLink TOC. Additionally, the City Police and Fire departments each have an ATMS fiber connection to monitor transportation and video feeds from the TOC. Camera control is supported by operators from either the City TCC or UDOT TOC.

2.2.5     Utah Transit Authority (UTA)

As a very active member of CommuterLink with an ITS program, UTA is setting new milestones from which other transit agencies measure their programs. Responsible for a number of transit operations, UTA has a strong congestion management program and has worked closely with UDOT and SLC officials to deploy priority systems for UTA’s light rail (TRAX) and bus services at signalized intersections. UTA has also deployed AVL; various traveler information programs (e.g., use of PDAs to access real-time schedule information); and has become the first transit agency in the nation to implement a connection-protection program designed to alert passengers and bus drivers alike of delayed light rail vehicles, which holds the bus until connecting passengers arrive.

2.2.6     Federal Highway Administration (FHWA)

A key member in the CommuterLink program, the local FHWA division is a major supporting partner of this statewide program, and will continue to be heavily involved as advisor, liaison to Washington, and local oversight administrator for this project.

2.3     Existing Systems

2.3.1    Computer-Aided Dispatch Systems

Several agencies have computer-aided dispatch (CAD) systems. The UDOT ITS Division has procured and installed a new CAD system for joint use by the TOC DOT dispatchers and UHP-DPS dispatchers. The UTA is also in the process of implementing its own CAD system to help manage its large fleet of personnel and vehicles.

2.3.2     CommuterLink Advanced Traffic Management System

The CommuterLink ATMS consists of several different systems installed at the roadway to actively manage, coordinate, and disseminate transportation-related events and information. These systems include:

• 260 Closed-Circuit Television Cameras (CCTV)
• 70 Variable Message Signs (VMS)
• 640 Centrally Controlled Traffic Signal Systems
• 23 Ramp Meters
• 191 Freeway Traffic Monitoring Stations
• 35 Vehicle Detection Systems
• 7 Highway Advisory Radio
• 35 Roadway Weather Information Systems (RWIS)

2.3.3    511 Traveler Information System

Utah was the first state to use interactive voice recognition (IVR) technology to disseminate real-time information on incidents, road conditions, and roadway weather conditions.

2.3.4    Automated Vehicle Location and Geographic Information Systems

DPS has completed an off-site pilot implementation of AVL/GIS for tracking UHP vehicles. Other projects are currently under review with the CommuterLink system to integrate the ITS components with GIS. Current efforts include configuration management of all ATMS devices with future applications being developed to identify shortest route; real-time estimated times of arrival (ETAs) based upon speed flow data; incident monitoring; and real-time dissemination to PDAs or MCS devices for use by field units.

2.3.5    Signal Priority Systems

The UTA, in cooperation with UDOT, has deployed signal priority systems for both light rail (TRAX) and buses at signalized intersections.

2.3.6     Mobile Computer Systems

The mobile computer systems (MCS) units are essentially laptop computers that are installed in field units that communicate with a command center via wireless communications. These MCS units are typically cellular digital packet data, although new generation and higher speed wireless options such as GSM/GPRS will be widely available in large urban areas within the next few years. The uses for MCSs are unlimited, and constrained only by the communication network and application.

2.3.7     UTA Systems

UTA receives approximately 25 video feeds from other agencies at its dispatch center. These views were selected based on their ability to cover key UTA facilities such as light rail stations. In addition, UTA uses a CAD system to support the operations of its bus and rail dispatchers, including revenue, supervisory, and maintenance vehicles. This system monitors the status of radio system communications and current incidents.

UTA dispatchers have a significant degree of responsibility for making on-the-spot decisions about the operational response to unexpected events (e.g., rerouting, assigning replacement/ additional vehicles), and these decisions have a significant effect on UTA’s operational efficiency. The types of unexpected events to which UTA dispatchers respond include traffic delays, equipment breakdowns, and driver absences. Currently, the information available to UTA dispatchers for these important decisions is primarily limited to radio communications with other UTA personnel, video feeds, and the media.

UTA currently has no direct data communications link or formal protocol for sharing incident information with the TMC, public safety, or law enforcement. Traffic or other incidents along UTA routes can have a tremendous impact on UTA’s ability to maintain its schedules; however, UTA often first learns of an incident when UTA personnel encounter it on the street. Subsequent transit operations can be rerouted, and the initial vehicle may be significantly delayed. When this incident is already known to another agency in the region, sharing this information with UTA could help overall operations tremendously. Conversely, when UTA is the first to encounter an incident, UTA personnel could help by sharing the information with other agencies. In addition, once rerouting is established, UTA often has difficulty determining when an incident is cleared so that normal operations may resume.

UTA uses its Website and telephone information system to publish information for the public on longer-term operational responses to road conditions (e.g., rerouting due to construction activity). This information is also available through the 511 Website and telephone information system. UTA does not, however, currently attempt to provide information to the public about shorter-term operational decisions that affect revenue service. One reason that UTA does not provide this information is that it does not feel it has reliable and current information as the basis for these types of reports.

The primary role and responsibility of UTA in the FOT will be to share data with the TMC and other responding agencies using the standardized incident status messages. Since UTA is not an incident responder, the data shared by UTA will primarily be limited to notification about incidents first detected by UTA personnel. UTA will be monitoring incidents already in progress that affect its operations, and would normally only provide notification about incidents not previously reported by other agencies.

This enhanced information for UTA is expected to significantly improve its ability to quickly implement effective reroutings after incidents occur, and to quickly end each rerouting once the incident is cleared. Subsequently, UTA also expects to build enough confidence in this information to start publishing short-term operational responses to the public using the UTA and 511 Websites and telephone information systems.

2.4     FOT Summary

The FOT will include the following elements:

• Develop a common message set, structured in a uniform and open format, to enable the exchange of information between multiple agencies with unique requirements, policies, and operating environments. Two interagency-shared data messages (ISDM) are planned: the interagency service requests (ISRs) and the interagency ATMS message (IAM). The ISR specifically requests services rendered by public safety agencies and secondary responder services. ISRs may be between CAD systems and/or between CAD systems and ATMS to specifically request public safety and secondary responder services. The IAM relates to traffic condition advisories and traffic control requests between CAD systems and the ATMS.
• Support the ISRs by data specification sets (DSS) that incorporate the standard data elements found in all CAD Systems. The DSSs will specify an XML application to Import and Export (I/X) the data sets. The DSS will also specify the data standards for each element, including IEEE 1512-2000, 1512.1, and 1512.2 as available and applicable. The ISR-DSS specifications will be in the public domain.
• Select a commonly used operating system and language (e.g., Windows 2000 and Visual Basic) to develop legacy system interfaces (LSI) between existing UHP and UDOT systems to enable information exchange. The LSI will be a stand-alone server program in the public domain designed for nationwide application at TMCs for the ISR and IAM messages between different vendor CAD systems and between CAD systems and ATMS.
• Develop legacy system interfaces between the State systems and county and municipal government systems (VECC, Salt Lake City).
• Integrate the new UTA CAD system currently under development.
• Continue UDOT ITS Division-developed unique browser-based Event Tracking System (ETS) to manage planned events (i.e., roadway construction), and to update these events in real-time for subsequent dissemination to the traveling public. The ETS is being deployed statewide, and will be used by local city, county, and state agencies. Information from both the ETS and the existing 511 system will be updated and integrated into the CommuterLink traffic management system using Extendable Markup Language (XML).

The system architecture developed for the FOT is shown in Figure 2-1. [5]

Figure 2-1. Proposed System Architecture

For the FOT, the participating agencies will use their current CAD and related technologies as depicted in Figure 2-2.[6] The TOC’s CommuterLink will continue to provide the current ITS technology, including CCTV roadway coverage. UDOT currently distributes CommuterLink’s CCTV video images and image selection controls to SLCPD, SLCFD, VECC, and UTA. Traffic information is also available via CommuterLink’s Web pages (www.commuterlink.gov ) . The FOT will then test the specific effects of the introduction of the shared data identified above, facilitated by CAD-to-CAD ISRs and CAD-to-ATMS IAMs, on the performance of responders and related benefits.

Figure 2-2. Utah CAD – TMC FOT – Existing Infrastructure

3.0     FOT and Evaluation Goals and Objectives

3.1     Overview

The RFP for the CAD-TMC FOT evaluation states that:

As part of the Evaluation Planning activities, it is critical that the contractor collaborate with WSDOT and UDOT to ensure evaluation needs are accommodated by the FOT. Collaboration will help to identify FOT benefits and challenges that are not quantifiable and assist in sharing lessons and approaches between the two sites. Collaboration provides the contractor the opportunity to coordinate with the FOT teams on issues that may arise including:

• Ability to collect, archive, and provide operational data needed
• FOT teams desired evaluation topics not initially identified by the contractor,
• FOT teams approach to implementing applicable standards such as IEEE and NTCIP,
• Access to operational personnel and facilities; and schedule information; and
• Address concerns identified during these reviews.[7]

The Evaluation Team recognizes that meeting these requirements is essential to ensuring a successful evaluation and has worked to develop evaluation goals and objectives that:

• Meet the stated evaluation needs of FHWA and the States of Utah and Washington.
• Enable the Evaluation Team to assess both quantitative and qualitative impacts.
• Rely on sources of data currently available for establishing the “before project” baseline and measuring the “after project” impact.
• Enable a “realistic” assessment of impacts, that is, assess impacts and benefits that are directly tied to the results of the FOT and do not overstate or understate the results.
• Are “flexible” in nature so that issues such as changes in the project schedule or availability of additional data do not require redefining significantly revising the evaluation.

The goals and objectives for this evaluation were developed using an iterative approach involving extensive review by FHWA and the states. First, the Evaluation Team reviewed all available project documentation, including the application submitted to FHWA by each state in response to FHWA’s Request for Applications distributed on May 16, 2002. Based on this review, the Evaluation Team presented high-level goals and objectives in its proposal submitted in response to FHWA’s March 7, 2003 RFP. These proposed goals and objectives were reviewed with the FHWA COTR and the Mitretek analyst on May 6, 2003, and then again during a June 2, 2003 kick-off meeting with Washington State. The proposed goals and objectives were revised based on these meetings, and presented to the FHWA COTR and the Mitretek analyst on June 16, 2003, and to Utah and Washington State during evaluation strategy briefings conducted on June 25 and June 26, 2003, respectively. The final evaluation and objectives presented in this plan reflect the input obtained from FHWA and two states throughout this process.

The remainder of this section of the Evaluation Plan is structured as follows:

• 3.2 – A summary of the Evaluation Team’s understanding of FHWA’s goals and objectives for the FOT.
• 3.3 – A summary of the Evaluation Team’s understanding of Utah’s goals and objectives.
• 3.4 – The goals and objectives developed for the evaluation based on the Evaluation Team’s review of project documents and feedback obtained from FHWA and Utah.
• 3.5 – A comparison of the goals and objectives established by the Evaluation Team with those established by FHWA and Utah for the FOT, with the intent of demonstrating how the evaluation goals and objectives track to the FOT goals and objectives.

3.2     FHWA Goals and Objectives

The Evaluation Team used the following high-level FHWA-established FOT goals and objectives for the FOT as the starting point for developing goals and objectives for the evaluation:

• The FOT will demonstrate the automating the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors.
• The FOT will incorporate ITS standards such as IEEE 1512 and NTCIP into the integration of public safety and transportation information systems. Other standards areas that will have to be addressed are those pertaining to GIS.
• The FOT will extend the level of integration to include secondary responders such as utilities; towing and recovery; public works; and highway maintenance personnel.

FHWA has also identified a number of specific quantitative goals and objectives to be assessed during the evaluation, in particular, to:

• Determine how the FOT enhances communications among responders.
• Assess the extent to which the FOT enhances efficiency in documenting incidents.
• Determine how the FOT enhances on-scene operations.
• Measure the extent to which the FOT reduces incident clearance times.

FHWA has also specified that the final evaluation report include an assessment of institutional challenges and technical issues, and a summary of lessons learned and benefits, both qualitative and quantitative.

3.3     Utah Goals and Objectives

Utah has already made significant progress in establishing a well coordinated and cooperative working relationship between the UDOT and the UHP. The agencies developed a close working relationship during the 2002 Winter Olympics, which led to significant coordination of emergency response activities. The result of this coordination has been a significant reduction in emergency response times.[8] UDOT and UHP have also co-located dispatchers at the Statewide TMC and have the ability to monitor the each agency’s systems during an incident. To this end, the project represents the logical next step in expanding the working relationship between the two agencies, as well as with local, municipal, and county government agencies. Field relationships are strongest between UDOT and UHP. Center to center relationships are strongest between UDOT and the DPS Communications Bureau. A goal of the project is to expand these close-working relationships to the other centers and field personnel of the participating agencies.

The high-level goal established for the FOT reflect this logical progression:

• To demonstrate that a ATMS common message set, structured in a uniform and open format, can be implemented so that each agency receives only useful and relevant information.
• To develop this system in a way that enables CAD vendors to integrate the message set without affecting their propriety products.
• To link a wide variety of emergency response agencies at the local, county, and municipal government levels with the Statewide TMC.[9]

The State also adopted the high-level goals and objectives established by FHWA for the FOT – automating the seamless exchange of data, using the appropriate ITS standards, and integrating local, municipal and county level emergency responders.

During the course of the June 25, 2003 evaluation strategy briefing, the Evaluation Team proposed a series of questions to Utah designed to obtain additional insight into the State’s view of more specific goals, objectives, and impacts that will be realized through the FOT. These questions, and the State’s responses, are presented in summary form in Table 3-1.

Table 3-1. Utah-Expected FOT Benefits

3.4     Evaluation Goals and Objectives

The evaluation goals and objectives established for the Utah CAD-TMC FOT are presented in Table 3-2.

Table 3-2. Evaluation Goals and Objectives

3.5     Combined Evaluation Goals and Objectives

Table 3-3 presents a comparison of the goals and objectives established by FHWA, Utah, and the Evaluation Team. The intent of this comparison is to demonstrate how evaluation activities will track directly to State project activities, while also conducting the assessment requirements of FHWA’s National Evaluation Program. As can be seen, the goals and objectives developed for the Evaluation Plan are derived from both the FHWA and State goals and objectives, which in turn, help to ensure that the evaluation correctly reflects stakeholder interests.

FHWA	Utah	Evaluation Plan
The FOT will automate the seamless transfer of information between traffic management workstations and police, fire and EMS CAD systems from different vendors.	Enable the automated exchange of data between multiple agencies.	Document System Component Performance
The FOT will incorporate ITS standards such as IEEE 1512 and NTCIP into the integration of public safety and transportation information systems. Other standards areas that will have to be addressed are those pertaining to GIS.	The State has committed to using ITS standards and develop a system that conforms with the National ITS Architecture.	Document System Component Performance
The FOT will extend the level of integration to include secondary responders such as utilities, towing and recovery, public works and highway maintenance personnel.	Integrate local, county, and municipal government emergency management and response agencies (fire and rescue, law enforcement).	Document System Component Performance
System Impact: ·    FOT enhances communications among responders. ·    FOT enhances efficiency in documenting incidents. ·    FOT enhances on-scene operations. ·    FOT reduces incident clearance times. ·    FOT improves information available to traveling public and media.	Expected project impacts include: ·    Improved visual confirmation provides better information to speed response and to improve effectiveness of response – getting the right people and assets deployed on scene. ·    Reduce exposure of response personnel and reduce secondary collisions resulting from the initial incident. ·    Improve the quality of information provided to the media and traveling public. ·    Integration of UTA CAD.	System Impact Study:
Assess Institutional Challenges and Technical Issues	Improved inter-agency relationships, in particular understanding of each agency’s role and duties. Enable the exchange of data between emergency responders at all levels of government.	Identify institutional and technical challenges, and document how they were resolved.
Document Lessons Learned.	Document lessons learned.	Lessons Learned Summary.
Summarize Benefits.	Summarize benefits.	Benefits Summary.

4.0    Evaluation Approach

4.1    System Component Performance Study

The System Component Performance Study will address two objectives of the CAD-TMC evaluation: (1) examine system component performance and (2) discuss how well the project meets the FOT objectives. These overall objectives can be met by completing the following activities:

• Describe the environment in which the FOT will operate that could affect the applicability of the CAD-TMC concept to other sites and the interpretation of the system impacts data. This will help other potential deployers better understand the applicability of the CAD-TMC concept to their site.
• Identify key performance measures that should be met by similar deployments to achieve the system impacts observed by the FOT deployment. This will help other deployments identify and focus on the performance goals needed to achieve similar results. Also, document the design basis for these performance measures to help other deployments adjust these measures to better suit their local conditions.
• Calculate and document the key performance measures for the system as it was deployed. This will help identify limitations in the deployed system that might affect the observed system impacts. Also, identify and document other performance measures that are gathered by the deployment team (e.g., during component and integration testing). While this data is not as critical to the evaluation as the key measures, the data should be available from the deployment team to reduce the cost associated with reporting the data.
• Identify other factors that affect the performance of the deployed system. After the system is deployed, users may identify other factors that could make the system more useful and knowledge that could benefit others in developing similar systems.

In addition to these activities related to evaluating the performance of the deployed system, the Evaluation Team will also:

• Evaluate the degree to which ITS standards such as IEEE 1512 and NTCIP were incorporated into deployed system.
• Address the approach used to share data between map databases from different vendors and GIS standards that were applied.

The plan for each of these activities is described in the following Sections 4.1.1 through 4.1.6.

4.1.1    Describing the FOT and Its Operating Environment

There are several reasons why it is important to document the FOT deployment and its operating environment. First, the national Evaluation Team cannot design the evaluation without a good understanding of the deployment, and second, those interested in the evaluation results cannot interpret them without understanding the deployment. Also, the deployment will change over time, and these changes not only have the potential to impact the evaluation (e.g., by altering the deployment schedule), but also are important evaluation results in and of themselves. Deployment changes often occur to accommodate important new information that was learned in the deployment process, so documenting changes to the planned deployment can be a key to identifying lessons learned.

This description is particularly important for the CAD-TMC evaluation because considerable cooperation already exists between UDOT, UDPS, and other agencies involved in responding to incidents. This high level of existing cooperation will limit the impact of the FOT on incident response rates in much of the State. Some regions with less existing levels of cooperation might expect greater system impacts than those observed in the State as a whole. Understanding that the State began with a high level of cooperation is important so that readers of the evaluation results do not misinterpret the expected low impact on response rates as general indicators that increasing cooperation and integration does not improve these rates.

For these reasons, the national Evaluation Team will maintain a version-controlled description of the UDOT CAD-TMC Integration FOT. As changes to the deployment and deployment plans occur, this description will be updated. When changes occur to the FOT description, the impact of these changes on the evaluation activities will be considered, and if necessary, the Evaluation Plans will be revised accordingly. Also, the FOT changes will be reviewed to ascertain whether the changes should be listed as a lesson learned during the FOT, and if so, the change will be further investigated.

4.1.2        Identifying Key Performance Measures and the Design Basis

The system performance measures are another important tool for interpreting the evaluation results. If an installed system has weak performance measures, then a lack of system impacts might be attributed to an inadequate deployment. However, the performance measures evaluated for the FOT can serve several other useful purposes:

• The key performance measures, how the deployment team identified appropriate values for these measures, and how they were computed during testing can help other deployments select and calculate appropriate performance measures for similar deployments.
• Identifying performance measures can help other sites build acceptance criteria into their contracting language for similar deployments that are based on these performance measures.

For these reasons, the national Evaluation Team will work with the UDOT CAD-TMC deployment team to identify the key performance measures of the UDOT CAD-TMC system. In general, these performance measures will involve the type, timeliness, accuracy, and quality of the data exchanged between the systems operated by the various stakeholder groups: the UDOT CAD and ATMS; the UTA, VECC, SLCPD, and SLCFD CAD systems; UDOT field vehicles; UDPS field vehicles; and the UDPS MCC. Other performance measures will help identify the ease of operating the resulting system and the degree to which the system was used. The following list describes some of the key performance measures identified by the Evaluation Team associated with the:

• Types of incidents “broadcast” on the CAD-TMC FOT system and the type of information available about those incidents.
• Frequency with which information on events is shared.
• Lag time between incident verification and information availability to the TMC, to other CAD users, and to the public.
• Quality and accuracy of information exchanged.
• Delay times in responding to dispatch requests.
• Type of TMC information available on the CAD-TMC FOT
• Ease of access to this TMC information
• Degree of interoperability between the participating FOT stakeholders outside of the CAD-TMC FOT.
• Extent to which the deployed system was used by CAD and TMC operators and by secondary responders.
• Degree to which the system decreased reliance on manual methods for exchanging data.

The Evaluation Team will work with the FOT implementers to identify other measures and the expected values of these measures after the design of the UDOT CAD-TMC is complete. At the same time, the Evaluation Team will document the design basis for the expected values of the performance measures and will work with the deployment team to determine those measures that will be computed as part of the deployment validation effort and those that will be computed by the Evaluation Team. For both types of measures, the data collection and analysis plans will be documented in the evaluation test plan.

4.1.3     Documenting Performance Measures

The primary purpose in identifying the performance measures is to compute them after the deployment is complete to ensure that the deployed system performs as expected. The computed measures help determine whether the deployed system lives up to its design expectations. If a system is not performing as designed, it will not be clear whether the lack of system impacts is because of weaknesses in the deployment or weaknesses in the overall approach. It is important for the Evaluation Team to compute and analyze these performance measures to properly interpret the System Impact Study results.The final set of performance measures and the methods for evaluating them will not be finalized until the Detailed Test Plans are produced. The Evaluation Team has identified the objective, hypothesis, measures of effectiveness (MOE), data source, and analysis that might apply for each of the measures listed in the previous sections. These elements are summarized in the following Table 4-1.

Table 4-1. System Performance Measures

Objective	Hypothesis	MOE	Data Source	Analysis
Document the system component performance.	The system meets functional specifications.	Types of incidents broadcast and data available for those incidents	Interviews with deployment staff.	Review and description of interview results.
			Design documents.	Review and description of these documents.
			CAD message logs.	Sampling and summarization of messages broadcast.
			CARS message logs
		The lag time between incident verification by WSDOT/WSP and information availability to the general public and partner agencies.	CAD message logs.	Analysis of message log time stamps.
			CARS message logs
			Operator interviews.	Review and description of interview results.
		The quality and accuracy of information exchanged.	Operator interviews.	Review and description of interview results.
				.
		The type of TMC information available.	Interviews with deployment staff.	Review and description of interview results.
			Design documents.	Review and description of these documents.
			CAD message logs.	Sampling and summarization of messages broadcast.
			CARS message logs
		Ease of access to CAD and TMC information.	Interviews with CAD and TOC operators.	interview results.
	The CAD and TMC systems will be able to link data on an incident	Use of common standards enabling the linking of information between the different systems	Interviews with deployment staff.	Review and description of interview results.
			Design documents.	Review and description of these documents.
			CAD message logs.	Sampling and summarization of messages broadcast.
		Ability to obtain the same data on an incident from each system
			CARS message logs
	Using the system -improved incident response procedures.	Percentage of events where information is ahared between agencies	Interviews with CAD and TOC operators.	interview results.
			CAD message logs.
		Degree of interoperability achieved.	CARS message logs	Review and description of interview results.
		The extent to which the system was used.	Interviews with CAD and TOC operators.	interview results.
			Software and Web site usage statistics.	Analysis of usage statistics.
Automate the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors.	The FOTs will decrease the reliance on manual methods for exchanging information.	Percentage of time that initial exchange of information is generated automatically.	Interviews with CAD and TOC operators and secondary responders.	Review and description of interview results.
			Observations of CAD and TOC operator activities.	Review and description of observation results.
	The FOTs will increase the extent and reliability of information exchanges.	Information will be used to improve responses.	Case analyses of events.	Review and summarization of events.
			Interviews with operators/facility managers.	Review and description of interview results.
Extend the level of integration to include secondary responders such as utilities, towing and recovery, public works, and highway maintenance personnel.	Improved integration of secondary responders will reduce incident recovery time by getting required recovery personnel to the incident site as quickly as possible to begin recovery operations.	Identify secondary responders who are utilizing the system.	Interviews with deployment staff.	Review and description of interview results.
		Document information made available to responders and the extent to which it is used.	Interviews with secondary responders.	Review and description of interview results.

4.1.4        Identifying Other Factors Affecting System Performance

In many cases, some factors that affect system performance – and in particular, user acceptance of the system – are difficult to identify up front. For example, a certain class of users may want the data presented to be organized differently from other users, and may have difficulty expressing how best to organize the data until a system is in place. Identifying these factors can be important both for the FOT deployment, which can use this feedback to improve the deployed system, and for those deploying similar systems in the future, because they can build those systems to consider these additional factors. The Evaluation Team will conduct periodic post-deployment interviews with the users of the resulting system that include questions regarding suggested improvements. These suggested improvements will be relayed to the deployment team and included in the evaluation report.

4.1.5        Evaluating the Degree to which ITS Standards Were Incorporated

One goal of FHWA is to encourage the use of ITS standards in ITS deployments. This is important not only because the use of standards can facilitate deployment – those who developed the standards are experts in the field and using the standards leverages their expertise – but also because standards-based deployments are more easily ported to other locales. The Evaluation Team will take the following steps to evaluate the degree to which ITS standards were incorporated in this FOT deployment:

• Conduct a scan of existing standards (e.g., IEEE 1512, NTCIP) to determine which standards are ready for deployment.
• Scan all standards activities to ensure that the most current standards information is available, including standards validation and vendor compliance.
• Identify which standards the FOT teams selected and why they selected them.

4.1.6        Identifying the Approach Used to Share Geographic Data

One difficulty that is often encountered in sharing data between road-based systems is overcoming the incompatibilities in the underlying map data in the systems. These incompatibilities can be as simple to correct as different naming conventions for roads (e.g., “Rd” instead of “Road”, “1^st” instead of “First”), or can be as complex to correct as actual differences in the road topology (e.g., missing roads), differences in the road names, or differences in the road coordinates. Taken together, these incompatibilities can degrade the effective communication between systems.[10] Documenting how the UDOT FOT overcame these difficulties – and the extent to which these difficulties impeded the effectiveness of the system – can help sites that might employ similar techniques.

To achieve this goal, the Evaluation Team will interview the deployment team to document the approach used to share data between these systems. The Evaluation Team will also conduct performance tests on the effectiveness of these approaches by conducting round-trip exchanges of location information and performing statistical tests on the differences introduced by these exchanges. The Evaluation Team will supplement these analytical results on the effectiveness of the data-sharing approach with interviews with system operators to identify the frequency with which the location data was incorrect and the extent to which poor location data impeded incident response.

4.2     System Impact Assessment

This section outlines the approach to be taken in estimating the system impacts of CAD-TMC integration. System impacts will be evaluated using elements of the framework provided by FHWA’s National ITS Program Goal Areas: Mobility; Capacity/Throughput; Productivity; Safety; and Customer Satisfaction[11]. The evaluation will seek to quantify and document the benefits across these measurable areas for two very broadly defined beneficiary groups: incident responders and travelers. The benefits that each group realizes are different. The evaluation will seek to determine the Productivity and Safety Benefits for the response community. It will seek to determine the Mobility and Safety benefit made possible by improved traffic flow conditions, which will result in improved traffic flow conditions and increased Capacity/Throughput.

The evaluation premise for the first group, the incident response community, is that anticipated benefits of the integration are the result of improved interaction between members of the incident management and response communities at three levels: center-to-center; center-to-responder; and responder-to-responder. Within the response community, the stakeholders will be considered to primary responders or secondary responders. Primary responders are the DOT response crews that are dispatched to verify the incident and establish the follow-on response requirements.

Secondary responders are the other agencies dispatched to the scene based on the nature of the incident. This group can include police, fire and rescue, hazardous materials crews, and towing and recovery crews. The benefits of integrating the dispatch systems for these two communities will likely be the product of a quicker and more accurate understanding of the incident management requirements for the specific incident in progress, which will lead to more efficient execution of the incident management activities.

The evaluation premise for the second group, the travelers, is that improvements in incident management will be realized in terms of increased mobility (reduced delay). In addition, increased safety (reduced secondary crash rates) will be realized by improved link performance (reduced incident duration, improved incident specific traffic management plans, and improved diversion route availability and performance).

4.2.1    Baseline System Performance

While this evaluation will seek to identify the benefits of CAD-TMC Integration projects, it will also serve an important role in documenting the baseline performance of mature, high-performance incident management systems currently in use. Capturing the system description and performance qualities of the system in its current form will establish a benchmark performance level that will be useful as other states and localities upgrade and modernize their incident management capabilities. In order to conduct this baselining activity, the incident management system must be documented and the performance of each of the components must be measured to determine expected performance levels as well as the variation in those measurements. Working with the agencies involved, the Evaluation Team will draft descriptions of each applicable activity in its baseline form and document current performance.

The quantitative aspect of the baselining activity will require access to a broad range of databases to document not only the incident management process, but to establish relationships between incident management processes, traffic flow characteristics (on the freeway and on the diversion routes), and secondary crash occurrence. The databases and information required are those that will support answering the following questions:

• What happened? What was the impact of the event on the highway? Where was it? What time did it occur? What were the weather conditions?
• What DMS messages were in place prior to the event and what was the message history after the event (i.e., warnings to drivers, variable speed limit responses, diversion instructions, etc.)?
• How long did the event clearance process last?
• How did the highway section perform over the timeframe (including a period before and after)?
• How was diversion route performance affected?
• Were there any secondary effects?

To support this effort, an initial screening process will be used to identify “hot spots”. Hot spots are those freeway locations that indicate an incident history that will prove useful for evaluation and which can support the data requirements outlined in the list of investigative questions. Data requirements to identify the hot spots are incident histories that identify location, direction of travel, incident type, time-of-day, day-of week, date, and weather conditions. A data search will be used to identify the high-incident locations and high secondary-incident locations (defined as secondary incidents occurring within 2 miles up or down stream of either direction). Once hot spots are identified, the following data can be pulled to develop a complete picture of the incident management activity at a “system level”:

• The incident history.
• Variable Speed Limit sign (VSL) history, if applicable.
• Variable Message Sign (VMS) history, if applicable.
• Highway performance indicators (volume, spot speeds, etc.).
• Highway shockwave attributes (length, time of accumulation, and time of dissipation).
• Diversion arterial performance indicators (volume, spot speeds, queue lengths at signals, etc.).
• Secondary incident-occurrence history, if any.

The evaluation objective will be to establish a performance baseline for defined incident classifications (type, number of lane closures, hazmat involvement, etc.) that includes an understanding of the relationships between these “system level” measures. Figure 4-2 provides an example of such an integrated picture, which can lead to a set of statistical process performance and control measures. The evaluation team recognizes that data availability may be problematic, and will target corridors where arterial data have been instrumented for data collection through the UDOT ATMS.

Figure 4-1. System Performance Measures

4.2.2     Documenting the Learning Curve

Documenting the “learning curve” associated with the CAD-TMC integration is an important part of the evaluation. The Evaluation Team will work closely with the stakeholders to identify the “milestone” events that will take place over the life of the deployment and evaluation.

Once the baseline system performance is documented qualitatively and quantitatively, the Evaluation Team, with the stakeholders, will identify key technology insertions and monitor the emergence of adjustments in operational concepts. The adjusted operational concepts will indicate opportunities for conducting follow-up interviews for revising the qualitative aspects of the baseline and produce a milestone-based documentation of the impact on process and procedures that may produce new efficiencies in communications, response, and overall incident management.

To capture the quantitatively measured aspects of the learning curve, the Evaluation Team will establish a data collection concept that will make periodic data pulls fixed around known technology insertions and stakeholder-identified changes in concepts of operation. This effort will capitalize on the improvements in automatic reporting that are anticipated as part of the integration effort. The Evaluation Team realizes that the range and depth of data available at the beginning of the effort may be less than that available at the end of the effort. This enrichment of data availability, particularly in relational format, will be a key aspect of the evaluation leading to improved ability to monitor and measure incident management system. Data collection to support evaluation will take into account the need to identify “burn-in” times associated with new technologies and operational concepts.

4.2.3    Systems Impact Evaluation

To evaluate the benefits of the CAD-TMC integration, the Evaluation Team developed a set of objectives and hypotheses to guide the identification of MOEs, data requirements, and analysis methods. Tables 4-2 through 4-5 presents the experimental design for evaluation of the system impact for each of the four National ITS Goal Areas.

Table 4-2. System Impact Experimental Design for Productivity

Objective	Hypothesis	Measure	Data Sources	Analysis Method
To determine if the CAD-TMC integration improves the efficiency and productivity of incident response.[12]	CAD-TMC integration enhances communications among responders.	Develop a process flow map of communications network used for specific incident classifications identifying all modes/all communications by type (voice or data and mode [wire or wireless).	Communication logs and a survey.	Quantitative/ qualitative survey analysis. Before/after comparison of communications systems.
	CAD-TMC integration improves efficiency of on-scene operations.	Determine total on-scene time required by incident classification from first arrival to last departure. Assess impact of CAD-TMC on reducing duration, shorter time, quicker response, etc. Compare baseline and after data.	Incident management logs to determine the on-scene time for each incident classification.	Descriptive statistical analysis.
	CAD-TMC integration reduces incident clearance times.	Determine total time from incident detection until incident clearance for each incident classification. Compare baseline data with after data.	Incident management logs, radio and communication logs.	Descriptive statistical analysis.
	CAD-TMC integration enhances efficiency in documenting incident management.	Determine number of incidents for which TMC traffic management logs, incident response dispatch logs, and highway performance monitoring system data are correctly merged in near-real time Determine ability of information management system to correctly archive incident management data in relational databases to support incident debriefs, statistical process control methods, and management level review.	Incident management records and surveys (designed to provide qualitative and quantitative data) of IM personnel from on-scene personnel to senior management within the major stakeholder groups, i.e., DOT, DPS, Highway Patrol, and Transit Operators.	Quantitative/ qualitative survey analysis. Before/after comparison of incident management logs.

Table 4-3. System Impact Experimental Design for Mobility

Objective	Hypothesis	Measure	Data Sources	Analysis Method
To determine if the CAD-TMC integration improves mobility and reduces delays during incidents.	CAD-TMC integration enhances mobility during incident management (IM) activities.	Determine speed profiles to determine duration/length of traffic characteristics (i.e., congestion and speed) in response to various incident classifications. Compare baseline and after data.	For high crash frequency freeway sections: average speeds for the location.	Descriptive statistical analysis and comparison of the no-incident case, the baseline w/incident case, and the after w/incident case.

Table 4-4. System Impact Experimental Design for Capacity/Throughput

Objective	Hypothesis	Measure	Data Sources	Analysis Method
To determine if CAD-TMC integration enhanced incident-specific traffic management plans	CAD-TMC integration enhances incident-specific traffic manage-ment plans.	Determine the diversion effect on traffic volumes over the affected link for specific incident classification. Compare baseline and after data.	For high crash frequency freeway sections: measure volume during incidents of each particular classification; measure the volume diverted to the arterial; measure the impact on arterial performance.	Descriptive statistical analysis of key measures and comparison of baseline and after cases.

Objective	Hypothesis	Measure	Data Sources	Analysis Method
CAD-TMC integration will reduce exposure of response personnel and secondary crashes during incident response activities.	CAD-TMC increases safety for response personnel.	Determine reduction in exposure time for response personnel from first arrival to last to leave. Determine traffic volume and speed at incident location and key diversion points to determine effects of TMC- provided traveler information on driver diversion decisions.	For exposure time: incident management logs to determine change in duration of on-scene operations for specifically defined incident classifications. Sources for incident location and key diversion point volumes include highway performance monitoring system data.	Descriptive statistical analysis of key measures and comparison of baseline and after cases.
	CAD-TMC increases safety the traveling public.	Determine local relationship between incident duration and occurrence of secondary crashes using the method developed in the Maryland CHART secondary crash study.[13]	Jurisdiction-identified high crash frequency freeway segments, and records for all crashes (same and opposite direction within 2 miles and 2 hours) to identify secondary crash patterns.	Descriptive statistical analysis of key measures and comparison of baseline and after cases.

4.2.4     Deployment Specific Evaluation Components

The UDOT FOT includes several unique aspects that require special attention during the evaluations. UDOT intends to include an automated interface to its 511 and Internet-based traveler information services to improve the quantity, quality, and timeliness of incident information provided to the traveling public. The UTA will participate in the Utah FOT as one of the agencies sharing information about events and incidents being managed by their staffs. UTA has three radio control centers: one each for light rail; fixed-route transit; and paratransit operations. Staff at these centers will access information managed by any of the other CommuterLink partners and will broadcast UTA-specific information to these same partners.The Evaluation Team developed the following hypotheses related to the integration of the 511 and Internet-based traveler information services, as presented in Table 4-6.

Table 4-6. Traveler Information Assessment

Objective	Hypothesis	MOE	Data Source	Analysis
To determine if CAD-TMC integration will improve incident management information available to travelers.	CAD-TMC integration enhances customer satisfaction and mobility during incident management activities by improving traveler information.	Determine change in the percent of eligible incidents reported on traveler information Website. Determine change in time between when the incident occurred and when information became available to the public. Determine if the number of Website hits and 511 calls increased. Determine if media were able to use information.	Sources include: Utah State Patrol and UDOT incident logs and Website logs; 511 call logs; Interviews with media.	Descriptive analysis of key measures and comparison of baseline and after cases.
		Assess satisfaction of the traveling public with improved traveler information.	Web-based survey of traveling public.	Quantitative/ qualitative survey analysis.

Currently, the safety and mobility of UTA passengers, personnel, and equipment is affected by the limitations of the real-time information available to UTA about ongoing incident management activities that affect its operations. UTA may not be informed quickly about a new incident that affects its operations, and thus, some UTA vehicles may be delayed. Depending upon the nature of the incident, the safety of passengers, personnel, and equipment may also be affected. UTA may also not be informed quickly once an incident is cleared. If UTA has rerouted vehicles around the incident, the reroute may continue for some time after normal operations could have been restored, thus, unnecessarily increasing the mobility impact.

This plan will assess the hypothesis that UTA reroute decisions will be more effective when based on immediate access to current information on the status of incident response activities being managed by the TMC, public safety or law enforcement.
The plan will use two different MOEs for this hypothesis. The first MOE will measure the delay in starting rerouting after incident response first began. The second MOE will measure the delay in ending rerouting after the incident was cleared. The expectation is that making available to UTA dispatchers the information on the current response status for all incidents reported by any participating agency will serve to reduce these delays, based on a comparison of incident samples from before and after the FOT.

One primary source of data for the test will be the logs maintained by UTA dispatch about the start and end times for fixed route bus reroutings in response to unexpected traffic or other incidents. These logs indicate the date/time as well as the route involved and the routing change (put into effect or ended). In addition, these logs sometimes indicate a cause for the rerouting (e.g., traffic accident).

Another primary source of data will be the TMC incident logs, indicating the date/time when an incident was first detected, and then logging subsequent incident response activities through to when the incident is logged as cleared to the point where it is no longer impacting traffic flow.

The basic data analysis methodology will select from the TMC logs a sample of incidents that occurred along bus routes during UTA operating hours. The sample will be from incidents for which the start and end times were recorded, and which appear significant enough in severity and duration that UTA might have made a reroute decision. UTA dispatch logs then will be reviewed for these time periods to identify the start and end times for any associated reroutings. By combining this information, the Evaluation Team will develop samples of the two MOEs for a sample set of incidents. This analysis will be undertaken first for a “before” period prior to the FOT implementation, and then again later for an “after” period once the FOT is in stable operation.

Table 4-7. UTA CAD Integration Assessment

Objective	Hypothesis	Measure	Data Sources	Analysis
To determine if the integration of the UTA CAD system improves UTA’s ability to respond to incidents.	The CAD-TMC integration will enable UTA to more effectively implement reroute decisions in response to an incident.	Changes in time needed to implement rerouting following an incident. Changes in time needed to end rerouting once an incident has been cleared.	Sources include: UTA CAD system and UTA logs; TMC incident logs.	Descriptive statistical analysis of key measures and compar-ison of baseline and after cases.

4.3    Institutional Challenges Evaluation Approach

The institutional challenges will be identified and documented through the following efforts:

• Stakeholder Interviews. Interviews will project stakeholders will provide the primary information source for identifying challenges and the processes by which they were resolved. These interviews will be conducted on a “before” and “after” basis.
• Document Review. Interviews will be supplemented by the review of documents (meeting minutes, correspondence, project reports) generated through project activities. Document review, in particular meetings minutes, will be used to document the processes by which the institutional challenges were resolved.

Issues to be assessed through the institutional challenges study include:

• Documenting inter-agency cooperation at the State level, in particular, the processes used for identifying and solving problems.
• Assessing how county and municipal agencies are integrated into the program (VECC, Salt Lake City).
• Identifying what information is shared, and how the agencies determined that this was the right information to share.
• Documenting how USP and UDOT determined what the information availability would be for exchanges between the CAD-TMC systems.
• Documenting how frequently the information provided through the project is used by:
  • Responders
  • Travelers
  • Media
• Documenting how these end-users used the information provided, and identifying how the information was used.
• Determining if end-users found the information useful and why or why not.
• Assessing how the various CAD vendors were able to establish working relationships and share data.

4.4    Technical Issues Evaluation Approach

This assessment will document how the FOT teams addressed technical challenges such as overcoming the barriers associated with incompatible and/or proprietary systems. In conducting the study, the Evaluation Team will review copies of technical documentation (e.g., concept of operations, requirements, and design documents) produced by the FOTs to identify challenges that they have referenced. Follow-up interviews with technical staff at each participating group will be used to review the specific challenges addressed in these documents; identify additional challenges that may have occurred; and evaluate how those challenges were resolved. The Evaluation Team will also become integrated with the FOT activities (e.g., by participating in FOT team meetings, joining email exchanges) in order to identify technical challenges as they occur.

Issues to be addressed include:

• Problems encountered with developing message sets
• Problems encountered with building LSIs
• Establishing interoperability between incompatible and/or proprietary systems

4.5     Lessons Learned Assessment

This assessment will meet FHWA’s objective to summarize lessons learned during the other portions of this evaluation. Lessons learned will be gleaned from all aspects of the evaluation. The Evaluation Team will also explicitly request information on lessons learned during all interviews associated with this evaluation. This process will be ongoing throughout the project and information will be shared on a regular basis with the project partners and with FHWA, such that the evaluation may serve to actually improve the deployments.

4.6     Benefits Summary

The Benefits Summary will address FHWA’s objective to document benefits pertaining to enhanced field operations associated with locating and responding to incidents; enhanced communications among responders; enhanced on-scene activities; and enhanced efficiency in documenting the incidents. The benefits from the FOT deployments will be derived from the Systems Impact Study, which measures quantitative impacts of the FOTs on system characteristics, (i.e., the time operators spend on notification activities, quicker response times, and congestion delays caused by an incident). The Benefits Summary will ensure that data collected to evaluate these system impacts is supplemented with other data necessary to estimate primary and secondary benefits of these system impacts. The Evaluation Team will plan the data collection to support a variety of benefits estimates, including decreased operator time, decreased dispatching errors, quicker injury treatment, decreased traffic delays, and reduction in secondary incidents and injuries.

5.0     Detailed Test Plans – Outline and Level of Effort

All test plans to be developed for the CAD-TMC evaluation shall be prepared as specified in the solicitation:

The contractor shall develop detailed Test Plans for both the UDOT and WSDOT test sites. The plans shall describe impacts of the FOT with a focus on comparison of impacts before and after the system implementation. Each test plan shall address, at a minimum, the following:

• Test objectives and approaches,
• Data collection methods,
• Methods for evaluating implementation of applicable standards,
• Test schedule,
• Pre-test activities,
• Test Activities,
• Post-test activities,
• Data requirements,
• Data analysis,
• Report format and expected contents; and
• Estimated resources required to complete all activities described in the individual test plans.[14]

This section of the Evaluation Plan presents a high-level overview of the elements that will be included in the test plans for each evaluation goal and study area.

5.1     High-Level Outlines

5.1.1     System Performance Assessment

The objectives of the System Performance Assessment are to determine if:

• The performance of the system components met functional specifications and requirements.
• Automation enabled the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors.
• Secondary responders, including both the private sector and municipal/county government, were successfully integrated.

An additional component of the System Performance Assessment will be to:

• Assess the degree to which ITS standards such as IEEE 1512 and NTCIP were incorporated into the system.
• Asses if the system enables the sharing of data between map databases from various vendors and GIS standards that were applied.

The hypotheses is that each of the these objectives is true:

• The system did meet functional performance specifications and did enable the seamless transfer of information.
• Secondary responders were successfully integrated into the system.
• ITS standards and the sharing of map databases using GIS were successfully incorporated into the system.

The measures of effectiveness that will be used to test the hypotheses include:

• Documenting actual system performance and comparing this to functional specifications to see if these were realized, and if not, identifying why.
• Assessing the degree of interoperability obtained – was information successfully exchanged between different vendor CAD systems and agency legacy systems?
• Documenting the extent to which existing manual information exchange systems were replaced by automated information exchange.
• Documenting the timeliness, quality, and accuracy of information exchanged.
• Determining the extent to which secondary responders are successfully integrated into the system, and the extent to which they use the system.

The Evaluation Team will also review which standards are used and why, and how these standards were incorporated into the system.

Data collection will be done through reviewing system performance logs; CAD message logs; developing “before” and “after” communication process flows to identify the impact of automation; the review of usage statistics; direct observation of dispatchers and operators using the CAD and TMC systems; and interviews with end-users and secondary responders.

5.1.2     System Impact Study

The objective of the System Impact Study will be to determine if the integration of CAD and TMC systems:

• Improves mobility and reduces incident-caused delays.
• Improves the efficiency and management of incident response activities.
• Reduces the exposure time of response personnel at roadside, thus reducing the risk of injury due to a secondary incident.
• Reduces secondary crashes related to incidents.

The hypotheses are that each of these facets is true. The measures of effectiveness that will be used to test these hypotheses include:

• Comparing the “before” and “after” data for total on-scene time required by incident classification.
• Comparing the “before” and “after” data from detection through clearance by incident classification.
• Comparing the “before” and “after” data for speed profiles and the diversion of traffic volumes during incident responses by incident classification.
• Comparing reductions in incident response personnel exposure time and compare existing research to estimate improved safety.
• Determine the “before” and “after” local relationship between incident duration and occurrence of secondary crashes using existing research to determine the extent of incremental impact.

The Evaluation Team recognizes that in conducting this test:

• Additional factors that may impact before and after conditions need to be considered so that the actual impact of the project is correctly measured.
• Significant reductions in mobility, efficiency