Computer-Aided Dispatch - Traffic Management Center Field Operational Test Final Detailed Test Plan: WSDOT Deployment

Contract #: DTFH61-02-C-00061
ITS Program Assessment Support Contract
Task SA61004

Submitted to:

United States Department of Transportation
Mr. David Helman
Dr. Joseph Peters

Submitted by:

Science Applications International Corporation
8031 Greensboro Drive
M/S E-7-6
McLean, VA 22102

October 2003

QUALITY ASSURANCE STATEMENT

The Federal Highway Administration provides high-quality information. To serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement.

NOTICE This report is in draft format. Comments on this
report should be provided to SAIC via email, fax, or mail, addressed to:

Nicholas D. Owens
Science Applications International Corporation
1710 SAIC Drive
McLean, VA 22102
Phone: 703-676-2408
Fax: 703-676-2432
owensni@saic.com

LIST OF CONTENTS

1.0 INTRODUCTION
2.0 EXPERIMENTAL DESIGN
2.1 Introduction
2.2 Experimental Design: 2.2.1 Describing the FOT Environment; 2.2.2 Identifying Key Performance Measures; 2.2.3 Experimental Design
3.0 Data Collection Plan
4.0 Detailed Test Plans
4.1 System Performance: 4.1.1 Integration of Secondary Responders
Test Schedule and Activities: 4.1.2 Standards Test Plan; 4.1.3 GIS Standards Test Plan
4.2 System Impact: 4.2.1 Test Objectives; 4.2.2 511/Internet Interface Test
4.3 Institutional Challenges
4.4 Technical Challenges
4.5 Lessons Learned Summary
4.6 Benefits Summary
5.0 Estimated Resource Requirements and Test Management
6.0 Schedule

LIST OF FIGURES

Figure 4-1. Key Data Points in Critical Incident Management Data Fields with Respect to the Time Period Immediately Before to Complete Recovery

LIST OF TABLES

Table 2-1. System Performance Experimental Design

Table 2-2. System Impact Experimental Design for Productivity

Table 2-3. System Impact Experimental Design for Mobility

Table 2-4. System Impact Experimental Design for Safety

Table 2-5. System Impact Experimental Design for Capacity/Throughput

Table 5-1. Test Component and Responsible Team Member Assignment

Table 5-2. Estimated Resource Requirements

Table 6-1. Evaluation Activities, Schedule, and Status

1.0 INTRODUCTION

The Statement of Work developed for IPAS Task 61004: Computer-Aided Dispatch (CAD) –Traffic Management Center (TMC) Field Operational Test (FOT) requires that Detailed Test Plans be developed to provide specific scopes of work for each part of the evaluation. This Detailed Test Plan document is submitted in response to this requirement.

Science Applications International Corporation (SAIC) was selected as the Independent Evaluator for this effort. This Detailed Test Plan document contains more detailed information than is presented in the Evaluation Plan document concerning the specific technical and data collection approaches being implemented for this evaluation. The Evaluation Plan document contains significant project background and system description information, as well as a discussion of the management approaches being utilized by SAIC in this effort, and should be reviewed alongside this Test Plans document.

An overview of the operational test and a review of the technical approach for this evaluation can be reviewed in the following reference:

Computer-Aided Dispatch - Traffic Management Center Field Operational Test Final Evaluation Plan: Washington State Department of Transportation Deployment, published September 22, 2003 for the Federal Highway Administration (FHWA) by SAIC in association with PB Farradyne and TranSystems.

The purpose of this document is to expand upon the evaluation components presented in the Final Evaluation Plan. This document defines the objective, approach, and work steps for each evaluation component, and describes the planned schedule for the completion of the evaluation. These evaluation components include:

System Performance:
- Document the system component performance.
- Automate the seamless transfer of information between traffic management workstations and police, fire, and emergency medical systems (EMS) CAD systems from different vendors.
- Extend the level of integration to include secondary responders such as utilities; towing and recovery; public works; and highway maintenance personnel.
- Evaluate the degree to which ITS standards, such as IEEE 1512 and NTCIP, were incorporated into deployed system.
- Address the approach used to share data between map databases from different vendors and GIS standards that were applied.
System Impact:
- To determine if the CAD-TMC integration improves the efficiency and productivity of incident response.
- To determine if the CAD-TMC integration improves mobility and reduces delays during incidents.
- To determine if CAD-TMC integration enhances incident-specific traffic management plans.
- To determine if the CAD-TMC integration will reduce exposure of response personnel and secondary crashes during incident response activities
- To determine if CAD-TMC integration will improve incident management information available to travelers.
Institutional Challenges
Technical Issues
Lessons Learned
Benefits Summary

The succeeding portions of this document are organized as follows:

Section 2 – Experimental Design. This section describes how the evaluation team will conduct the assessment and define how the FOT met the stated objectives using an experimental design to facilitate the evaluation.
Section 3 – Data Collection Plan. This section provides the process by which the Evaluation Team will collect and store data over the course of the CAD-TMC FOT evaluation…
Section 4 – Detailed Test Plans. This section provides the detailed plans developed by the evaluation team for conducting each of the individual tests.
Section 5 – Estimated Resource Requirements and Test Management. This section presents the estimated resource requirements needed for completing the proposed tests.
Section 6 – Schedule. This section provides the overall evaluation activities, schedule, and status.

2.0 EXPERIMENTAL DESIGN

2.1 Introduction

The Federal Highway Administration has established the following high-level objectives for the CAD-TMC Integration FOT:

To automate the seamless transfer of information between traffic management workstations and police, fire and EMS CAD systems from different vendors. Integration between CAD systems and TMC information systems, where it has occurred, has been somewhat limited, and has been implemented with workstations and software provided by the same vendor.
To incorporate ITS standards (such as IEEE 1512 and NTCIP) into the integration of public safety and transportation information systems. Other standards areas that need to be addressed are those pertaining to Geographic Information Systems (GIS). When automated vehicle location (AVL) is incorporated into emergency response CAD systems and traffic management systems, compatibility issues must be resolved when attempting to share data between map databases from different vendors. These issues must be solved for emergency response and transportation agencies in order to display the location and coordinate dispatching of both sets of vehicles.
To extend the level of integration to include secondary responders such as utilities, towing and recovery, public works, and highway maintenance personnel.[1]

The Evaluation Plan and the Detailed Test Plans developed by the Evaluation Team are designed to conduct the assessment and define how the FOT met these stated objectives. This section of the Detailed Test Plan document presents the experimental design developed by the Evaluation Team to facilitate this assessment.

This evaluation is being structured around a concept that involves documenting a baseline process and providing follow-up documentation of the improvements over time. Baseline data will be taken at a point in time close to “system turn-on”. Information and data gathered at this point will serve as the basis for comparison with multiple future measurement points taken at key opportunities following technology insertion or incident management team training.

Ideally, the Evaluation Team, with the stakeholders, will identify these key opportunities and monitor emergent adjustments in operational concepts as shown in Figure 2-1. The adjusted operational concepts will indicate opportunities for conducting follow-up interviews. These follow-up interviews will be used to revise the qualitative aspects of the baseline and produce a milestone-based documentation regarding the impact on process and procedures. The impact elements may then produce new efficiencies in communications, response, and overall incident management.

To capture the quantitatively measured aspects of the learning curve, the Evaluation Team will establish a data collection concept and make periodic data pulls fixed around known technology insertions and stakeholder-identified changes in concepts of operation. This effort will capitalize on the improvements in automatic reporting that are anticipated as part of the integration effort. Data collection to support evaluation will take into account the need to identify “burn-in” times associated with new technologies and operational concepts.

2.2 Experimental Design

The following activities will provide the basis of the experimental design for this evaluation:

Describe the environment in which the FOT will operate that could affect applying the CAD-TMC concept to other sites and interpreting the system impacts data. This description will help other potential deployers better understand the applicability of the CAD-TMC concept to their site.
Identify key performance measures that should be met by similar deployments to achieve the system impacts observed by the FOT deployment. These performance measures will help other deployments identify and focus on the performance goals needed to achieve similar results.
Document the design basis for the identified performance measures to help other deployments adjust these measures to better suit their local conditions.
Calculate and document the key performance measures for the system as it was deployed. Documenting the key performance measures will help identify limitations in the deployed system that might affect the observed system impacts.
Identify and document other non-key performance measures gathered by the Deployment Team (e.g., during component and integration testing). While this data is not as critical to the evaluation as the key measures, the data should be available from the Deployment Team to reduce the cost associated with reporting the data.
Identify other factors that affect the performance of the deployed system. After the system is deployed, users may identify other factors that could make the system more useful and knowledge that could benefit others in developing similar systems.

2.2.1 Describing the FOT Environment

The first step in examining system component performance is defining the system. The Evaluation Team will maintain a description of the WSDOT CAD-TMC Integration FOT, and update this document whenever significant changes occur in the plans for the FOT or during the deployed FOT. The only data requirements are to document information about changes to the FOT deployment and deployment plans. Most changes to the FOT deployment will be noted during other evaluation activities and relayed to the Evaluation Team member responsible for maintaining the FOT description. In addition, the Evaluation Team will periodically review deployment activities with the FOT Deployment Team in order to identify changes that might not have been noted.

The following activities will occur throughout the evaluation period to ensure and maintain the accuracy of the system description. When a change to the FOT is noted, the FOT description will be updated, and the impact of the change on the evaluation will be identified. If any impacts are likely, the following steps will be taken:

Task managers responsible for tasks that might be affected by the change are notified to ensure that they are aware of the change.
If the change is likely to affect the evaluation schedule, budget, or deliverables, FHWA will be notified of the change and the potential effect the change may have.

Each change will also be reviewed to determine if the knowledge of or reason for the change might benefit other states interested in similar systems. If so, the Evaluation Team will gather additional information about the change and document that information as a “lesson learned”.

2.2.2 Identifying Key Performance Measures

Identifying key system performance measures is important for two reasons:

To define a framework for interpreting the evaluation results, in that a similar system with different performance measures might produce different results.
To document how the appropriate values for those measures were identified, and how they were computed during testing can help other, similar deployments select and calculate appropriate performance measures. For example, a deployment might use these performance measures to establish acceptance criteria.

Three types of data are required for this part of the evaluation: the key system performance measures; the design basis for those measures and their computation;, and the observed values of those measures for the deployment. The Evaluation Team, in working with FHWA and the State of Washington, has identified a number of key performance measures, as presented in the Evaluation Plan. The Evaluation Team selected those performance measures for which there is a high degree of certainty that qualitative and quantitative data are available to measure the before and after system performance.

The following process was used to identify key performance measures:

The first data collected was a list of performance measures that were considered as key, and established the rationale for this classification. These performance measures were identified through the review of project documents and interviews with project staff.
For each key performance measure, the Evaluation Team documented the desired goal/value for that measure, and the basis for identifying that goal/value.
After the system is deployed, the Evaluation Team will document the actual values of the key performance measure. For those measures that fail to meet their expected values, a description will be included regarding the impact this might have on the deployment. (This topic is more fully described in Section 2.2.3.)
At the end of the FOT, the Evaluation Team will review the list of key performance measures with the Deployment Team to identify any changes in perspective as to which performance measures should be considered key, and any lessons learned regarding the performance measures.

2.2.3 Experimental Design

Using the process described in the preceding Sections 2.2.1 and 2.2.2, the Evaluation Team developed an experimental design for the evaluation, as summarized in Tables 2-1 through 2-5. For each table, the evaluation objective, hypothesis, measures of effectiveness (MOE), data source, and analysis will be described. The experimental design is developed to focus on the following areas:

System Performance
System Impact for Productivity
System Impact for Mobility
System Impact for Safety
System Impact for Capacity/Throughput

Table 2-1. System Performance Experimental Design

Objective	Hypothesis	MOE	Data Source	Analysis
Document the system component performance.	The system meets functional specifications.	Types of incidents broadcast and data available for those incidents.	Interviews with deployment staff.	Review and description of interview results.
			Design documents.	Review and description of these documents.
			CAD message logs.	Sampling and summarization of messages broadcast.
			CARS message logs	Sampling and summarization of messages broadcast.
		The lag time between incident verification by WSDOT/Washington State Police (WSP) and information availability to the general public and partner agencies.	CAD message logs.	Analysis of message log time stamps.
			CARS message logs	Analysis of message log time stamps.
			Operator interviews.	Review and description of interview results.
		The quality and accuracy of information exchanged.	Operator interviews.	Review and description of interview results.
		The type of TMC information available.	Interviews with deployment staff.	Review and description of interview results.
			Design documents.	Review and description of these documents.
			CAD message logs.	Sampling and summarization of messages broadcast.
			CARS message logs	Sampling and summarization of messages broadcast.
		Ease of access to CAD and TMC information.	Interviews with CAD and TOC operators.	Interview results.
	The CAD and TMC systems will be able to link data on an incident	Use of common standards enabling the linking of information between the different systems.	Interviews with deployment staff.	Review and description of interview results.
			Design documents.	Review and description of these documents.
			CAD message logs.	Sampling and summarization of messages broadcast.
		Ability to obtain the same data on an incident from each system.	CAD message logs.	Sampling and summarization of messages broadcast.
			CARS message logs
	Using the system-improved incident response procedures.	Percentage of events where information is shared between agencies.	Interviews with CAD and TOC operators.	Interview results.
			CAD message logs.	Interview results.
		Degree of interoperability achieved.	CARS message logs.	Review and description of interview results.
		The extent to which the system was used.	Interviews with CAD and TOC operators.	Interview results.
		The extent to which the system was used.	Software and Website usage statistics.	Analysis of usage statistics.
Automate the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors.	The FOTs will decrease the reliance on manual methods for exchanging information.	Percentage of time that initial exchange of information is generated automatically.	Interviews with CAD and TOC operators and secondary responders.	Review and description of interview results.
			Observations of CAD and TOC operator activities.	Review and description of observation results.
	The FOTs will increase the extent and reliability of information exchanges.	Information will be used to improve responses.	Case analyses of events.	Review and summarization of events.
		Information will be used to improve responses.	Interviews with operators/facility managers.	Review and description of interview results.
Extend the level of integration to include secondary responders such as utilities, towing and recovery, public works, and highway maintenance personnel.	Improved integration of secondary responders will reduce incident recovery time by getting required recovery personnel to the incident site as quickly as possible to begin recovery operations.	Identify secondary responders who are utilizing the system. Document information made available to responders and the extent to which it is used.	Interviews with deployment staff. Interviews with secondary responders.	Review and description of interview results.

Table 2-2. System Impact Experimental Design for Productivity

Objective	Hypothesis	Measure	Data Sources	Analysis Method
Improve the efficiency and productivity of incident response.	CAD-TMC integration enhances communications among responders.	Develop a process flow map of communications network used for specific incident classifications identifying all modes/all communications by type (voice or data and mode [wire or wireless]).	Communication logs and a survey.	Quantitative/ qualitative survey analysis. Qualitative before/after comparison of communications systems.
	CAD-TMC integration reduces incident clearance times.	Determine total time from incident detection until incident clearance for each incident classification. Compare baseline data with after data.	Incident management logs, radio and communication logs.	Descriptive statistical analysis and conduct tests for statistical significance of differences between before and after data.
	CAD-TMC integration improves efficiency of on-scene operations.	Determine total on-scene time required by incident classification from first arrival to last departure. Assess impact of CAD-TMC on non-value-added on-scene time for the various responding agencies. Compare baseline and after data.	Incident management logs to determine the on-scene time for each incident classification.	Descriptive statistical analysis and conduct tests for statistical significance of differences between before and after data.
	CAD-TMC integration enhances efficiency in documenting incident management.	Determine ability of information management system to correctly archive incident management data in relational databases to support incident debriefs, statistical process control methods, and management level review.	Incident management records and surveys (designed to provide qualitative and quantitative data) of incident management personnel from on-scene personnel to senior management within the major stakeholder groups.	Quantitative/ qualitative survey analysis. Before/after comparison of incident management logs.

Table 2-3. System Impact Experimental Design for Mobility

Objective	Hypothesis	Measure	Data Sources	Analysis Method
Improve mobility and reduce incident-caused delays.	CAD-TMC integration improves mobility during incident management activities.	Determine speed and/or lane occupancy profiles to determine duration/length of traffic characteristics (i.e., congestion and speed) in response to various incident classifications. Compare baseline and after data.	For high crash frequency freeway sections: hourly volumes for the identified locations during incident and non-incident periods.	Descriptive statistical analysis and conduct tests for statistical significance of differences between incident and non-incident data.

Table 2-4. System Impact Experimental Design for Safety

Objective	Hypothesis	Measure	Data Sources	Analysis Method
Reduce exposure time of response personnel.	CAD-TMC increases safety for response personnel.	Determine the reduction in exposure (time on-scene) for personnel from each responding agency.	Incident management activity record analysis for on-scene time by agency.	Descriptive statistical analysis of key measures and conduct tests for statistical significance of differences between before and after data
Reduce secondary crashes related to incidents.	CAD-TMC increases safety the traveling public.	Determine incidence rate of secondary crashes and the local incident duration (mean and std dev) for specific incident classes for both the before and after cases.	Identified high crash frequency freeway segments through evaluation of records for all crashes (same and opposite direction within 2 miles and 2 hours) to identify secondary crash patterns.	Descriptive statistical analysis of key measures and conduct tests for statistical significance of differences between before and after data. Identify and document confounding factors.

Objective

Hypothesis

Measure

Data Sources

Analysis Method

Reduce exposure time of response personnel.

CAD-TMC increases safety for response personnel.

Determine the reduction in exposure (time on-scene) for personnel from each responding agency.

Incident management activity record analysis for on-scene time by agency.

Descriptive statistical analysis of key measures and conduct tests for statistical significance of differences between before and after data

Reduce secondary crashes related to incidents.

CAD-TMC increases safety the traveling public.

Determine incidence rate of secondary crashes and the local incident duration (mean and std dev) for specific incident classes for both the before and after cases.

Identified high crash frequency freeway segments through evaluation of records for all crashes (same and opposite direction within 2 miles and 2 hours) to identify secondary
crash patterns.

Descriptive statistical analysis of key measures and conduct tests for statistical significance of differences between before and after data.

Identify and document confounding factors.

Table 2-5. System Impact Experimental Design for Capacity/Throughput

Objective	Hypothesis	Measure	Data Sources	Analysis Method
To determine if CAD-TMC integration enhances incident-specific traffic management plans.	CAD-TMC integration enhances incident-specific traffic manage-ment plans.	Determine the diversion effect on traffic volumes over the affected link for specific incident classification. Compare baseline and after data.	For high crash frequency freeway sections: measure volume during incidents of each particular classification to approximate the level of traffic diversion.	Descriptive statistical analysis of key measures and comparison of baseline and after cases.

3.0 Data Collection Plan

The Data Collection Plan provides the process by which the Evaluation Team will collect and store data over the course of the CAD-TMC FOT evaluation. The purpose of the Data Collection Plan is to ensure that data are collected in an efficient, systematic, cost-effective manner, and that the integrity of collected data are upheld. These efforts will help ensure that the CAD-TMC FOT evaluation can be completed with as little time and effort as feasibly possible.

Data collection will occur before (baseline) and after the FOT deployment. Collection of before data will focus on establishing a baseline that will be used to measure the impact of the FOT deployment. Collection of after data will provide the data that will be compared to the baseline data to determine the impact of the FOT deployment. Collection of before and after data will occur via any of following identified methods:

Stakeholder/Vendor Interviews. The Evaluation Team will interview stakeholders/vendors in person or via phone as the primary means to collect the information/data needed to successfully perform the CAD-TMC FOT evaluation. Stakeholder interviews will also be used as a means of identifying issues relevant to the CAD-TMC evaluation, including other relevant data sources, but not limited to existing documentation or vendors.
Document Reviews. The Evaluation Team will identify, request, and review documentation applicable to the CAD-TMC FOT evaluation. Documentation that may be reviewed includes meeting minutes, project notes, project reports, and Website/CAD system logs.
Manual Data Collection/Extraction. Data often will be manually collected or extracted from agency systems. For instance, several types of data exist within the CARS database that when extracted and compared with CAD system data, may provide valuable insight into how incidents are reported. Additionally, manual data collection provides the Evaluation Team the ability to become more familiar with the data.
Site Visits. The Evaluation Team will request and schedule site visits with appropriate stakeholders/vendors to collect needed data that cannot be easily transmitted via phone, e-mail or other convenient means. Site visits may be appropriate for determining the type, condition, and number of systems each stakeholder currently has in place.
Observations. Visual observations may be used as a means of collecting data that is not otherwise documented or easily conveyed. For instance, visual observations may be used to document activities of CAD and TMC operators.
Questionnaires. Questionnaires will be developed to obtain input from several groups both before and after the deployment. Groups targeted by the questionnaires will include incident management personnel and the public. A questionnaire targeting the traveling public will be posted on WSDOT’s Website, and will gauge public satisfaction before and after the deployment.

The Evaluation Team identified the following data requirements for the CAD-TMC FOT evaluation:

Types of incidents “broadcast” on the CAD-TMC FOT system and the type of information available about those incidents.
Frequency with which information on events in shared.
Lag time between incident verification and information availability to the TMC, to other CAD users, and to the public.
Quality and accuracy of information exchanged.
Delay times in responding to dispatch requests.
Type of TMC information available on the CAD-TMC FOT.
The ease of access to TMC information on the CAD-TMC FOT.
Degree of interoperability between the participating FOT stakeholders outside of the CAD-TMC FOT.
Extent to which the deployed system was used by CAD and TMC operators and by secondary responders.
Degree to which the system decreased reliance on manual methods for exchanging data.
The identification of secondary responders who are utilizing the system.
The documentation of information made available to responders and the extent to which it is used.
Determination of applicable standards associated with existing systems.
Determination of system types and system vendors.
Identification of standards considered for use by the FOT including the reasons why or why not standards were used.
Identification of differences between different GIS systems.
Crash Statistics and Impacts.
Time to post CAD incident information.
Number of incidents that can be reported on the CAD system.
Time it takes from when incidents occur to when information becomes available to the public via 511 or Website.

Once data is collected it will reside in a single, centralized database over the life of the evaluation study. Data will be routinely downloaded from the FOT sites to the centralized platform, processed, and analyzed. Individual databases – or database subsets – will be copied to the evaluators’ local computers, as appropriate, in support of more sophisticated analyses, special studies, etc. Updates to the databases will always be made on the central platform and then copied, as needed, to the local platform.

The following elements will be stored on the central platform:
Raw data precisely as downloaded from the FOT sites.
Sanitized data after it is certified “compliant”.
Archived analysis tools and data queries.
Outputs of the assessment process.

Naming convention safeguards and control procedures will be implemented to ensure that individual “snapshots” of the raw data, as downloaded from the FOT sites, are maintained and not overwritten by subsequent updates to the database. As new tools and queries are defined, they will be added to the archive. Access to the central platform will be carefully limited and controlled.

Quantitative data will generally be stored as files using Microsoft^® Access. Qualitative data will be stored as text files, typically in Microsoft^® Word, and will be organized in directories and files by subject matter.

Access to the central platform will be password-protected so that only authorized members of the Evaluation Team can successfully log on to system. Even among Evaluation Team members, only those individuals who are designated as Database Administrators will have rights to update the original databases. Other users will be able to copy the databases only, customize them to address specialized needs, and generate and execute queries. Those users will not, however, be authorized to change or update the databases.

All data saved to the central platform will be simultaneously imaged to dual hard drives, to ensure ongoing data backup activities. As an additional precaution, the hard drives will be backed up daily, whenever there is activity on the platform.

The databases, analysis tools, and system outputs will all be archived with date-and-time stamps. At the conclusion of the study, the final databases and archived analysis tools will be delivered to FHWA.

4.0 Detailed Test Plans

This section provides the detailed test plans for the CAD-TMC evaluation. The remainder of this section is organized as follows:

4.1 System Performance
4.2 System Impact
4.3 Institutional Issues
4.4 Technical Issues
4.5 Lessons Learned
4.6 Benefits Summary

4.1 System Performance

The System Component Performance Study will address two primary objectives of the CAD-TMC evaluation: (1) examine system component performance, and (2) discuss how well the project meets the FOT objectives.

In addition to examining the system component performance and determining how well the project meets the FOT objectives, these evaluation activities will also:

Extend the level of integration to include secondary responders such as utilities; towing and recovery; public works; and highway maintenance personnel.
Evaluate the degree to which ITS standards (such as IEEE 1512 and NTCIP) were incorporated into deployed system.
Address the approach used to share data between map databases from different vendors and GIS standards that were applied.

Data Requirements and Collection Methods

Three types of data are required for this part of the evaluation: the key system performance measures; the design basis for those measures and their computation; and the observed values of those measures for the deployment. The following list provides a starting point for data collection of key system performance measures as identified as part of the Evaluation Plan:

The types of incidents “broadcast” on the CAD-TMC FOT system and the type of information available about those incidents.
The frequency with which information on events is shared.
The lag time between incident verification and information availability to the TMC, to other CAD users, and to the public.
The quality and accuracy of information exchanged.
The delay times in responding to dispatch requests.
The type of TMC information available on the CAD-TMC FOT.
The ease of access to this TMC information.
The degree of interoperability between the participating FOT stakeholders outside of the CAD-TMC FOT.
The extent to which the deployed system was used by CAD and TMC operators and by secondary responders.
The degree to which the system decreased reliance on manual methods for exchanging data.
The identification of secondary responders who are utilizing the system.
The documentation of information made available to responders and the extent to which it is used.

This list of performance measures will be refined and explored via email exchanges and discussions at meetings with the Deployment Team, as well as by reviewing acceptance criteria that are part of the system testing. Documenting the observed values of the key system performance measures will help determine whether the deployed system lives up to its design expectations.

Test Activities and Schedule

For the purpose of this evaluation, test activities are further defined as pre-test, test, and post-test to indicate pre-deployment, during deployment, and post-deployment evaluation activities.

Pre-Test Activities

The pre-test activities will focus on identifying the design values for the performance measures. These activities will include the following:

Pre-deployment interviews will be conducted with deployment staff to identify the expected values for the key performance measures.
Post-deployment interviews will be conducted with deployment staff to identify the values of key performance measures that were verified during system testing.
Design document review to identify expected values of key performance measures.

Test Activities

The test activities will focus on identifying the values of key performance measures of the operational system early in the operational period. Post-assessment activities will complement these early values with those values captured late in the assessment after the system has matured and the users are more familiar with it. The test activities will include:
Interviews will be conducted with deployment staff and CAD, TMC, and secondary responder operators to identify their perceptions of the system performance, and in particular, the values of key system performance measures.
Observations of CAD and TMC operator activities will be recorded to gauge the level of reliance on manual systems for coordinating CAD and TMC activities.
CAD and CARS message logs will be collected to support analyses of the data transmitted by the system and the timing of data transmissions. These logs will be gathered periodically during the assessment period so that changes to the operational characteristics over time can be observed.
Data collection and interviews will be conducted to support case analyses of incident response activities. Incidents will be selected for case analysis to represent instances where the system provided typical performance, as well as instances where the system performed both especially well and especially poorly.
Case analyses will be conducted both very early and very late in the assessment phase to identify changes in response activities that occur during the assessment period.
Software and Website usage statistics will be collected periodically during the assessment period so that changes to the operational characteristics over time can be observed.

Post-test Activities

Post-test activities will consist primarily of conducting interviews with deployment staff and CAD, TMC, and secondary responder operators focused on how system usage and the system performance measures changed during the assessment period.

Data Analysis Report

The majority of the analysis will include a review of interview findings. Software and Website usage will be compared to the number of incidents that occurred to estimate the fraction of incidents for which the system was used. Trend analysis will also be used to examine changes in usage rates as users become more familiar with the system.

Format and Expected Contents

The report will document the observed performance measures and changes in those measures during the assessment period. The report will also summarize interview findings and provide a text description of the other factors affecting system performance.

Data Requirements and Collection. The Evaluation Team will gather information about the perceptions of FOT stakeholders on factors, other than the system performance measures, that may have affected the use of the system.
Data Analysis. The Evaluation Team will review the interview findings.
Report Format and Expected Contents. A text description of the other factors affecting system performance will be included in the report.

4.1.1 Integration of Secondary Responders

The objective of the secondary responder assessment will be to document how:

Participating agencies at each level are notified of the need to respond to incidents.
The CAD integration program benefits the secondary responders.
Future changes to the program can further improve the secondary response capability

A baseline Secondary response issues will be developed as a component of the overall “before” data collection. The evaluation team will then monitor how these challenges are addressed and resolved throughout the course of the FOT. The evaluation recognizes that this list of challenges will change during the course of the FOT as existing challenges are resolved and new challenges are identified through the course of the deployment. The “after” data collection will be used to obtain stakeholder assessments on how the FOT improved the process of notification and deployment of secondary responders including the impact on overall response and clearance times.

Data Requirements and Collection Methods

The data required to conduct this particular test will be qualitative in nature. Data will be collected through:

Stakeholder interviews – The primary information source for identifying issues and the processes by which they were resolved will be through interviews with project stakeholders with specific focus on secondary responders.
Document review – Interviews will be supplemented by the review of documents (meeting minutes, correspondence, project reports) generated through project activities. Document review, in particular meetings minutes, will be used to document the processes by which institutional issues were resolved.

Test Schedule and Activities

The baseline stakeholder interviews will be conducted as part of the overall FOT “before” and “after” data collection activities. Review of documents and observations by the evaluation team will be done on an on-going basis throughout the FOT. Findings from the document review and evaluation team observations will be incorporated into the “before” and “after” stakeholder interviews.

Pre-test activities will focus on identifying and listing secondary responders, and how they are now notified and deployed. Reviewing the results of the evaluation kick-off meeting and strategy briefing will be used to identify issues from secondary responders.

In developing questions for the secondary responders, the evaluation team will review questions used for stakeholder interviews in deployment evaluations and select those that can be tailored to address the needs of the FOT. The evaluation team will also rely on the experience of individual team members in working with FOT agencies to develop additional questions and revise the format of questions as necessary.

The questions will be reviewed with the Project Manager to ensure that secondary responder issues are adequately addressed.

Test activities will include scheduling and conducting secondary responder interviews for the “before” and “after” phases of data collection.

Post-test activities will include the analysis of information obtained through secondary responder interviews and document review and the preparation of the final evaluation report and the final evaluation briefing.

Data Analysis

The analysis will focus on determining how secondary responders are requested, by whom, when they are notified, and how the CAD FOT impacts that process.

Report Format and Expected Contents

The secondary responder report will be incorporated into the overall final evaluation report and the final evaluation briefing as separate sections of each deliverable. The sections will be structured as follows:

Statement of requirement from RFP and summary of proposed evaluation plan and detailed test plans.
Summary of methodology and process used to design the plan and conduct the test.
Findings - this will include the results of the data analysis.

Conclusions and recommendations derived from the results of this review will be incorporated into a stand-alone section in the final evaluation report and final evaluation briefing.

4.1.2 Standards Test Plan

The objective of the standards assessment will be to document the degree to which standards were incorporated into the FOT deployment. This includes determining the degree to which systems were integrated before and after FOT deployment. To achieve this objective the Evaluation team will perform the following steps:

Conduct a scan of existing standards (e.g., IEEE 1512, NTCIP) to determine which standards are ready for deployment.
Scan all standards activities to ensure that the most current standards information is available, including standards validation and vendor compliance.
Identify which standards the FOT teams selected and why they selected them.

Data Requirements and Collection Methods

The data required to conduct this particular test will be qualitative in nature. Data will be collected through:

Stakeholder/Vendor interviews – The primary information source for identifying the standards that were incorporated in the FOT deployment will be via stakeholder interviews. Stakeholder interviews will focus on determining the applicable systems operated by each agency.
Review of Documentation – Existing standards documentation will be reviewed to obtain the most current standards information.
System analysis – System analysis will supplement stakeholder interviews by allowing the project team the ability to identify systems and then comparing systems with vendor specifications and standards.

Test Schedule and Activities

Pre-test Activities

Pre-test activities will focus on scanning the standards development arena to determine applicable standards that are ready or near-ready for deployments and identifying systems currently in place as a basis to determine the applicable standards associated with each system. This includes scheduling site visits with appropriate agencies that operate systems to determine the system types and system vendors.

Test Activities

Test activities will consist of scheduling and conducting stakeholder/vendor interviews for the “before” and “after” phases of data collection. The interviews will focus on the standards considered for use for the FOT and reasons for either using a selected standard or not using an applicable standard.

Post-test Activities

Post-test activities will include the analysis of information obtained through stakeholder/vendor interviews and systems identification and the preparation of the final evaluation report. The analysis will primarily be a comparison of standards used and those that could be applied to this type of effort and are ready for use.

Data Analysis

Qualitative data collected from site visits and interviews will be analyzed to determine the degree to which standards were incorporated into in the FOT deployment. This includes a comparison of standards in use before and after the deployment. The analysis will include why the data is applicable, and whether ready to deploy standards were used or were not used. For those used, the analysis will include a discussion of how easy it was to implement the standard and how well the standard matched the needs of the project.

Report Format and Expected Contents

The standards assessment report will be incorporated into the both overall final evaluation report and the final evaluation briefing as separate sections of each deliverable. The sections will be structured as follows:

Statement of requirement from RFP and summary of proposed Evaluation Plan and Detailed Test Plans.
Summary of methodology and process used to design the plan and conduct the test.
Findings, which will include the results of the data analysis.

Conclusions and recommendations derived from the results of the standards assessment will be incorporated into a stand-alone section in the final evaluation report and final evaluation briefing.

4.1.3 GIS Standards Test Plan

Test Objectives

One difficulty that is often encountered when sharing data betweeen road-based systems is that of overcoming the incompatibilities in the underlying map data in the systems. These incompatibilities can be simple to correct as different naming conventions for roads (e.g., “Rd” insteas of “Road”; “1^st” instead of “First”); or can be as complex to correct as actual differences in the road topology (e.g., missing roads); differences in the road names; or differences in the road coordinates. Taken together, these incompatibilities can degrade the effective communication between systems.[2] Documenting how the WSDOT FOT overcame these difficulties – and the extent to which these difficulties impeded the effectiveness of the system – can help future sites that might employ similar techniques. The objective of these assessments will be to document how ITS standards and the sharing of map databases using GIS were successfully incorporated into the system.

Date Requirements and Collection Methods

The Evaluation Team will obtain qualitative assessments on how well ITS standards and the sharing of map databases using GIS through interviews with the Deployment Team, system operators, and end-users. These interviews will be conducted in conjunction with interviews conducted for other tests. The Evaluation Team will identify questions in support of this test that will be incorporated into an overall interview guide.

The Evaluation Team will also conduct performance tests on the effectiveness of the approaches used to share data by conducting round-trip exchanges of location information.

Test Schedule and Activities

Pre-Test Activities

Pre-test activities will focus on scanning the GIS systems used by different vendors to identify differences that need to be addressed to share information and identify the approaches to be used for sharing this information.

Test Activities

Test activities will consist of scheduling and conducting stakeholder/vendor interviews for the “before” and “after” phases of data collection. Test activities will also include the conduct of the performance tests.

Post-Test Activities

Post-test activities will include the analysis of information obtained through stakeholder/vendor interviews, and the performance tests and the preparation of the final evaluation report.

Data Analysis

The Evaluation Team will document the results of all interviews to document:

The approaches used to share data between systems.
The frequency with which location data was incorrect.
The extent to which poor location data impeded incident response.

The Evaluation Team will analyze the results of the round-trip exchanges of location information by performing statistical tests on the differences introduced. The intent of these analyses will be to determine the accuracy of the information exchanged.

Report Format and Expected Contents

The GIS standards assessment report will be incorporated into both the overall final evaluation report and the final evaluation briefing as separate sections of each deliverable. The sections will be structured as follows:

Statement of requirement from RFP and summary of proposed Evaluation Plan and Detailed Test Plans.
Summary of methodology and process used to design the plan and conduct the test.
Findings, which will include the results of the data analysis.

Conclusions and recommendations derived from the results of the standards assessment will be incorporated into a stand-alone section in the final evaluation report and final evaluation briefing.

4.2 System Impact

4.2.1 Test Objectives

The system impact test will be conducted to quantify and document the impact of CAD-TMC Integration on Incident Management processes. Ultimately, the benefits of will be measured from the points of view of two different beneficiary groups: incident responders and travelers. The objectives of the System Impact Study will be to determine if the integration of CAD and TMC systems:

Improves the efficiency and productivity of incident response.
Improves mobility and reduces incident-caused delays.
Reduces the exposure time of response personnel on-scene.
Reduces secondary crashes related to incidents.

The evaluation premise for the incident response community is that integration yields the following benefits:

Reduced responder dispatch time
Improved information available to the responders
Increased likelihood of more tailored response
Faster and more accurate motorist information improves on scene safety
Decreased number of secondary incidents

Within the response community, the stakeholders are defined as primary responders and follow-on responders:

Primary responders are the WSP troopers, Fire Departments, EMS, and DOT response crews that are dispatched to handle the incident and establish the follow-on response requirements.
Follow-on responders from other agencies are dispatched to the scene based on the nature of the incident. This group can include hazardous materials teams, towing and recovery crews, DOT maintenance, and others as needed for specific types of incidents such as fatalities.

The benefits of integrating the dispatch systems for these two communities are anticipated to be quicker response and a more accurate understanding of the incident management requirements for the specific incident in progress. This improved situation awareness is anticipated to lead to more efficient execution of the incident management activities.

The evaluation premise for the second beneficiary group, the travelers, is that improvements in incident management:

Increased mobility (reduced delay).
Increased safety (reduced secondary crash rates) (reduced incident duration, improved Incident specific traffic management plans
Improved diversion route availability and performance information).

To support the tests required to prove or disprove the system impact hypotheses, the Evaluation Team developed an assessment framework that facilitates the measurements required to support experimental analysis. The method is an adaptation of standard process performance measurement tools used in project management.

The first step in developing the method is generation of an Incident Management Process description, which identifies the activities that take place within the process. Activities within the process will be defined as measurable tasks.

A measurable task is one for which the beginning and end times can be determined; the resource requirements of the task can be easily determined; and the prerequisites for task initiation can be identified.[3]

Individual measurable tasks that are related in purpose or sequence will be grouped into phases to provide visibility into the interactions between the activities within the phase and between activities in one phase with the activities of those in another phase. A conceptual illustration of the incident management process, illustrated in general phases, is shown in Figure 4-2[4]. This figure shows the functional areas of traffic incident management and how they are linked, beginning with: initial detection/verification; incident response; site management of the incident; and, clearance of the incident. These phases include response initiation, life-saving response, environment mitigation, restored roadway operations, and repair and recovery.

With the individual incident management activities defined, the complete inventory of tasks required to manage a particular incident (or class of incident) will be used to define the process in the form of an activity network diagram. The network diagram will represent the process as a system allowing quantitative assessment along key performance measures using Critical Path Methods[5].

Data Requirements and Collection Methods

The data and information required will be defined by answering the following questions:

Type of incident
Where was it?
What time did it occur?
When was it reported?
When was response dispatched?
When did they arrive?
When were secondary responders requested?
When did they arrive?
How many lanes were blocked?
When did each lane get cleared?
What were the weather conditions?
When was the TSMC advised?
What actions were taken by the TSMC? (DMS messages, 511 information, etc.)
When were the messages or motorist information updated?
How did the highway section perform over the timeframe (including a period before and after)?
Were there any secondary effects (i.e., crashes, severe impacts on diversion route flow, etc.)?

Figure 4-1 provides an example of the integrated picture, which can be developed if the methods used to answer these questions can be fully supplied with data. The Evaluation Team recognizes that full data availability across an entire region may be problematic, and, therefore, will work with DOT personnel to target corridors where there are significant incident histories and high levels of data available.

Figure 4-1. Key Data Points in Critical Incident Management Data Fields
with Respect to the Time Period Immediately Before to Complete Recovery

The methods used to gather this data and information will include the following elements: surveys; interviews; data queries of the incident management system; and data queries of important supplemental databases, including highway performance data and crash histories. Survey groups will include incident management personnel both before and after deployment. The surveys will be supplemented with interviews that will provide the Evaluation Team the important contextual information. Databases archived by the incident management system will be accessed using routine and/or special queries to support generation of the data records described in a subsequent section.

The data will be comprised of several databases that must be time-tagged to support future analysis queries. A key component of the data collection will be the development of a time synchronization method to allow relational analysis. Representative examples of key incident management data fields are illustrated and described in the following tabular field context. These tabular fields may be embellished as required to describe complex incidents. Where a single entry is shown for a responder category and multiple agencies are involved, the data fields will need to increase to capture the same level of detail for each responding agency and/or crew.

Incident Identification: Data in these fields will be used to determine the nature of the incident, the source of the notification and will start the incident management duration clock for timing purposes. Data fields are identified as Incident ID, Incident Type, Initial Notification ID, and Notification Time.

Incident Identification

Incident Location and Impact on Traffic Flow: Data in these fields will be used to determine the location of the incident and the impact on the traffic carrying capacity of the affected roadway section. Data fields are identified as Location, Shoulder Blocked, and Lanes Blocked.

Incident Location and Impact on Traffic Flow

Primary Response Information: Data in these fields will be used to determine primary response patterns (distribution between agencies) and primary response times for CAD-TMC integrated and non-integrated agencies. Data fields are identified as Primary Response Agency, CAD-TMC Integrated, Primary Response Dispatch Time, and Primary Response Arrival.

Primary Response Information

Follow-on Response Information: Data in these fields will be used to determine follow-on response patterns (distribution between agencies) and follow-on response times for CAD-TMC integrated and non-integrated agencies. Data fields are identified as Follow-On Response Agencies, CAD-TMC Integrated, Follow-On Dispatch Time, and Follow-On Response Arrival.

Follow-on Response Information

Responder Departure, Lane Restoration History, and Incident Termination: Data in these fields will be used to determine demobilization patterns and the progression of lane re-opening throughout the incident management process. Data fields are identified as Primary Response Departure, Follow-On Response Departure, Lane Restoration History, and Incident Termination.

Responder Departure, Lane Restoration History, and Incident Termination

The Evaluation Team realizes that the range and depth of data available at the beginning of the effort may be less than that available at the end of the effort. This enrichment of data availability, particularly in relational format, will be a key aspect of the evaluation leading to improved ability to monitor and measure the incident management system.

Supplemental data to relate the changes in incident management to the impacts on mobility and safety include:

Highway performance indicators (volume, spot speeds, etc.).
Traffic queues or congestion measurements.
Crash histories.
ITS device histories, if applicable.
VMS history, if applicable.

Test Activities and Schedule

Pre-Test Activities

Pre-test activities will be directed towards achieving the following three objectives:

Document the current incident management process in terms of an incident management activity inventory develop a list of the agencies that supply incident management services.
Document the current communications network used to notify and dispatch the required agencies to the scene of an incident. This effort should identify the communications pattern (center-to-center, center-to-responder, and responder-to-responder). This activity should include a description of the communication technology most often employed on each of the communications network links.
Identify the geographic locations (within the region served by the CAD-TMC integration) with significant incident histories. For those locations, identify the instrumentation associated with that geographic location and determine which of the desired highway performance measures can be supported.
Inventory the traffic management devices (lane control signals, VMSs, VSL signs, and TMC-controlled traffic signals) in the area of the candidate locations to ensure that the use of those in the incident management process can be captured and the effects can be considered in future analysis.

Test Activities

Test activities can be divided into two broad categories depending on the time frame of interest: documentation of the baseline system performance, and documentation of system performance over time. Activities within each are briefly described as follows.

An important first step evaluating the CAD-TMC integration is to document the baseline performance of the mature, high-performance incident management system currently in use. Capturing the system operational description and performance qualities in its current form will establish a benchmark performance level. This effort will include documenting the incident management system and evaluating the performance of each component along key measures of effectiveness.

Once the baseline process is documented, the next step is to conduct a series of measurements over time to document the “learning curve” associated with the CAD-TMC integration. The Evaluation Team will work closely with the stakeholders to identify the “milestone” events that will take place over the life of the deployment and evaluation.

Post-Test Activities

Post-test activities will include documenting the findings in the framework and format presented in a subsequent section of this DetaiCed Test Plan. Within the Post-Test Activities report, the technical content for the final report will be developed and archived.

Data Analysis

Data to support the CAD-TMC integration evaluation is expected to be collected in three forms: incident management personnel survey results; interview results; and quantitative data. The first step in the survey data analysis will be to develop the descriptive statistics for the responses to each type of question. Using this information, a before and after comparison will be made, and the statistical significance of the change will be computed. If required, due to the relatively small population in each personnel category, non-parametric methods will be used.

The primary purpose of the interviews is to generate descriptions of different aspects of the system’s operational concept and the resulting performance. In addition, the interviews will provide the opportunity to capture anecdotal benefit statements that may be paired with survey or data analysis.

The second purpose of the interviews is to capture some response quotes which will enhance the final report by relating issues, concerns, and breakthroughs to the target audience (public officials and system implementation personnel) in the terms used by their peers.

Quantitative data is being sought in several areas as highlighted in the Data Requirements section. The first objective in the quantitative analysis will be to determine the descriptive statistics and frequency distributions of the data sets in question. With this information, the analysis will progress to determine the statistical level of significance in response to the implementation the CAD-TMC integration system.

Both parametric and non-parametric methods will be used, if required. Methods will be selected based on analysis of the data. Candidate parametric methods include determination of the Statistical Power of the difference given a desired significance level and the T-Test or the F-Test to determine the existence of a difference between before and after data sets. Non-parametric methods include Chi-Squared and Kilmogorov-Smirnoff tests for difference in system performance due to the treatment in question.

4.2.2 511/Internet Interface Test

The objectives of the 511/Internet interface portion of the System Impact Study is to determine if integrating CAD and TMC systems result in the following:

Enables near real-time data exchange with 511 and Internet-based traveler information.
Improves customer satisfaction and mobility during incident management activities by improving traveler information.
Reduces the time needed for the news media to obtain and disseminate improved traveler information.
Proves the hypothesis that each of these facets is true.

The measures of effectiveness that the Evaluation Team will use to test the hypothesis are:

Determine the change in the percent of eligible incidents reported on the traveler information Website and the 511 systems.
Determine the change in time between when the incident occurred and when information became available to the public via the Website and 511 systems.
Assess the satisfaction of the traveling public with improved traveler information.

Data Requirements and Collection Methods

Data to test the hypotheses will be collected through:

WSP CAD Report: This report will be taken as the basis for measuring both the time it takes to post incident information and the number of incidents that could be reported. The CAD data will essentially be considered “ground truth” for evaluating (1) the change in the percent of eligible incidents reported on the traveler information Website and the 511 systems, and (2) the change in time between when the incident occurred and when information became available to the public via the Website and 511 systems.
WSDOT CARS Database: The CARS database will represent the time that incidents are entered into the WSDOT reporting systems. By comparing this time of incident the time in the CAD reports, the Evaluation Team will be able to assess the change in time to report incidents. By comparing the number of incidents in the CARS database to the incidents in the CAD system, the Evaluation Team will be able to assess the percent of the eligible incidents that are reported to the 511 and Internet systems.
Website Logs: Website logs will be used to determine if there is a significant delay from the time data gets into the CARS database and the time it is reported to the public.
Web-Based Questionnaire: The questionnaire will be presented to the traveling public who use the WSDOT Website. The questionnaire will not be a statistical sample of the public at large, or even of those who use the Website. The results will provide an indication of customer satisfaction, not a representation of overall customer satisfaction. The questionnaire will be designed to take only a few minutes and will try to gauge not only how well satisfied users are, but to glean specific concerns about the incident related traveler information provided.

Test Schedule and Activities

Pre-Test Activities

Pre-test activities will focus on scanning the data from the CAD system, CARS, and Website logs to become familiar with the data and how it is reported. Pre-test activities will also include designing the Web-based questionnaire.

Test Activities

Test activities will consist of collecting CAD, CARS, and Website log data, and fielding the Website questionnaire for the before and after phases of data collection.

Post-Test Activities

Post-test activities will include the analysis of information obtained through the data collection effort and the preparation of the final evaluation report. The analysis will include both quantitative comparisons of before and after data from the CAD logs, CARS database, Website logs, and numerical input on the questionnaire and a qualitative analysis of comments provided from the Website questionnaire.

Data Analysis

Quantitative comparisons of time to post information and percent of eligible incidents that are reported will be undertaken by the team. Likewise, a quantitative comparison will be made of the numerical input to the before and after questionnaires. Statistical analysis of the data will be undertaken to determine the significance of any before and after differences. In addition, comments submitted as part of the questionnaires will be reviewed to see if any conclusions can be drawn from that source of information.

Report Format and Expected Contents

The 511/Internet interface report will be incorporated into the system impact study portion of the final evaluation report and the final evaluation briefing. The 511/Internet interface section will be structured as follows:

Statement of requirement from RFP and summary of proposed Evaluation Plan and Detailed Test Plans.
Summary of methodology and process used to design the plan and conduct the test.
Findings - this will include the results of the data analysis.

Conclusions and recommendations derived from the results of the 511/Internet interface assessment will be incorporated into a stand-alone conclusions and recommendations section in the final evaluation report and final evaluation briefing.

4.3 Institutional Challenges

The objectives for the institutional challenges assessment will be to document how:

Participating agencies at the state level identified and resolved institutional challenges. This will include an assessment of the effectiveness of the project management process.
Obtaining the participation of municipal, county, and local government agencies in the project was accomplished.
The high level of integration and coordination of services between state agencies that was already in place at the start of the FOT was achieved.

A baseline of institutional challenges will be developed as a component of the overall baseline (before) data collection. The Evaluation Team will then monitor how these challenges are addressed and resolved throughout the course of the FOT. The Evaluation Team recognizes that this list of challenges will change during the course of the FOT as existing challenges are resolved and new challenges are identified through the course of the deployment. The after data collection will be used to obtain stakeholder assessments on how institutional challenges were resolved.

Institutional challenges will be identified in three ways: interviews with FOT stakeholders (most important); review of FOT related documents; and observations by the Evaluation Team. The identification of challenges will include:

Challenges directly related to the successful deployment of the FOT.
Challenges that impact the scalability of the FOT in other jurisdictions.

Data Requirements and Collection Methods

The data required to conduct this particular test will be qualitative in nature. Data will be collected through:

Stakeholder Interviews. The primary information source for identifying issues and the processes by which they were resolved will be through interviews with project stakeholders.
Document Review. Interviews will be supplemented by the review of documents (meeting minutes, correspondence, project reports) generated through project activities. Document review, in particular meetings minutes, will be used to document the processes by which institutional challenges were resolved.

Test Schedule and Activities

The baseline stakeholder interviews will be conducted as part of the overall FOT baseline (before) and after data collection activities. Review of documents and observations by the Evaluation Team will be performed on an ongoing basis throughout the FOT. Findings from the document review and Evaluation Team observations will be incorporated into the before and after stakeholder interviews.

Pre-Test Activities

Pre-test activities will focus on identifying institutional challenges and developing questions that will be used for stakeholder interviews. Reviewing the results of the Evaluation Kick-off Meeting and strategy briefing will be used to identify institutional challenges. In addition, institutional challenges will be identified through the review of all FOT-related documents and initial conversations with the FOT Project Manager and key staff supporting the FOT.

Questions developed for interviews will be designed to:

Obtain information about a particular challenge. This information will be descriptive in nature and will be used to present details of what the challenge is, how it was identified, and what its potential impact on the FOT might be if not resolved.
Establish a qualitative baseline against which the after FOT information can be compared to determine the extent to which the challenge was resolved and how this was accomplished.

In developing questions, the Evaluation Team will review questions used for stakeholder interviews in deployment evaluations and select those that can be tailored to address the needs of the FOT. The Evaluation Team will also rely on the experience of individual team members in working with FOT agencies to develop additional questions and revise the format of questions as necessary. The questions will be reviewed with the Project Manager to ensure that any institutional sensitivity is adequately addressed and that all challenges are identified and addressed by the questions.

Test Activities

Test activities will include scheduling and conducting stakeholder interviews for the before and after data collection phases.

Post-Test Activities

Post-test activities will include the analysis of information obtained through stakeholder interviews and document review and the preparation of the final evaluation report and the final evaluation briefing.

Data Analysis

The information obtained will be synthesized and categorized as necessary in terms of the particular challenges identified. The analysis will focus on determining how a particular challenge was resolved, if the stakeholders felt the resolution was appropriate, and what impact the particular challenge had on the FOT. The analysis will be qualitative in nature, and will focus on the following elements:

Contrasting each challenge as identified and summarized in the before phase with the after phase.
Summarizing what actions occurred during the FOT to address the particular challenge.
Assessing how stakeholders felt the challenge was resolved, and how the resolution of each challenge impacted the overall FOT.

Report Format and Expected Contents

The institutional challenges report will be incorporated into the overall final evaluation report and the final evaluation briefing as separate sections of each deliverable. The sections will be structured as follows:

Statement of requirement from RFP and summary of proposed Evaluation Plan and Detailed Test Plans.
Summary of methodology and process used to design the plan and conduct the test.
Findings - this will include the results of the data analysis.

Conclusions and recommendations derived from the results of the institutional challenges review will be incorporated into a stand-alone section in the final evaluation report and final evaluation briefing.

4.4 Technical Challenges

The objectives of the Technical Challenges Assessment are to document the key technical challenges faced by the FOT team, how these challenges are addressed, and the final resolution of the challenges. This evaluation test plan describes the approach that will be used by the Evaluation Team to document this information. The Evaluation Team developed this plan on the premise that this particular test will be ongoing throughout the course of the evaluation.

Data Requirements and Collection Methods

There are two types of data required to assess technical challenges: identifying challenges that occur, and documenting how the FOT Deployment Team resolves those challenges.

The data required to conduct this particular test will be qualitative in nature. Data will be collected through:

Stakeholder Interviews. The primary information source for identifying issues and the processes by which they were resolved will be through interviews with project stakeholders.
Document Review. Interviews will be supplemented by the review of documents (meeting minutes, correspondence, project reports) generated through project activities. Document review, in particular meetings minutes, will be used to document the processes by which technical issues were resolved.

Test Schedule and Activities

Pre-Test Activities

Pre-test activities will focus on identifying technical challenges and developing questions that will be used for stakeholder interviews. Reviewing the results of the Evaluation Kick-off Meeting and strategy briefing will be used to develop an initial list of challenges. In addition, technical challenges will be identified through the review of all FOT-related documents and initial conversations with the FOT Project Manager and key staff supporting the FOT.

Test Activities

Test activities will include the following elements:

During the deployment phase, the Evaluation Team will continue to review technical documentation for identified technical challenges.
During the deployment and operational phase, the Evaluation Team will also review management documentation to identify technical challenges that might be addressed.
When the Evaluation Team interviews FOT Deployment Team members, they will be asked about any technical challenges that may have recently occurred.
Whenever a technical challenge is identified and thought to be of potential interest to other sites deploying similar systems, the Evaluation Team will identify the FOT Team members who are addressing that challenge. The Evaluation Team will periodically interview individuals at those sites to identify how the challenge was addressed. The last such interview will identify the final resolution and any residual effect on the FOT.

Post-Test Activities

Post-test activities will include the analysis of information obtained through stakeholder interviews and document review, and the preparation of the final evaluation report and the final evaluation briefing.

Data Analysis

Contrasting each challenge as identified and summarized in the before phase with the after phase.
Summarizing what actions occurred during the FOT to address the particular challenge.
Assessing how stakeholders felt the challenge was resolved, and how the resolution of each challenge impacted the overall FOT.

Report Format and Expected Contents

The evaluation report will include a section for the Assessment of Technical Challenges. In this section, the Evaluation Team will document the following information about each technical challenge that was assessed during the evaluation:

The nature of the technical challenge.
The potential implications of the challenge for the FOT.
The approach used to resolve the challenge.
Any residual effect of the challenge on the FOT.

The text will also highlight any lessons learned about deploying and operating a system such as the WSDOT CAD-TMC Integration FOT.

4.5 Lessons Learned Summary

Test Objective and Approach

The Lessons Learned Summary will document the lessons learned, insights ascertained, and any other identified element by the FOT Team during the process of integrating the CAD and TMC systems. These lessons will be identified and documented by the Evaluation Team using a two-fold approach:

17. A content analysis, or scan, of existing CAD/TMC documentation and documentation resulting from the other test plan components. The documentation will be scanned to identify and extract “lessons” of interest to other states and regions potentially integrating CAD/TMC systems.

18. Interviews with key members of the FOT Team and selected stakeholders to glean major insights and sensitivities into the process and issues. This assessment activity is qualitative in nature.

Data Requirements and Collection Methods

At the outset of this effort, an electronic matrix will be created and used for collecting and capturing the lessons learned data. The matrix will be organized by activity sequence, or step, in the CAD/TMC integration process, beginning with “cultivating stakeholder support in CAD/TMC integration”, and extending through “operation and maintenance of an integrated CAD/TMC system.” Major categories in the matrix will be subdivided into activities or function areas, as appropriate. Lesson learned data will be entered into the matrix as they are gathered.

Documentation reviewed by the Evaluation Team will include materials describing the existing WSP/WSDOT integration, how the integration was accomplished, and other factors. Document review will also include outputs of other test plan components, i.e., the system performance assessment, system impact study, institutional challenges assessment, and technical issues assessment. Information will be recorded within the matrix when these test findings identify or point to lessons learned.

Questions related to lessons learned will be part of the instrument guiding the general interviews with FOT Team members and other stakeholders. As previously noted, the focus will be on lessons and insights into sequential components of the CAD/TMC integration process. Individual stakeholders, of course, will only be queried about the parts of the process with which they are familiar.

Test Schedule and Activities

Pre-Test Activities

Pre-test activities will involve defining the categories or steps for which lessons learned data will be captured. Once these categories are defined, the electronic matrix will be created. The lessons learned queries to be covered during stakeholder interviews also will be identified during the pre-assessment period.

As an aid to FOT Team members and stakeholders participating in the interview process, a short 2- or 3-page Lessons Learned Guide will be prepared and distributed shortly before the interviews are conducted. The guide will encourage interviewees to think systematically and structurally about the process as they identify the most vital lessons to be imparted to other states and regions heading down the CAD/TMC integration path. The reason for distributing the guide before the interviews will be to give stakeholders an opportunity to ruminate on the issues.

Test Activities

Test activities conducted during the assessment period will include Evaluation Team scanning documentation for lessons learned and conducting stakeholder interviews. Lessons information will be entered into the electronic matrix as it is identified and collected.

The capturing of lessons learned will be an ongoing process. However, a part of the assessment effort will not be able to be conducted until late in the period. For instance, other test plan components cannot be reviewed for lessons learned until after the tests are completed and documented. Similarly, it will probably be preferable to interview stakeholders late in the period, after they have experienced most of the integration process.

Post-Test Activities

Post-test activities will consist of analyzing the information in the lessons learned matrix and documenting the results.

Data Analysis

The contents of the lessons learned matrix will be analyzed for completeness, comprehen-siveness, and consistency. Redundancies will be eliminated, as appropriate. Data will then be extracted from the matrix. Recurring and “big-picture” themes will be identified and highlighted, and materials will be readied for presentation in narrative and display formats.

Report Format and Expected Contents

A major section of the final evaluation report will be devoted to lessons learned. The section will be organized by activity sequence in the CAD/TMC integration process. Lessons will be presented using a series of matrices and accompanied by succinct narratives.

4.6 Benefits Summary

Test Objective and Approach

The Benefits Summary will document the primary benefits that accrued as a consequence of CAD/TMC integration during the FOT. It will identify which benefits expected by the state were actually realized, which hypotheses in the CAD/TMC FOT Test Evaluation Plan were borne out, and whether any unanticipated gains were observed. The Benefits Summary will be compiled, principally, through extraction, assessment, and analysis of findings from the Systems Impact Study.
Development of the Benefits Summary will follow this general approach:

Develop a working baseline table of the expected benefits.
Utilize the table to monitor, on an ongoing basis, benefits realized as the CAD-TMC integration matures and the FOT studies are executed.
Conduct follow-up validation discussions, as necessary, with members of the FOT team and key integration stakeholders to ensure that all expected benefits expected are addressed and that the benefits captured are adequately understood.

Data Requirements and Collection Methods

Both quantitative and qualitative data, primarily from the Systems Impact Study, will be used to assess benefits. Most of the quantitative assessments will center on before and after data. In contrast, the bulk of the qualitative data will come from interviews with stakeholders.

Key data to be used in the assessment of findings will include:

Response and Clearance Times. Prior to the integration effort, each stand-alone system was capable of providing outputs and logs relaying incident response and clearance times. Consequently, access both to the pre- and post-integration system logs will be essential to executing the before and after quantitative assessments of the incident management benefits.
Operator Time per Incident/Activity. Each agency’s incident management operations will be observed and evaluated in relation to overall impacts of the integrated system on the center- and dispatch-related tasks that operators execute.
FOT Team Member/Stakeholder Input. A key source for identifying qualitative benefits will consist of interviews with a range of system stakeholders, including: CAD operators; TOC operators; first responders; secondary responders; facility managers; and police/incident management personnel (from on-scene to senior management). These interviews will help in the identification and assessment of non-quantifiable benefits, and in documenting how the benefits were realized.

Test Schedule and Activities

Pre-Test Activities

Pre-test activities will focus on understanding the expected benefits and compiling them in a table organized by user category, e.g., responder, system user, traveler, etc.

The identification of expected benefits will include a scan of existing CAD/TMC documentation and summarizing those benefits referenced in the draft test plan. The table will also capture how data from the Systems Impact Study are expected to support the benefit.

Test Activities

Test activities during the assessment period, the Systems Impact Study will be examined with regard to the beneficial impacts that resulted from integration. These impacts will be cross- tabulated against the table of expected benefits. A summary table, showing both benefits and the interrelationships between system impacts and measures of effectiveness, will also be generated. The most critical measures of effectiveness are expected to be incident response and clearance times, operator time per incident/activity, and FOT Team/stakeholder input.

Post-Test Activities

Post-test activities will consist of analyzing the information in the Benefits Summary Table and documenting the assessment findings.

Data Analysis

The Benefits Summary will assess both qualitative and quantitative data in order to determine the benefits of the integration and indicate the extent of benefits realized. It will center on expected benefits, including those articulated by FHWA, state stakeholders, key FOT Team members, and those identified in other recent sources.

Report Format and Expected Contents

The Benefits Summary will be a section of the final evaluation report. The summary will consist of a series of data tables accompanied by a succinct narrative. These materials will identify the benefits realized from CAD/TMC integration and describe how key benefits were achieved.

5.0 Estimated Resource Requirements and Test Management

This section of the Detailed Test Plans document presents the estimated resource requirements needed for completing the proposed tests. To ensure that the tests are conducted in a cost effective manner that uses available resources efficiently, the Evaluation Team will:

Consolidate data collection activities across all tests. To demonstrate how this will work, if the results of a particular stakeholder interview will be used to support more than one test, all necessary questions will be consolidated into a single questionnaire so that all information can gathered in one interview. The same approach will be used in collecting quantitative data – all data elements needed to conduct individual tests will be consolidated and collected at the same time.
Use on-site staff to lead the collection of before and after data. Mr. Leslie Jacobsen of PB-Seattle will be responsible for coordinating the identification and consolidating of all data requirements and for coordinating data collection activities. PB staff located in Washington State will primarily conduct the data collection activities.

The Evaluation Team will also assign a member of the team to manage each test component, as shown in Table 5-1. These staff will be responsible for working with Mr. Jacobsen to ensure that all data elements are identified and incorporated into the data collection plan. These staff will also be responsible for identifying issues and developing questions that will be incorporated into stakeholder interviews and the collection of qualitative data.

Table 5-1. Test Component and Responsible Team Member Assignment

Test Component

Assigned To

Computer-Aided Dispatch - Traffic Management Center Field Operational Test Final Detailed Test Plan: WSDOT Deployment

Contract #: DTFH61-02-C-00061 ITS Program Assessment Support Contract Task SA61004

1.0 INTRODUCTION

2.0 EXPERIMENTAL DESIGN

2.1 Introduction

2.2 Experimental Design

2.2.1 Describing the FOT Environment

2.2.2 Identifying Key Performance Measures

2.2.3 Experimental Design

3.0 Data Collection Plan

4.0 Detailed Test Plans

4.1 System Performance

4.1.1 Integration of Secondary Responders

4.1.2 Standards Test Plan

4.1.3 GIS Standards Test Plan

4.2 System Impact

4.2.1 Test Objectives

4.2.2 511/Internet Interface Test

4.3 Institutional Challenges

4.4 Technical Challenges

4.5 Lessons Learned Summary

4.6 Benefits Summary

5.0 Estimated Resource Requirements and Test Management

Contract #: DTFH61-02-C-00061
ITS Program Assessment Support Contract
Task SA61004