ITS - Intelligent Transportation Systems Report ITS Home Page

FY 2000 Integration Earmarks: National Evaluation Program
CapWIN: The Capital Wireless Integrated Network

Evaluation Strategy
ITS Program Assessment Support
Primary Contract No. DTFH61-96-C-00098
Task 9807

March 2002

PDF Version 83KB

Logo. Capital wireless integrated network

Submitted to:

Federal Highway Administration
Joint Program Office
Washington, DC

Submitted by:

Science Applications International Corporation
8301 Greensboro Road, MS E-7-6
McLean, VA

Logo. SAIC

NOTICE

This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange.  The United States Government assumes no liability for its contents or use thereof.     

Table of Contents

1.0 Introduction

The Washington Metropolitan Region is home to one of the most congested highway systems in the country.  In addition, the highway system passes through the states of Virginia and Maryland, and the District of Columbia.  The responsibility for managing the region’s transportation system is divided between these three jurisdictions and within the two states, is further sub-divided between counties and municipalities.  Incident management and response, in particular on I-495 (the Capitol Beltway around the metropolitan area), may involve response personnel from each of the jurisdictions as well as from neighboring counties and municipalities.

A major concern in the region at present is the lack of an integrated communications system that enables these jurisdictions to communicate directly with each other.  The current process used by incident response personnel in one area to relay a message to emergency response personnel in a neighboring jurisdiction is described as follows:

To address this lack of an integrated communications system for the region, the states of Maryland and Virginia, and the District of Columbia have established a partnership to implement the Capital Wireless Integrated Network (Cap-WIN) project.  Through the CapWIN project, an integrated transportation and public safety information wireless network will be developed and implemented for the Washington Metropolitan region.  This unique project will integrate transportation and public safety data and voice communication systems in two states and the District of Columbia, and will be the first multi-state transportation and public safety integrated wireless network in the United States. The University of Maryland Center for Advanced Transportation Technology, working with the University of Virginia and the International Association of Chiefs of Police, is providing project management and staff support to this project.

Based on the results of a study conducted under Contract # DTFH61-96-C-00098 (SAIC - ITS Program Assessment Support), Task #9809 titled “Phase I  - National Evaluation of Selected FY 2000 Earmarked ITS Integration Program Projects”, a decision was made to include the CapWIN project in the National Evaluation Program.  Under Phase II, the evaluation team is expected to complete work through the baseline data collection and analysis stage of the evaluation.  This report presents the evaluation plan required for the CapWIN Phase II project.

Four goals have been identified for the CapWIN evaluation.  Table 1.1 describes these goals, and provides a brief summary of what aspects of the project will be evaluated to support each goal.  The evaluation will focus on the transportation-related impacts of CapWIN, and data collection efforts will center on incident response and emergency management activities.  Data will be obtained from CapWIN end-users, who are anticipated to include roadside personnel involved in actual response activities, as well as from management and support personnel.  The focus of the evaluation will be primarily on assessing the transportation related benefits of the CapWIN project. The evaluation will also assess how the use of CapWIN can improve inter-agency and inter-jurisdictional coordination of emergency management response activities.

Table 1.1 Proposed CapWIN Evaluation Goals

Evaluation Goal

What Aspect of CapWIN Will Be Evaluated

Assess the Level of Customer Satisfaction with the CapWIN System

The evaluation team will assess end-users acceptance of the CapWIN system.  Measures of effectiveness (MOEs) to be considered will include end-user perceptions   of system reliability, system functionality, and benefits realized through enhanced communications capabilities.  Data will be collected through an end-user survey, which may be supplemented by interviews with select end-users.

Determine the impact of CapWIN on Mobility During Incident Conditions

The evaluation team will identify the impacts of CapWIN on system mobility, specifically during incident conditions.  MOEs will be developed to determine if the use of CapWIN reduces the time needed to respond to and clear an incident.  To the extent feasible, this analysis will be quantitative, but will also be supplemented by a qualitative analysis.

Determine impact of CAPWIN on safety of response personnel and frequency of secondary crashes

The evaluation team will assess the impact of CapWIN on.  It is hypothesized that the more efficient deployment of assets will result in a reduction in time spent by response personnel at roadside.  The premise of this hypothesis is that reductions in response time and incident duration will reduce the probability and frequency of secondary crashes.  To the extent feasible, quantitative data supporting this analysis will be collected.

Assess the cost savings of CapWIN in terms of incident response and management

The evaluation team will identify incident response and management cost savings resulting from the use of the CapWIN system.  The hypothesis is that reducing the time needed to respond to incidents and reducing the duration of incidents will in turn reduce the cost of incident response.  Data will be collected from agency records and post-incident reports and also from the survey of response personnel.  If necessary, the surveys may be supplemented by interviews with emergency response personnel.

2.0 Project Overview

2.1 Why CapWIN Is Needed

The Washington Metropolitan Area has one of the most congested highway systems in the United States.  In addition, the highway system traverses two states (Maryland and Virginia) and the District of Columbia.  The result of this is that incidents that can bring traffic to a virtual standstill (such as the 2001 event where an individual threatened to jump from the Woodrow Wilson Bridge) often require responses from multiple jurisdictions (in this instance, the jumper was located on the section of the bridge under the authority of the District of Columbia, but the highway back-ups were in Maryland and Virginia, thus requiring a coordinated response from all three jurisdictions).  To further complicate this scenario, incident response may be performed by state-level agencies (Virginia DOT or Maryland State Highway Administration), county-level agencies (Montgomery or Prince George’s Counties in Maryland, Arlington and Fairfax Counties in Virginia), or by municipalities (the City of Alexandria).

While coordination of incident and emergency response activities continues to improve, incident response and scene management is hampered by the inability of these myriad agencies to communicate directly, particularly in a mobile environment. Instead, these agencies must communicate by contacting their own communications centers, which then contact other agency/jurisdiction communications centers, which then finally contact their mobile/roadside response units.  This inability to communicate directly is particularly evident during major traffic incidents such as those that happen on the Capital Beltway on an all too frequent basis.  Transportation, law enforcement, and public safety agencies, for example, responding to the jumper incident on the Woodrow Wilson Bridge, were not able to communicate (either data or voice) directly with each other.   In order to get a message from one agency’s response unit to another, responders had to communicate with their respective communication centers and request that they phone their counterpart agency communication center and have them relay a message to their responding mobile unit.

Coordinating traffic incident management is increasingly recognized as an important tool to alleviate incident related congestion on the Capital Beltway and the surrounding road network. Effective incident management requires coordination and information sharing among multiple responders including: law enforcement, fire and rescue, emergency medical services, transportation agencies, motorist assistance services, information service providers, and the media. The current scenario of fragmented and indirect communication takes time and adds unnecessary delay in situations where every second counts.

2.2  Role of CapWIN

The Cap-WIN project is a partnership between the States of Maryland and Virginia and the District of Columbia to develop an integrated transportation and public safety information wireless network.  This unique project will integrate transportation and public safety data and voice communication systems in two states and the District of Columbia and will be the first multi-state transportation and public safety integrated wireless network in the United States.  The purpose of the CapWIN project is to greatly enhance incident response communications by integrating transportation and public safety data and voice systems in two states (Virginia and Maryland) and the District of Columbia, effectively creating the first multi-state, inter- jurisdictional transportation and public safety integrated wireless network in the United States. The project will have national implications in technology transfer concerning integrated transportation and public safety applications, including wireless image/video transmission.  The progress of CapWIN is being tracked at a national level as it has the potential to provide a roadmap for implementing similar networks throughout the United States and other countries. 

2.3  Project Stakeholders

The CapWIN project is being led by an Executive Group and Steering Group representing a partnership of transportation and public safety agencies (at all levels – state, local, and Federal), elected officials, and the Metropolitan Washington Council of Governments.  Project sponsors include the Maryland State Highway Administration, Virginia Department of Transportation, Maryland State Police, Virginia State Police, Washington Metropolitan Police, National Institute of Justice – Office of Science and Technology, Public Safety Wireless Network (Department of Justice/Treasury), and the U.S. Department of Transportation’s Federal Highway Administration.

The University of Maryland Center for Advanced Transportation Technology (UMD-CATT) is providing overall project management and staff support to CapWIN, with assistance from the University of Virginia (UVA), the International Association of Chiefs of Police (IACP), and the Washington-Baltimore High Intensity Drug Traffic Area (HIDTA) Research Program.   The UMD-CATT will have the lead responsibility to oversee the work of the contractor on behalf of the project Executive and Steering Groups.  In addition, the contractor shall act as the prime contractor to the UMD-CATT lead point-of-contact and designated staff during the development, deployment, and operation of the CapWIN infrastructure.  The City of Alexandria Police Department is allowing the use of their message switch and mobile client software for the purpose of the demonstration.  Other participants in the pilot project include the Virginia State Police, the Maryland State Police, the US Park Police, Prince George’s County Police and Fire Departments, the Maryland State Highway Administration, the City of Alexandria Fire Department, the Virginia Department of Transportation, and the Washington Metropolitan Police.

2.4  Project Schedule

Implementation of CapWIN will involve three one-year phases.  Contract award is expected by the end of March 2002, with implementation targeted to begin in April 2002.  The selected vendor will be responsible for implementing the following tasks under each phase of the project.

Phase One: Development and implementation of initial CapWIN infrastructure including a mobile data system and a message gateway to connect disparate systems.  Phase One consists of three tasks, all to be completed within one year of project start-up. 

Phase Two: Addition of priority functionality, expansion of interfaces, system operations and maintenance, with completion required within two years of contract award.  Potential functionality and interfaces to be added during Phase Two includes, but is not limited to the following:

Phase Three: Addition of priority functionality, expansion of interfaces, system operations and maintenance.  Completion of Phase Three is required within three years of contract award.  Potential functionality and interfaces to be added during Phase Three includes, but is not limited to the following:

3.0 Proposed Evaluation Goals and Objectives

Four goals have been identified for the CapWIN evaluation:

Goal #1: Assess the Level of Customer Satisfaction with the CapWIN System.

This goal is intended to evaluate how CapWIN end-users feel the system enables them to improve their job performance and resolve existing communications problems.  The key issue to be examined will be an assessment of the level of CapWIN user-acceptance.  In particular, this assessment will determine whether or not end-users view CapWIN as a system that they will use and support, and whether or not they perceive CapWIN as providing a significant benefit that justifies the cost of system development, deployment, and maintenance.  Absent a significant level of user-acceptance, the CapWIN system runs the risk of not being utilized as planned. 

End-users to be surveyed for this goal will be primarily transportation personnel involved with incident response and traffic management in the Washington Metropolitan region.  For example, personnel from the VDOT TMC located in Northern Virginia, MD SHA’s CHART system, and Montgomery County’s TMC will be included in the survey.  Within each TMC, emergency response personnel assigned to highway operations, supervisory personnel, and TMC operators responsible for incident identification, notification, and coordinating response activities with other jurisdictions will be included in the survey population.  This proposed personnel survey population will enable the evaluation team to obtain information on CapWIN performance from a wide range of end-users.

The evaluation team recognizes, however, that a potential benefit of CapWIN will be improved communications with law enforcement, fire and rescue, hazardous materials response units, and other agencies involved with incident response.  The evaluation team recommends that a sample of personnel from these agencies be included in the survey population as a means of documenting how the use of CapWIN improves inter-agency communication, and how this improved communications capability can further improve incident response and management activities.

Specific issues to be examined will include:

Data will be collected through an end-user survey.  Once surveys are returned and the data are analyzed, a subset of end-users may be selected for interviews to obtain additional information on key findings, if necessary.  The evaluation team anticipates that the survey will be developed in such a way as to enable quantitative analysis of findings.  The follow-up interviews will be used to develop qualitative findings and also to identify potential cost savings resulting from the use of CapWIN.

The objectives, hypotheses, measures of effectiveness, data sources, requirements, and analyses developed to support Evaluation Goal #1 are presented in Table 3.1.

Table 3.1 Objectives, Hypotheses, MOEs, and Data Components Supporting Goal #1

Objective

Hypothesis

Measures of Effectiveness

Data Sources and Requirements

Data Analysis

To determine end-user acceptance of CapWIN

CAPWIN will be positively received by end-users

End-user perceptions on the user-friendliness of CapWIN

End-user survey

Follow-up end-user interviews (selective only to expand and clarify survey findings)

Qualitative analysis of survey responses

To determine if end-users believe the use of CapWIN improves communications

CapWIN will improve intra- and inter- agency and inter-jurisdiction communications

End-user perceptions of improved communications capabilities provided by CapWIN

End-user survey

Follow-up end-user interviews (selective only to expand and clarify survey findings)

Qualitative analysis of survey responses

To determine if end-users believe the CapWIN system is reliable

The performance of the CapWIN system will be reliable

End-users will be able to access CapWIN without problem

End-user perceptions of reliability, timeliness, and effectiveness of system

End-user survey

Follow-up end-user interviews (selective only to expand and clarify survey findings)

Qualitative analysis of survey responses

Goal #2: Determine the Impact of CapWIN on Mobility During Incident Conditions

This goal is intended to identify and evaluate the impact of CapWIN on improving system mobility during incidents by reducing incident response and clearance times.  The evaluation team does recognize that overall system mobility is contingent upon many factors (weather, time of day, number and type of incidents) that are beyond the control of the CapWIN project.  Therefore, through the evaluation activities supporting this goal, the evaluation team will develop and assess measures of effectiveness that identify CapWIN-specific mobility related impacts.

In addition, the evaluation team believes that the most effective means of identifying CapWIN benefits will be through a “before” and “after” comparison of how the use of CapWIN improves the ability of emergency response personnel to respond to a particular incident.  The evaluation team recommends that the CapWIN baseline data collection be concentrated on developing process flows for a series of incident response scenarios that incorporate the diversity of traffic conditions in the Washington Metropolitan Area.  These scenarios would include geographic and jurisdictional considerations (i.e., incidents at Potomac River Crossings on I-495 or I-395), time of day (rush-hour vs. non-rush hour), and type of incident (hazardous material response, commercial vehicle accident, passenger vehicle accident).  Using these types of scenarios to establish a baseline and to facilitate the “before” and “after” comparison will enable the evaluation team to identify CapWIN-related benefits that address not only transportation specific activities but also improved inter-agency and inter-jurisdictional coordination of emergency response activities.  This will enable the evaluation team to consider the full range of potential CapWIN-related transportation benefits.

Data will be collected on two objectives in support of this goal.  The first objective will measure reductions in incident notification and response times.  The second objective will measure any reduction in incident duration.  The intent of these objectives is to determine if the use of CapWIN will enable the more timely deployment of incident response assets and reduce the time needed to clear an incident.  Reductions related to these objectives will serve as an indication of how CapWIN might improve mobility during incidents. 

Agency communication logs (telephone, dispatch, computer, radio) will be reviewed to determine changes in the number and duration of incident response-related messages.  In addition, post-incident reports will be reviewed, and questions related to this goal will be included on the survey to be developed in support of Goal #1.  As needed, questions related to this goal will be included in the list of questions developed for any follow-up interviews.  To the extent feasible, quantitative data will be collected and analyzed.

The objectives, hypotheses, measures of effectiveness, data sources, requirements, and analyses developed in support of Evaluation Goal #2 are presented in Table 3.2.

Table 3.2 Objectives, Hypotheses, MOEs, and Data Components Supporting Goal #2

Objective

Hypothesis

Measures of Effectiveness

Data Sources and Requirements

Data Analysis

To determine if CapWIN reduces the time needed to respond to an incident

CapWIN will enable more timely notification of emergency response personnel and deployment of assets

Agencies will revise response procedures to incorporate the use of CapWIN

Reduction in number of communications (radio, telephone, computer) messages needed to coordinate incident response activities

Reduction in time needed to deploy incident response personnel and assets

Survey of response personnel

Select follow-up interviews

Agency communications logs (telephone, computer, radio message)

Post-incident reports

Comparison of before and after communications data (number and duration of messages)

Comparison of before and after incident response times

Analysis of survey results

To determine if CapWIN reduces the overall impact of an incident

CapWIN will reduce the duration of an incident

CapWIN will reduce traffic delays caused by an incident

Reduction in time needed to clear an incident

Improved management of incident impact on traffic flows resulting from improved communications capabilities:

-reductions in lane closures (time and number)

-More timely clearance of incidents from the roadway

Survey of response personnel

Select follow-up interviews

Agency communications logs (telephone, computer, radio message)

Post-incident reports

Comparison of before and after communications data (number and duration of messages)

Comparison of before and after incident response times

Analysis of survey results

Goal #3: Determine Impact of CAPWIN I on Safety of Response Personnel and Frequency of Secondary Crashes

This goal is intended to assess the impact of CapWIN on improved safety for response personnel and also the impact on secondary crashes.  Incident response personnel are often required to spend extended periods of time at the roadside while responding to an incident.  This wait time may involve waiting for additional assets to arrive (i.e., a tow truck) or for traffic management personnel to respond (i.e., law enforcement personnel arriving on site to direct traffic).  Extended time at the roadside, in particular on an interstate highway during rush hour, increases the possibility that response personnel may be involved in an accident.  In addition, the longer the duration of an incident, the more likely traffic is to back up, creating congestion and resulting in slowdowns due to drivers “rubbernecking.” Both of these factors can contribute to secondary accidents. 

The evaluation goal will test the assumptions that the more efficient deployment of assets will reduce the time spent by response personnel at roadside and will also reduce the duration of incidents.  The premise of this goal is that reductions in response time and incident duration will in turn reduce the probability and frequency of secondary crashes and accidents.  To the extent feasible, quantitative data supporting this analysis will be collected.

The evaluation team will work to collect data on two objectives to support Evaluation Goal #3 designed to test both assumptions.  Data will be collected through an examination of agency safety records, incident records, and post-incident reports on secondary crash rates and respondent safety.  To the extent feasible, these data will be analyzed and presented in a quantitative format.  In addition, interviews will be conducted with responds personnel to obtain their perceptions on safety, secondary crashes, and other relevant data.  This qualitative assessment will be used to supplement the quantitative analyses.

The objectives, hypotheses, measures of effectiveness, data sources, requirements, and analyses developed in support of Evaluation Goal #3 are presented in Table 3.3.

Table 3.3 Objectives, Hypotheses, MOEs, and Data Components Supporting Goal #3

Objective

Hypothesis

Measures of Effectiveness

Data Sources and Requirements

Data Analysis

To determine if CAPWIN reduces secondary crashes during incidents

CAPWIN will reduce incident response times, thereby reducing secondary crashes

Reductions in secondary crashes

Reductions in incident response time and duration

Post-incident reports

Agency safety records

Survey of emergency response personnel

Follow-up interviews with select emergency response personnel (as determined by review of survey results)

Compare before and after records to identify any incremental changes

Qualitative analysis of survey and interview responses

To determine if CAPWIN enhances the safety of response personnel

CAPWIN will reduce incident response time, thereby reducing exposure of response personnel to potentially hazardous roadside conditions

Reductions in injuries

Reductions in time that response personnel are at the scene of an incident

Post-incident reports

Agency safety records

Survey of emergency response personnel

Follow-up interviews with select emergency response personnel (as determined by review of survey results)

Compare before and after records to identify any incremental changes

Qualitative analysis of survey and interview responses

Goal #4: Assess the Cost Savings of CapWIN in Terms of Incident Response and Management 

This evaluation goal is intended to measure any cost savings resulting from the use of CapWIN.  The expectation is that improved communications, improved coordination of incident management activities, and reduced time needed to respond to and clear an incident will reduce overall incident response and management costs.  One objective has been developed in support of this goal to determine any cost savings resulting from the use of CapWIN.  The evaluation team recognizes that the data sources and requirements presented are comprehensive, and that obtaining reliable data may be problematic.  To this end, the evaluation team recommends that both quantitative and qualitative data be collected in support of this goal.

The objectives, hypotheses, measures of effectiveness, data sources, requirements, and analyses developed in support of Evaluation Goal #4 are presented in Table 3.5.

Table 3.5 Objectives, Hypotheses, MOEs, and Data Components Supporting Goal #5

Objective

Hypothesis

Measures of Effectiveness

Data Sources and Requirements

Data Analysis

To determine if the use of CapWIN reduces the cost of incident response activities

Through more efficient coordination of incident response capabilities, the cost of incident response activities will decrease

Reductions in the amount of time spent by response personnel in incident response

Reductions in the amount of time vehicles are in use for incident response

Reductions in the deployment of unnecessary equipment due to improved communications

Agency time sheets and overtime records

Post-incident reports

Survey of response personnel

Follow-up interviews, as needed and on a select basis, with response personnel

Vehicle performance and use logs

Before and after analysis of time sheets, post-incident reports to identify any reduction in personnel hours used to respond to incidents

Analysis of before and after vehicle logs and post incident reports to identify any reduction in vehicle use to respond to incidents

4.0 Proposed Evaluation Management Plan

The SAIC team responsible for completing the CapWIN Phase II evaluation task is shown in Table 4.1.

Table 4.1 Proposed Staffing Plan

Individual

Position

Mark Carter

Program Manager

Nicholas Owens

Principal Investigator

Kelley Pecheux

Senior Transportation Engineer

Stephanie Kullman

Administrative Support

As Program Manager, Mark Carter will be responsible for ensuring that the evaluation is completed on schedule and within budget.  He will also be responsible for ensuring that the evaluation meets the Statement of Work objectives, and that all tasks are completed as specified.

Nicholas Owens will be responsible for day-to-day management of the evaluation.  He will be responsible for ensuring that the tasks specified in the Evaluation Strategy and the individual test plans are completed as specified.  He will be responsible for completing the Phase II Report, including the collection and analysis of “before” data, and will be responsible for completing quarterly, monthly, and weekly reports.

Kelley Pecheux will be responsible for assisting with test plan development, data collection and analysis, and preparation of the Phase II report.  She will be primarily responsible for analyzing the technical and system impact components of the evaluation.

Stephanie Kullman will provide all administrative support to the evaluation team.  She will also be responsible for preparing periodic cost analyses of the evaluation.  These analyses will be used by the evaluation team to track expenditures with evaluation progress to ensure that all tasks are completed within budget.  These analyses will also be used to identify any need to reallocate resources to ensure that the evaluation remains on schedule and within budget.

5.0 Proposed Evaluation Schedule

The proposed schedule for completing the CapWIN Phase II evaluation report is shown in Table 5.1. 

Table 5.1 Proposed Phase II Schedule

Table 5.1 Proposed phase II schedule

Based on the current CapWIN project schedule, the collection of “after” data and the evaluation impact analysis will not be completed until 2005.  The evaluation team therefore recommends that the final Phase II report include an assessment of the feasibility and benefit of an interim evaluation based on the results of CapWIN Phase I implementation.  While Phase I will include the use of approximately 30 CapWIN units, it may be possible to assess CapWIN benefits qualitatively. 

The CapWIN project received a significant increase in budget resources following the events of 9/11. CapWIN has the potential to be a national model in improving incident response and management activities and highway safety through the use of improved communications and data exchange capabilities, and improved inter-agency and inter-jurisdictional coordination of emergency response and management activities.  An interim report based on Phase I implementation results may provide other jurisdictions with both a “road-map” on issues to consider in developing and deploying a CapWIN-type system, and also the benefit/analysis information needed to justify the necessary expenditures for deploying and operating a CapWIN-type system.