Key Informant Interviews Test Plan Model Deployment of a Regional, Multi-Modal 511 Traveler Information System Task Order BA7746

Quality Assurance Statement

The Federal Highway Administration provides high-quality information to serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement.

Table of Contents

1.0 INTRODUCTION AND BACKGROUND.......................................................... 1

1.1 511 Model Deployment.................................................................................. 1

1.2 Evaluation Plan Overview................................................................................ 2

1.3 Report Organization........................................................................................ 3

2.0 TEST PLAN OBJECTIVE..................................................................................... 5

2.1 Hypotheses..................................................................................................... 5

2.2 Key Supporting Conditions............................................................................. 6

3.0 TEST DATA........................................................................................................... 7

3.1 Data Requirements.......................................................................................... 7

3.2 Data Sources.................................................................................................. 7

3.3 Data Collection Methods................................................................................. 9

3.3.1 Grouping Interviews.............................................................................. 9

3.3.2 Interview Questionnaire....................................................................... 10

4.0 ANALYSIS METHODS...................................................................................... 17

5.0 TEST SCHEDULE............................................................................................... 18

6.0 RESULTS REPORT FORMAT AND CONTENTS......................................... 19

7.0 ESTIMATED RESOURCE REQUIREMENTS............................................... 20

List of Tables

Table 1: Evaluation Tests...................................................................................................... 3

Table 2: Hypotheses Related to Other (Non-Management and Deployment Issues)

Project Objectives.................................................................................................. 6

Table 3: Key Informant Interview Subjects............................................................................ 8

Table 4: Interview Sessions................................................................................................. 10

Table 5: Round 1 Interview Questionnaire........................................................................... 11

Table 6: Potential Issues List by 511 Enhancement.............................................................. 12

Table 7: Round 2 Interview Questionnaire........................................................................... 15

Table 8: Estimated Resource Requirements......................................................................... 20

1.0 Introduction and Background

This document presents the detailed plan to conduct the Key Informants Interviews Test, one of several test activities to be conducted as part of the national evaluation of the regional, multi-modal 511 Traveler Information System Model Deployment. The model deployment is an enhancement of an existing statewide 511 telephone traveler information system operated by the Arizona Department of Transportation. As a United States Department of Transportation (U.S. DOT) Intelligent Transportation System (ITS) Model Deployment, the project is intended to generate findings that will help shape U.S. DOT approaches to 511 and that will be of use to others implementing and operating 511 systems. The national evaluation is the primary mechanism for documenting the performance of the model deployment and the lessons learned.

1.1 511 Model Deployment

The Arizona Department of Transportation (ADOT) is leading the 511 Model Deployment in partnership with other transportation agencies in the Phoenix and Tucson regions, including local traffic jurisdictions, airports and public transit operators. The 511 Model Deployment implements a number of key enhancements to the existing statewide 511 system that became operational in March 2002, when ADOT converted their ten-digit system that had been operational for several years to 511. The enhancements to the system include:

· new content on arterial streets, airports, neighboring states (Utah), transit (major service disruptions and estimated arrival times), weather, and downtown Phoenix special events and parking;

· a complete redesign of the menu system, conversion from a keypad system to voice recognition, conversion from a highway route-based reporting to segment-based reporting; and

· enhanced 511 marketing.

Planning for the Model Deployment began in earnest in August 2002 when ADOT convened the 511 Task Force, composed of representatives of the various participating agencies. Planning and design continued through early 2003. Since that time, the focus has been on implementation, with a certain degree of final design occurring as part of the implementation process. The model deployment enhancements are planned to become operational in a phased manner, beginning in late November 2003 and continuing through early-mid 2004.

The current menu system contains only three options: “roadways”, “transit” and “comments” (callers can leave voicemail comments). No transit information is available directly on the current 511 system; all transit selections are routed to the Phoenix and Tucson transit agencies’ customer service lines. Under “roadways” callers can either input the route number of the roadway or, for local street information, the first three letters of a city name. However, the current approach to gathering city street information (which relies on local traffic jurisdictions to enter information) is not effective and essentially, no city information is available.

The roadway information available on the current 511 system is oriented largely toward intercity travelers. Much of the information that is available is for so-called “planned incidents”—non-real-time information entered by ADOT in advance describing roadway construction, route closures and restrictions. Information is provided by Interstate and State Highway routes, rather than route segments, and is reported by milepost. Therefore, someone interested in a quick report on commute conditions on I-10 in central Phoenix, located in the central part of Arizona, would have to listen through (or know enough to skip past) all the information for I-10 between California and Phoenix. The statewide system does cover major metropolitan areas like Phoenix but information of interest to commuters, like travel times and information on local streets, is not available. One important goal of the model deployment is to make the 511 system more useful for commuters in the two largest urbanized areas in Arizona: the Tucson region and the Phoenix region.

Little is known about current usage of the 511 system. The only statistics that are tracked are total hourly call volumes, which are aggregated to daily and monthly totals. That data indicates that prior to converting to the 511 number in March 2002, the usage of the ADOT traveler information telephone system ranged from about 4,000 calls per month upwards to about 20,000 calls during winter storms or during unusual events like major forest fires. After the conversion to 511, usage spiked from around 7,000 calls in January and February to about 20,000 calls per month in March. Since that time, usage has remained at that level or higher, ranging up to as high as 100,000 calls per month (in December 2003). No information on call origin (geographic or cell phone/wireline) is available.

The only interaction ADOT has had with 511 users has occurred as part of the development of the model deployment, when they convened focus groups in Phoenix, Tucson and Flagstaff. A total of 67 people participated in the six focus groups. Very few of the focus group participants had used 511 prior to being recruited, at which time they were asked to familiarize themselves with the system. Most focus group participants found the existing system at best only moderately useful (average ranking of 4.7 out of 10) but indicated that with improvements, they would expect to find the system very useful (8.8 out of 10). Desired improvements included many of the planned model deployment enhancements, including voice recognition; segment-based and sub-region-based reporting; more information on incidents; and “quick reports” highlighting conditions in specific geographic areas, similar to commercial radio traffic reports.

1.2 Evaluation Plan Overview

The overall plan for the national evaluation of the 511 model deployment is presented in the Evaluation Plan report. This section highlights the major elements of the evaluation.

The Statement of Work for the national evaluation identifies the following purposes:

Document how the model deployment was implemented, including system costs and how technical and institutional (especially cross-modal and interstate) issues were resolved.
Provide an independent review of the performance of the model deployment.
Deliver lessons learned to other 511 system development and deployment efforts.

The evaluation consists of three phases. Phase I consists of the evaluation planning activities, including developing the Evaluation Strategy, Evaluation Plan and Detailed Test Plans, and preliminary analysis of baseline data. Phase II is scheduled to conclude in early 2004. Phase II consists of the analysis of the implementation phase portion of the enhancement process, including conclusion of baseline data analysis and documentation of management and deployment issues. Phase II is scheduled to be completed in February 2004. Phase III consists of the analysis of post-enhancement system performance, concluding with the Evaluation Report that will present all findings. Phase III is scheduled to be completed in April 2005.

Evaluation analyses have been organized into individual “tests”, with each test focusing on a particular type of data. The tests and their relationship to the major focal points of national Intelligent Transportation System (of which 511 systems are an example) evaluations are shown in Table 1. In addition to these formal tests, a cost analysis will be performed and various other types of supporting data will be collected and utilized either to aid in the interpretation of test results or to identify management and deployment issues and lessons learned.

Table 1 Evaluation Tests
Analysis Area	Tests
Analysis Area	Usage Logs⁽¹⁾	User Survey	Key Informant Interviews
Customer Satisfaction	X	X
Mobility	X	X
Efficiency	X	X	X
Management and Deployment Issues			X

⁽¹⁾The analysis of system usage will include consideration of system data content (inputs from the various agencies that supply data to the system) and the reliability (e.g., system downtime) and availability (phone line utilization) of the system.

1.3 Report Organization

This report is the Test Plan for the Key Informant Interviews. Following this Introduction, this report contains the following sections:

· Test Plan Objective – including the hypotheses to be tested and key supporting conditions necessary to successfully complete the test.

· Test Data – a description of data requirements, data sources, and data collection methods.

· Analysis Methods

· Test Schedule

· Results Report Format and Contents

· Estimated Resource Requirements

2.0 Test Plan Objective

As indicated in Table 1 on page 3, the Key Informants Interviews test will serve two purposes. First, the interviews will be the primary mechanism for identifying various management and deployment issues associated with the model deployment (including lessons learned). Second, the interviews will help support the evaluation of several of the project partners’ objectives related to improving the efficiency of the 511 system operation.

In regard to addressing efficiency objectives, interviews with ADOT technical staff responsible for overseeing the operation of the Highway Conditions and Reporting System (HCRS) and the Voice Response Activated System (VRAS) servers will complement the system data in the assessment of system availability. Interviews with ADOT Traffic Operations Center (Phoenix headquarters) HCRS operators and supervisors will complement the system input data in the assessment of improvements in the capture of arterial street data. Interviews with HCRS operators and supervisors at Tucson area agencies (ADOT District Offices, City of Tucson traffic, airport and transit) will help assess the extent to which the model deployment is successful in making available more information on Tucson. Interviews with HCRS operators and supervisors will also help assess the effectiveness of increased training and changes in data entry and verification procedures.

2.1 Hypotheses

The primary purpose of key informant interviews is to identify deployment issues and their resolutions, and, thus, the Evaluation Plan does not identify any specific management and deployment-related hypotheses. Rather, the interviews will be guided by questions/topics and prompts which reflect the evaluation team’s expectations regarding potential issues of interest, and where appropriate they will help illuminate results found in data collected relative to hypotheses in other test plans. As summarized in Table 2, the key informant interviews are expected to provide information that will assist in the evaluation of several hypotheses related to certain project objectives. The far right column in Table 2 identifies (in bold) potential roles for key informant interview data and identifies data from other test plans that will support the hypothesis testing of specific model deployment objectives.

Table 2

Hypotheses Related to Other (Non-Management and Deployment Issues)

Project Objectives

National ITS Goal Area

Objective

Hypothesis to be Tested

Key Informant Interviews and

Other Data⁽¹⁾

Efficiency

Maintain acceptable system availability.

Improved VRAS server and HCRS server reliability will maintain acceptable system availability.

· Interviews with ADOT technical staff

· ADOT HCRS and VRAS system data

· ADOT 511 caller comment records

Increase the effectiveness of capturing arterial street incident data.

Having ADOT Traffic Operations Center Staff monitor law enforcement scanners will increase the amount of arterial street traffic information.

· Interviews with ADOT TOC HCRS operators and supervisors

· System input data

Providing Tucson area agencies the ability to enter information to 511 will increase the amount of Tucson area information.

· Interviews with Tucson area agency HCRS operators and supervisors

· System input data

Improve the quality of data entered into the 511 system.

Increased training of HCRS operators, refinement of landmarks and terminology used in HCRS, and enhanced on going monitoring of data quality will result in more effective data entry, with fewer inaccurate and duplicative entries.

· Interviews with HCRS operators and supervisors and staff responsible for developing and implementing training and revisions to manuals

· HCRS operators and supervisors’ manuals and training materials.

⁽¹⁾ Other data (other than key information interview data) is addressed in other test plans. These other data sources are shown here to illustrate how interview data will be used in conjunction with other data.

2.2 Key Supporting Conditions

The key supporting condition for successful execution of this test is the cooperation of the interviewees. They will need to devote the necessary time to participate in the interviews and to speak freely regarding their perspective on the model deployment.

3.0 Test Data

3.1 Data Requirements

Key requirements of the key informant interview data consist of the following:

· To be useful, the input from the interviewees should be direct and forthright, which requires that they feel reasonably at ease in sharing their perspectives on the model deployment.

· The input from the interviewees should conceptually address all of the various potential types of management and deployment issues relevant to the specific interviewee. That will require that a comprehensive list of potential questions and examples be prepared that will effectively probe for input across the full range of potential areas. Any given interviewee may not have input in all areas, but it will be important to structure the interviews so as to draw out all of the potential input.

· Interview data will consist of notes taken by the interviewer. The notes must be detailed and accurate.

3.2 Data Sources

Table 3 identifies the key informants that will be interviewed. The primary focus of most interview sessions will be on identifying management and deployment issues. However, the sessions that include the ADOT HCRS Supervisor, ADOT HCRS operators, City of Tucson HCRS entry personnel, and staff of PBS&J will include deeper discussion about the various data quality and data capture activities of the model deployment. That information will contribute to investigation of hypotheses related to improving the efficiency of the 511 system (see Table 2, above). Interview subjects include all of the major active participants in the model deployment, including those leading the design, implementation and operation (ADOT and their consultants) and an organization or individual associated with each new data source, e.g., airports, transit, etc.

Two rounds of interviews will be conducted for identification of management and deployment issues, the first focusing on implementation and the second focusing on operation. Information needed to support the testing of efficiency-related hypotheses will be collected during the management and deployment interviews.

Not all management and deployment issue interviewees will participate in both rounds of interviews. The first round of interviews will include only those organizations and individuals playing a major role in planning, designing and implementing the system, consisting of ADOT, their consultants, and Valley Metro (Phoenix transit). The second round of management and deployment interviews will include all of the organizations and individuals identified in Table 3.

Table 3 Key Informant Interview Subjects
Type of Interview/Organization	Interviewee	Role in the Model Deployment
Agencies
Federal Highway Administration	Alan Hansen Transportation Engineer	FHWA Project Liaison
Arizona Department of Transportation – Headquarters (Phoenix)	Tim Wolfe Assistant State Engineer	Project Manager
	Darrell Bingham Information Technology Section Manager	Project Technical Coordinator and Evaluation Point-of-Contact
	Donna Ayers Public Information Officer	Compiles caller comments from 511 voicemail box. Coordinates marketing activities.
	Linda Anestasi HCRS Supervisor	Overseeing revisions in HCRS operator procedures to improve data quality.
	To be identified HCRS Operators	Inputting information into HCRS.
Arizona Department of Transportation – Tucson District	To be identified	Inputting Tucson area freeway information into HCRS.
Valley Metro (Phoenix Transit)	Mike Nevarez	Providing transit major service disruption and estimated bus arrival time information to the 511 system.
Sky Harbor (Phoenix Airport)	To be identified	Providing airport construction and parking information to the 511 system.
City of Phoenix Traffic	Bob Steele	Providing information from Downtown Traffic Management System and participating in Arterial Street Travel Time enhancement.
City of Tucson Traffic	Richard Nassi Ron Platt	Inputting City of Tucson roadway information.
Pima County	Albert Letzkus	Inputting Pima County roadway information.
SunTran (Tucson Transit)	Mike Beel	Providing major service disruption information.
Tucson International Airport (Tucson Airport Authority)	Jill Merrick	Providing airport construction and parking information to the 511 system.
Maricopa County (REACT)	Barbara Hauser REACT Manager	REACT (arterial street incident response team) personnel are inputting information to the 511 system via PDA’s.
Utah Department of Transportation	Brian Chamberlin	Contributing information on Utah-Arizona border roadway conditions to the 511 system.
National Park Service (Grand Canyon)	Jim Tuck	Inputting Grand Canyon information.
Consultants/Vendors
OZ Engineering	Tomas Guerra 511 Project Manager	Performing all enhancements to HCRS.
Call Processing	Larry Atkins 511 Project Manager	Performing all enhancements to the 511 phone system (VRAS), including implementation of new menu system and voice recognition.
PBS&J	Eli Sherer Rick Schuman Pete Costello	Providing overall program management support (developing individual task work orders, conducting and documenting meetings, monitoring schedule and budget, etc.) and leading several tasks, including design of new menu system, data quality enhancements (HCRS operator procedures), and segment weather reports.
Kimley-Horn Associates	Pierre Pretorius 511 Project Manager	Developed Marketing Plan.
Kimley-Horn Associates	Lisa Burgess	Developed Marketing Plan.
PIPS Technology, Inc.	Tom Owen 511 Project Manager	Providing arterial street travel time estimates to the 511 system.

If it can be coordinated, after each of the two rounds of management and deployment issue interviews, the regular 511 Task Force meetings will be used as an opportunity to review and potentially verify and expand input of the individual interviewees. It is likely that the task force will hold a meeting at approximately the time of the two rounds of interviews, which represent key milestones, and so it is expected that this activity can occur.

3.3 Data Collection Methods

3.3.1 Grouping Interviews

Given the large number of individuals and organizations participating in the model deployment who are expected to provide useful input on management and deployment issues, and given limited evaluation resources, interview subjects have been grouped into sessions, as indicated in Table 4. In grouping interviewees, the potential for different individuals to either inhibit or stimulate information exchange if grouped together both have been considered and groupings were identified accordingly. For example, grouping Phoenix and Tucson transit agencies together will hopefully stimulate discussion of transit 511 issues in general. Table 4 indicates which interview sessions, and in some cases which interviewees within a given session, will be interviewed in-person and which will be interviewed via teleconference.

The shaded interview sessions are those that will occur in both the first (oriented around planning, design and implementation) and second rounds (operation and overall project) of management and deployment interviews. The non-shaded sessions, which include organizations and individuals involved primarily in the operational phase of the deployment, will occur only in the second round of interviews.

Table 4 Interview Sessions
Interview Session	Participants	Method – Location
1.	ADOT Headquarters - Management Kimley-Horn	In person – Phoenix
2.	Valley Metro SunTran	Valley Metro in person - Phoenix SunTran via teleconference
3.	ADOT Headquarters – HCRS Operators	In person - Phoenix
4.	ADOT Tucson District City of Tucson Traffic Pima County	In person – Tucson
5.	Sky Harbor International Airport Tucson International Airport	Sky Harbor in person - Phoenix Tucson via teleconference
6.	City of Phoenix Traffic Maricopa County	In person – Phoenix
7.	FHWA	Via teleconference
8.	National Park Service	Via teleconference
9.	Utah Department of Transportation	Via teleconference
10.