CHAPTER 4. EVALUATION GOALS AND APPROACH

This chapter describes the process by which the CVISN evaluation goals and associated hypotheses were developed and summarizes the technical approach that was undertaken. A complete discussion of the evaluation planning process, including additional information on the evaluation goals and approach is contained in the CVISN MDI Summary Evaluation Plan (July 1998). Additional details, including the designs of specific studies that were carried out in the areas of safety, costs, and customer satisfaction, are provided in Chapters 5, 6, and 7, respectively. The approach to benefit/cost analysis (BCA) is described in Chapter 8. Supporting information appears in the appendices (Volume II).

Purpose and Scope of the Evaluation

The purpose of the CVISN MDI was to demonstrate the technical and institutional feasibility, costs, and benefits of the primary ITS user services for commercial vehicle operations (CVO) and to encourage further deployment of these services. As required under the Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA), the evaluation of CVISN was undertaken to furnish information to U.S. DOT, Congress, states, public interest groups, and others on the desirability of making CVISN investments and corresponding enhancements to national, state, regional, and local transportation programs. In particular, it is important to document benefits associated with the national ITS/CVO goals and to learn as much as possible about how CVISN changes the way in which commercial vehicle operations are carried out. In addition, the results must permit comparisons between, and aid in developing priorities among, alternative investments within the U.S. DOT’s ITS program and between ITS and non‑ITS programs. For this reason, the evaluation included a comprehensive BCA to determine the economic worth of CVISN deployments. Thus, the expected outcomes of the evaluation projects are

1. Analyses and documentation of the outcomes and benefits of CVISN deployment that are of interest to various stakeholders

2. A rigorous BCA to determine the net economic benefits of CVISN deployment on a national level.

The measures for which data were needed, both for the analysis of outcomes and benefits and as input to the BCA, were established by considering the potential changes to the transportation system, identifying groups impacted by the changes, and obtaining their input on potential benefits and costs. Section 4.1 describes the process undertaken to develop the evaluation strategy and priorities, and Section 4.2 presents the specific objectives and methods within the four main study areas, safety, cost, customer satisfaction, and benefit‑cost analysis.

4.1 Evaluation Strategy and Priorities

To develop the evaluation strategy, the interests of stakeholders or customers had to be considered early in the planning process. First we identified the “customer” groups affected by the deployment of CVISN:

· Motor Carriers

· State governments

· Law enforcement agencies

· Shippers/receivers

· Members of the public

· Federal government.

In January 1997, the CVISN Planning and Evaluation Workshop was held at The Johns Hopkins University, with more than 100 participants representing law enforcement, IRP and IFTA credentials administrators from states and industry, and federal employees involved in CVO. The workshop consisted of break-out sessions focusing on CVISN deployment plans, expected changes to CVO, and potential benefits of CVISN. The benefits identified by the participants were grouped under the five ITS goal areas as follows:

Safety

· Fewer crashes involving trucks

· Increased personal safety of the motoring public

Efficiency (increased throughput or capacity)

· Increased throughput at inspection sites

· Increased throughput of credentialing process

Productivity (cost savings, revenue increases, increased output)

· Reduced time, cost, and uncertainty in credentialing

· Reduced cost of inspections

· Transit time reduced by bypassing inspection sites

· Transit time reduced by shorter stops at inspection sites

· Reduced accident costs

· Decreased tax and fee evasion

· More equitable treatment in paying taxes and fees

· Transit time decreased as a result of fewer crashes

· Reduced accident cleanup costs

Mobility

· Reduced cost of goods movement to shippers/receivers and the public

Decreased goods movement transit time and increased reliability of delivery schedules to/from shippers/receivers

· Increased cargo safety and security

· Reduced highway delays to public from fewer accidents

Energy/Environment

· Reduced energy consumption of trucks

· Reduced environmental impacts of trucks.

To help establish priorities, participants in the workshop were asked to rate the potential benefits according to their perceived importance. This was done after considering both the value of the benefits and the potential magnitude of the benefits. As shown in Figure 4-1, all of the groups participating in the workshop rated safety benefits the highest priority and efficiency benefits the second highest. Mobility, productivity, and energy/environment, in that order, were rated lower. Recall, however, that the relative importance of these benefits, as assigned by the participants, is inherently linked to their assessment of the potential for achieving these benefits.

Figure 4-1. Rating of Evaluation Goals by CVISN Workshop Participants

Figure 4-1. Rating of Evaluation Goals by CVISN Workshop Participants

According to the workshop participants, demonstrating safety benefits is the number one priority. The ultimate safety benefits of CVISN roadside enforcement systems are reduced numbers of truck‑related crashes, injuries, and fatalities. Efficiency, also cited as an important benefit during the workshop, is usually considered to be part of the safety and productivity benefits. Improved mobility and reduced energy and environmental costs were considered to be of relatively less importance.

The next step was to organize these benefits according to the features of CVISN that the states were planning to deploy. Key features and anticipated benefits of CVISN‑enhanced roadside enforcement (safety information exchange and electronic screening) and credentials administration are highlighted in Tables 4‑1 and 4‑2, respectively. Although electronic screening and safety information exchange consist of different technology systems, they are sometimes treated together (and referred to as “roadside enforcement”) because the systems work together to benefit roadside enforcement practices and to improve efficiency for safe and legal participating carriers.

Table 4-1. Key Features and Anticipated Benefits of CVISN Roadside Enforcement

Deployments

Key Features	Anticipated Benefits
Mainline screening with dedicated short‑range communication (DSRC) and weigh‑in‑motion (WIM) capability	Time and cost savings and increased customer satisfaction for registered carriers. Improved targeting of high-risk carriers.
Sorter lane screening at weigh stations using license plate reader (LPR), optical character recognition (OCR), automated vehicle identification (AVI), and/or low-speed WIM	Improved targeting of high-risk carriers through application of screening criteria on a broader population of trucks (i.e., of carriers not registered for mainline screening)
Mobile roadside enforcement units equipped with networked screening data	Improved targeting of high-risk carriers. Identification of and reduction in number of out-of-service (OOS) order violators
Timeliness of the screening data used in the inspection units (fixed or mobile)	Increased compliance with safety regulations. Improved targeting of high-risk carriers. Identification of and reduction in number of OOS order violators
Facilities for screening on bypass routes	Increased safety through identification of violators of safety regulations

Table 4-2. Key Features and Anticipated Benefits of CVISN Credentials Administration Deployments

Key Features	Anticipated Benefits
End-to-end electronic application and processing of credentials; includes electronic submittals, direct links to legacy (predecessor computer) systems for automated processing (i.e., edit checks, fee calculation, invoice generation), funds transfer, and production of credentials.	Time and cost savings and increased customer satisfaction for both carriers and states Fewer delays to carriers for obtaining credentials
Use of PC-based and web-based Carrier Automated Transaction (CAT) software to submit applications for credentials	Time and cost savings and increased satisfaction for both carriers and states Relative benefits of PC- and Web‑based CATs may depend on size of carrier.
Printing of permanent or temporary credentials in carrier offices—especially for trip‑related credentials; e.g., oversize/overweight (OS/OW)	Avoids delays in getting vehicle on the road
Interface with IRP and IFTA clearinghouses	Cost savings to states

The anticipated benefits can be grouped into three categories: safety, productivity (cost savings), and customer satisfaction. Efficiency, cited as an important benefit during the workshop, is usually considered to be part of the safety and productivity benefits. Customer satisfaction is an important area because (1) we need to understand the factors that will ensure a successful deployment, and (2) customer satisfaction is an important means of determining the value of other non-monetary benefits for the benefit-cost analysis.

4.2 Evaluation Study Areas—Objectives and Methods

Based on the types of benefits expected, the CVISN evaluation project was divided into four study areas: Safety, Costs, Customer Satisfaction, and Benefit/Cost Analysis. The objectives of each study area and summaries of the methods used to accomplish the objectives are presented below. Additional details can be found in Chapters 5 through 8 and the accompanying appendices (Volume II).

CVISN Safety Studies

The primary goal of the CVISN safety study was to identify and document the safety benefits of deploying CVISN technologies. The safety benefits are expected to include a reduction in the number of highway crashes involving trucks, the number of related injuries and fatalities, and the cost of property damage from these crashes. However, the particular CVISN technologies that are included in the model deployment initiative achieve these benefits only through improvements in carrier and driver compliance with safety regulations. Thus, the main focus of this study was on the relationship between CVISN deployment and its impact on enforcement practices. The relationship between enforcement practices and safety impacts (i.e., reduced crashes and fatalities) was established to link safety benefits to the deployment of CVISN services. Results from the literature, as well as new analyses, were used to help determine this relationship.

CVISN technologies are expected to help improve compliance with safety regulations in two ways, both resulting from increased effectiveness of roadside inspection operations. The direct, but smaller, impact is the removal of unsafe drivers and vehicles from the highways. It is anticipated that the screening and safety information exchange technologies will allow inspectors to select commercial vehicles for inspection rapidly, based on the carrier’s safety record. Also, on‑line access to driver violation records and results of recent truck inspections will help target unsafe drivers and commercial vehicles.

The indirect effect, which is expected to be much larger, is that drivers and carriers will modify their behavior in response to the improved, more targeted inspections. Specifically, it is assumed that carriers will expend resources to ensure that their vehicles stay in compliance. Carriers with good safety records (low risk) will have a small probability of being inspected. High‑risk carriers will try to improve their safety rating to avoid increased inspections. Of course, if CVISN does not help inspectors target the high‑risk carriers, there will not be any added incentive for a carrier to maintain a good safety rating.

The goals of the CVISN safety study were defined by four study questions:

1. What is the impact of CVISN on the numbers of crashes, injuries, and fatalities involving large commercial motor vehicles?

2. What is the impact of CVISN on rates of driver and carrier compliance with the FMCSR?

3. To what extent does CVISN help roadside safety enforcement officials identify high‑risk commercial vehicles and motor carriers?

4. To what extent does CVISN help roadside safety enforcement officials identify OOS violators?

The approach to addressing these questions and estimating safety benefits consisted of several elements. The first step was to develop a crash avoidance model. Because CVISN deployment has not advanced to the stage where the safety benefits can be measured directly (i.e., by comparing the numbers of crashes before and after deployment), it was necessary to develop a probability model that predicts the number of crashes, injuries, and fatalities under several different scenarios. Each scenario was defined by specific assumptions concerning the future deployment of CVISN.

The next step was to identify sources of data. Specifically, data were needed on the number of historical crashes and to estimate probabilities that a crash involving a large truck was caused by vehicle and driver OOS conditions. These data were obtained from the literature. Additional data were needed to determine the impact of CVISN technologies on inspection efficiencies and compliance rates. Therefore, studies were conducted in the CVISN Pilot states of Oregon, Connecticut, and Kentucky.

The two-part Oregon study (1) examined the effects of CVISN on carrier and driver compliance with the FMCSRs and (2) quantified the effectiveness of roadside enforcement staff at targeting vehicles from high-risk carriers with and without using the CVISN Inspection Selection System (ISS). Similarly, the Connecticut roadside study was conducted to estimate the effectiveness of Aspen/ISS to select high-risk vehicles for inspection. The Kentucky screening study was conducted to compare the inspection efficiency of stations with and without electronic transponder facilities that allow participating vehicles to bypass inspection stations. Further details on the approach and methods for these studies are presented in Section 5.2 and all three full study reports are included in Appendix A.

CVISN Cost Studies

The cost analysis considered three major cost-related questions:

1. What are/were the baseline costs associated with CVO processes prior to CVISN technology deployment?

2. What are the one-time start-up costs to the states to deploy CVISN systems, and what are the key drivers or major elements contributing to those costs?

3. What recurring (annual) capital and labor, operating, and maintenance costs do states incur as they use CVISN technologies, and what are the key drivers or major elements contributing to those costs?

In addition, two hypotheses were tested:

· CVISN credentialing systems will result in reduced time, costs, and uncertainties involved with handling (applying for and administering) commercial vehicle credentials for both state agencies and motor carriers

· The deployment of CVISN systems for roadside enforcement operations will result in capital cost increases to state agencies, but are expected to improve (1) inspection efficiency for states and motor carriers and (2) highway safety.

The CVISN cost study consisted of a literature review and on-site, in-person interviews. The primary references on sources of CVISN-related cost data were the National Governors’ Association (NGA) Study (Apogee 1997), American Trucking Associations (ATA) Foundation Study (1996), Maryland Benefit/Cost Study (Bapna, et al. 1998), and the Washington State CVISN Pilot Project report (1998). Most of these studies reported costs and cost savings, which were estimated by state and industry officials prior to the deployment of CVISN. Therefore, it was necessary to conduct on-site in-person interviews to obtain actual costs for this study. The objective of the interviews with state agencies was to collect information on

· Costs associated with the current credentialing processes and roadside screening and inspection activities

· Costs associated with deploying and operating various CVISN systems

· Resources (staff and equipment) committed to CVISN deployment.

Interviews were also conducted with representatives of selected motor carriers participating in the pilot testing of CVISN systems. The objectives of these interviews were to

· Gather information on the costs incurred (or savings realized) by the motor carrier industry resulting from CVISN systems deployment

· Learn about the impacts of CVISN systems on the efficiency and productivity of the motor carrier industry, as they affect costs.

All interviews were fact-finding, interactive discussions aimed at gaining an understanding and collecting data on the costs of CVISN systems. The interview guides were developed based on the evaluation strategy, evaluation data requirements plan, experiences from similar studies (e.g., NGA), and information gathered from the literature review.

Cost information was drawn mainly from interviews with state transportation officials from four states: Kentucky, Maryland, Connecticut, and Virginia. Additional information was obtained from field operational tests involving states in the I-95 Corridor Coalition CVO Working Group and a special cost study involving states participating in the IRP Clearinghouse project.

CVISN Customer Satisfaction Studies

To measure customer satisfaction with CVISN, several surveys and other measures were planned and carried out. These included a national motor carrier survey, a driver survey, and surveys and focus groups involving state inspectors and law enforcement personnel. Originally, a separate motor carrier survey was planned, to quantify the benefits of electronic credentialing for motor carriers. However, at the time this study was scheduled, there were not enough carriers with experience in electronic credentialing to constitute a valid study population.

Table 4-3 shows the customer groups who were surveyed to determine their experiences in using CVISN technologies and their satisfaction with those technologies. Shippers/receivers and the general public are also recognized as stakeholders in CVO. Benefits to each are discussed indirectly in Chapter 8 on benefit/cost analysis, but direct measures of the satisfaction of these customer groups were beyond the scope of this evaluation.

Table 4-3. General Topics Covered in Surveys and Other Evaluations of Customer Satisfaction

Customer Group	Survey Topics Relating to
Customer Group	Electronic Credentialing	Roadside Inspections/Enforcement
Motor Carriers	· Experience with credentialing · Current credentialing procedures · Awareness and use of electronic credentialing · Opinions about electronic credentialing · Likelihood of using electronic credentialing	· Experience with roadside inspections · Current inspection procedures · Awareness of electronic screening · Opinions about electronic screening
Drivers	· Likelihood of owner-operators to enroll in electronic credentialing	· Opinions about roadside enforcement · Likelihood of owner-operators to enroll in electronic credentialing
State CVO Administrators	· Institutional issues and benefits	· Institutional issues and benefits
State CVO Inspectors		· Inspection systems in use · Satisfaction with equipment · Perceived benefits

The purpose of the CVISN Motor Carrier Survey was to

· Collect baseline information concerning the relevant behaviors, awareness, and attitudes of motor carriers, and

· Identify the incentives and barriers to more widespread deployment of CVISN‑type initiatives across the country.

A mail survey of carriers was designed to be representative of the trucking industry throughout the contiguous 48 states. The sample was a stratified random one, selected from the mid‑1999 records of firms in the federal government’s MCMIS Census database. Stratification was used both to compensate for the highly skewed distribution of firms in the industry by size, and to place emphasis on firms operating in the states where CVISN deployment was most highly advanced. Compared to a simple random sample of all firms, the stratified sample contained much higher proportions of larger firms and ones with registered home addresses in five “CVISN focus” states: Connecticut, Kentucky, Maryland, Oregon, and Virginia. A total of 158 complete responses were obtained from approximately 2,000 firms that were mailed questionnaires. Sampling weights were established for each sample stratum to restore correct proportionality of the achieved sample by firm size and by geography.

A qualitative driver survey was conducted to explore the opinions of truck drivers about recent, CVISN-related changes in roadside inspection methods, and the opinions of owner‑operators about electronic credentialing. The operators’ inputs are intended to help color, interpret, and better understand the information gathered in complementary evaluation activities.

One-on-one interviews were conducted with 61 truck drivers intercepted at large rest/refueling stops located adjacent to major truck routes in Connecticut and Kentucky. These two states are ones that have implemented significant electronic credentialing initiatives, and have been the focus of other, complementary evaluation activities. The interviews took place at four locations, two in each of the states, in late November and early December 2000.

Sample quotas were set to ensure the representation of owner-operators and of drivers employed by firms of varying sizes. Using in-depth, semi-structured personal interviews, all of the respondents were asked about roadside safety and weight inspections. The owner‑operators were also asked about electronic credentialing methods.

The primary objective was to identify the range of opinions on various aspects of these topics, and to form hypotheses about any apparent areas of consensus or disagreement. Because of the small size of the sample and the method of sample recruitment, the degree to which the people interviewed are representative of any group larger than themselves cannot be determined.

Methods for evaluating of the satisfaction of state CVO administrators with CVISN technologies were less formal than the measures used to gauge the satisfaction of motor carriers and commercial vehicle drivers. Evaluation contractor staff participated in many meetings, conferences, and other forums, where the attitudes of state administrators and other CVISN stakeholders were directly solicited and discussed in detail. These include various pilot/prototype state workshops at The Johns Hopkins University, CVISN MDI Program Managers Meetings, and CVISN Deployment Forums and Mainstreaming Conferences. At each of these forums, participants were encouraged to offer opinions on successes, failures, obstacles, lessons learned, and issues to be resolved. Information gathered from these kinds of meetings was taken into account in all phases of evaluating the CVISN MDI.

The attitudes and opinions of state CVO inspectors regarding the use of CVISN roadside enforcement technologies were addressed through focus groups and a formal survey conducted in collaboration with the evaluation of the I-95 Corridor Coalition and SAFER Data Mailbox FOTs (Battelle 2000). Over 50 inspectors from six eastern states (Connecticut, Maryland, New Jersey, New York, Pennsylvania, and Rhode Island) participated in the focus groups, and approximately 370 inspectors from these states completed formal questionnaires. Topics included background information, system usage, satisfaction, and perceived benefits.

CVISN Benefit-Cost Analysis

A comprehensive benefit/cost analysis (BCA) was carried out for the CVISN MDI. Benefit/cost analysis is a public sector evaluation tool that compares all of a project’s benefits to society to all of the project’s costs to society. The question to be answered in a BCA is: Do these benefits exceed the costs? If the answer is yes, the benefit/cost ratio (BCR) is greater than one, and the project is said to be economically “feasible” or economically “justified.” Commercial feasibility, the analogous private sector criterion, is much narrower in the benefits and costs it compares. Benefits are restricted to commercial revenue, and costs are limited only to those paid directly by the project developer.

In the case of CVISN, considerable public benefits can be expected. However, revenue paid by CVISN users is essentially zero, because CVISN is intended to make a regulatory system operate at lower cost and increased effectiveness to both its users and to society. The benefits quantified for inclusion in this BCA do not include every conceivable public benefit of CVISN, but they do include the major categories of benefits, such as crashes avoided, transit time savings, operating cost savings to states from electronic credentials administration, and savings to carriers from more efficient inspections and credentialing. The corresponding costs included in the BCA cover capital equipment purchases and periodic replacements/upgrades, software development, and increases in operating costs to states and carriers for roadside enforcement, for example. Costs are analyzed over a hypothetical 25-year life cycle beginning in 2000, using appropriate discount rates to achieve constant dollar estimates.

A literature search was conducted to monetize or determine cash values for any costs or benefits unavailable in the data collected from the participating states. For example, the prevailing cost of a crash was determined, as were cost or benefit values for the time vehicles spend in transit or in weigh station queues, and the inventory cost to a motor carrier for every day of delay in obtaining credentials for a new vehicle. A supplemental analysis was done on potential cost savings from reduced pavement damage, assuming that improved roadside enforcement leads to fewer overweight vehicles on the highways. This was a preliminary, qualitative assessment to obtain rough estimates to highlight the importance of this benefit. However, the results have not been included in the BCA in this report. A more rigorous study would be required to verify the assumptions that were made. Further details on the approach and methods for the BCA are presented in Section 8 and in Appendix D.

4.3 References

American Trucking Associations (ATA) Foundation, “Assessment of Intelligent Transportation Systems/Commercial Vehicle Operations Users Services: ITS/CVO Qualitative Benefit/Cost Analysis¾Executive Summary,” Report to FHWA, Alexandria, Virginia, June 1996

Apogee Research, Inc., “Budgetary Implications of ITS/CVO for State Agencies,” Report EDL 10125 to the U.S. Department of Transportation, FHWA [directed by National Governor’s Association], November 1997, available at http://www.itsdocs.fhwa.dot.gov/jpodocs.

Bapna, S., Zaveri, J., and Farkas, Z.A., “Benefit-Cost Assessment of the Commercial Vehicle Information Systems and Networks (CVISN) in Maryland,” National Transportation Center/Morgan State University, Report EDL 9369 to U.S. Department of Transportation Research and Special Programs Administration, November 1998.

Battelle, “Evaluation of the I-95 Commercial Vehicle Operations Roadside Safety and SAFER Data Mailbox Field Operational Tests,” Draft Evaluation Report to ITS Joint Program Office, U.S. Department of Transportation, June 29, 2000.

Battelle, “CVISN Model Deployment Initiative Draft Summary Evaluation Plan,” report to ITS Joint Program Office, U.S. Department of Transportation, July 1998.

ISTEA (Intermodal Surface Transportation Efficiency Act of 1991). Public Law 102-240, December 18, 1991.

VNTSC (John A. Volpe National Transportation Systems Center), “OMCHS Safety Program Performance Measures: Assessment of Initial Models and Plans for Second Generation Models,” Federal Highway Administration, Office of Motor Carrier & Highway Safety, U.S. Department of Transportation, May 28, 1999.

VNTSC (Volpe National Transportation Systems Center), “An Effectiveness Analysis of SafeStat (Motor Carrier Safety Status Measurement System),” by D.G. Madsen and D.G. Wright, Paper to U.S. Department of Transportation, No. 990448, November 1998.

Washington State Patrol, “Information Technology Feasibility Study for the Washington State CVISN Pilot Project,” Report to DOT FHWA, DOT EDL #9363, January 8, 1998.