Evalulation of the I-95 Commercial Vehicle Operations Roadside Safety and SAFER Data Mailbox Field Operational Tests

EXECUTIVE SUMMARY

Background

In 1998, the United States experienced nearly 400,000 crashes involving large trucks, resulting in approximately 5,000 deaths. Although new research (e.g., the Large Truck Crash Causation Project) is being planned by the Federal Motor Carrier Safety Administration (FMCSA) to better understand the causes of these crashes, vehicle safety defects and driver violations of the Federal Motor Carrier Safety Regulations (FMCSR) are known to contribute to some portion of these crashes. Recent studies indicate that approximately 10 percent of all large truck crashes, and the resulting lives lost, could have been avoided if all trucks and drivers were in compliance with safety regulations.

The Federal Motor Carrier Safety Administration (FMCSA) was established within the Department of Transportation on January 1, 2000. Formerly a part of the Federal Highway Administration, the FMCSA has as its primary mission the prevention of commercial motor vehicle‑related fatalities and injuries. Administration activities contribute to ensuring safety in motor carrier operations through strong enforcement of safety regulations, targeting high‑risk carriers and commercial motor vehicle drivers; improving safety information systems and commercial motor vehicle technologies; strengthening commercial motor vehicle equipment and operating standards; and increasing safety awareness. To accomplish these activities, the Administration works with Federal, state, and local enforcement agencies, the motor carrier industry, labor safety interest groups, and others. In 1999, the Agency announced as one of its specific goals to reduce commercial truck-related injuries and fatalities by 50 percent before 2010.

Over the past few years new technologies, including computer software and database and communication systems, have been developed to assist in the enforcement of motor carrier safety regulations. These systems, which are now being tested by state enforcement agencies, have significant potential for improving highway safety.

This document summarizes the results of the evaluations of two field operational tests (FOTs) of innovative technologies for deployment and exchange of roadside safety information:

• The Safety and Fitness Electronic Record (SAFER) Data Mailbox (SDM) system, a real-time data exchange system that enables roadside enforcement staff to submit commercial vehicle inspection results to a centralized database (SAFER) and to obtain prior inspection reports from other locations, including out of state, in order to identify carriers violating out-of-service (OOS) orders.

• I-95 Corridor Coalition’s Field Operational Test (FOT) 7, which tested a wide range of inspection procedures and technologies used by roadside enforcement personnel to target high-risk carriers.

In 1997, the states of Delaware, Maryland, New Jersey, New York, Pennsylvania, and Virginia agreed to participate in a FOT to evaluate the performance, costs, and benefits of SDM. Funding for the SDM system was originally authorized by the U.S. Congress in the Department of Transportation and Related Agencies Appropriations Bill, 1996 (published August 4, 1995). Connecticut, a state with extensive experience using laptop computers in its commercial vehicle enforcement program, joined the project in early 1998. However, Connecticut’s participation in the SDM project was not funded through the Eastern States coalition grant. Because all of the state participants in the SDM project were members of the I-95 Corridor Coalition, it was agreed to continue the deployment of SDM under the I-95 FOT program. The Coalition’s FOT 7, which focused on roadside safety enforcement technologies, included six eastern states: Connecticut, Maryland, Massachusetts, New York, Pennsylvania, and Rhode Island.

The results of these projects are presented jointly in this report because of the overlap in states participating and the shared evaluation objectives. Many states combined funds from the SDM project and FOT 7 to accomplish the same objectives. In addition, Battelle, as evaluation contractor for both projects, as well as for the Commercial Vehicle Information System and Networks (CVISN) Model Deployment Initiative, coordinated evaluation data collection and analysis activities among these projects. While SDM focused more narrowly on technologies for identifying out-of-service order violators, there are common issues related to time, cost, and institutional concerns raised by all three projects, leading to closely related conclusions.

A challenge in evaluating SDM and FOT 7 was the variation among states in commercial vehicle enforcement practices and in the degree to which they have adopted safety information exchange technology. The uses of these technologies and related systems vary with both individual characteristics of the inspectors and with the characteristics of the administrative systems in which they operate. Chapter 2 of this report provides an overview of current commercial vehicle enforcement practices in the participating states.

Objectives

The evaluation goals for CVISN, the SDM project, and for FOT 7 were established by the member states. Specific hypotheses or study questions were developed to guide the analysis of each evaluation goal. While the goals for the two projects were not identical at the outset, they had common components which can be summarized as follows:

FOT 7, SDM, and CVISN Evaluation Goals

• Demonstrate the effectiveness of using current safety performance data to help identify high-risk carriers, drivers, and vehicles and to identify out-of-service (OOS) order violators during roadside enforcement,

• Evaluate the time, cost, and other impacts of electronic collection of roadside safety information for upload and dissemination to regional and national databases,

• Identify institutional issues and benefits related to the use of this technology, and

• Assess the effectiveness of public outreach programs for deterring OOS violations.

As detailed in the SDM and FOT 7 Evaluation Plans, a single set of tests, representing a variety of data collection and/or analysis efforts, were proposed to address the evaluation goals and hypotheses. Several of these tests were also designed to be used in the CVISN evaluation. The tests were modified as necessary to ensure that SDM and FOT 7 were addressed. Chapter 4 provides a summary of each test conducted, and Chapter 5 details the findings of these tests.

Findings

The findings from this study lead to several general conclusions and trends:

• Utilization of laptop computers with ASPEN software, including components of SAFER Data Mailbox, has increased steadily since the system became operational in 1997. Most eastern states are uploading inspection results to SAFER on a regular basis, but the time between completion of the inspection and uploading the report varies from state to state, depending on the type of communication technologies used.

• Inspectors report a general satisfaction with the ASPEN system, and report that laptop computers have become an integral part of conducting motor carrier inspections.

• Computer technology is seen as helping inspectors (a) gather more complete inspection information, (b) work more efficiently, and (c) save time compared with traditional paper-based inspection systems. Findings on actual time savings versus paper were equivocal. Some inspectors reported a net time savings, while others reported that computer-based systems required just as much time as paper‑based systems to conduct inspections at roadside or at weigh stations.

• Inspectors perceive that using more current and accurate inspection data, as provided by computer-based inspection technologies, helps them (a) target their inspection efforts better, (b) find recent out-of-service orders more readily, and (c) spot patterns in motor carrier violations more easily.

• Until electronic screening technologies are deployed and integrated with the Inspection Selection System (ISS), it is not practical to screen all trucks on the highway using ISS. However, it was demonstrated that inspection selection efficiency, measured in number of out of service orders per inspection, increased by about 2 percent when ISS is used in combination with manual pre-screening. Simulation results indicated that inspection selection efficiency will increase by 11 percent when ISS is integrated with electronic screening.

• Few violators of out-of-service orders have been identified using SAFER Data Mailbox. However, inspectors have found that past inspection results provide useful information for detecting current violations.

• The full potential for SAFER Data Mailbox will not be realized until all states upload inspection results in a timely manner (i.e., in less than 2 hours). Greater potential is possible if the system is used in combination with electronic screening systems which automatically identify the vehicle at highway speeds.

• Inspectors responded most positively to the improved uniformity, legibility, and neatness of the computer-generated inspection reports.

• Roadside tests of the Inspection Selection System in Connecticut showed that computers offered a marginal advantage in helping inspectors target high-risk carriers for inspection over vehicles from other carriers in the general population.

• Costs for equipment were estimated to range from $7,500 to $9,175 per system, with itemized component costs as follows:

- Sierra Wireless MP210 $800 – 1,615

- Desktop PC plus internal modem $1,200 – 1,600

- Brayley box $2,300

- Laptop PC $3,000 – 3,360

- Printer $200 – 300

• Inspectors tended to speak more of immediate, day-to-day operational benefits of the computers than any perceived long-term, national benefits in highway safety resulting from the wider adoption of computer-based inspection technologies.

• Issues remaining to be resolved include

- The overlapping of government jurisdictions and responsibilities for purchasing equipment, maintaining systems, and training staff

- Data security and reliability

- Convenience of laptop computer and peripheral equipment used in patrol vehicles and at roadside inspection sites

- Costs and availability of wireless communication services, especially in rural areas.

Lessons Learned

The I-95 FOT-7 and SDM projects encountered many of the challenges that are typical of attempts to deploy new technologies to improve complex operations, such as those involved commercial vehicle safety enforcement. All of the states participating in these FOTs agree that these improvements are needed and support the use of information exchange technologies such as ASPEN software and programs like SAFER Data Mailbox. However, there are many factors that affect the success of such deployments. For example, ASPEN underwent several revisions during the testing phase of SDM. Also, each state participating in the project had to deal with unique problems involving software installations, hardware maintenance, system training, and integration of databases. Most states agree that smaller, more targeted projects may be more effective in testing technologies like SDM.

The biggest challenge faced by states implementing SDM involved the use of wireless communications. Connecticut was successful in converting to wireless systems partly because of their own in-house capabilities, but also because they have wide coverage with CDPD communication services. Other states found themselves investigating more costly and technically challenging alternatives. For example, New York and Pennsylvania investigated the use of satellite technology. However, it was never implemented due to cost constraints, technical challenges, safety concerns, and delays in deploying the required satellite infrastructure.

The following are some lessons learned by participants in the SDM project. These lessons are expected to provide guidance for long-term implementation of SDM statewide, in other states, and implementation of similar projects in the future.

• States should be included in the consultant selection process.

• Project responsibilities should be shared among all participating states.

• Identify features unique to a state and take those into account in designing and implementing the project. Information exchange and other things that work in one state may not necessarily work with other agencies or states.

• Redesign the Brayley box with commercial vehicle inspectors and their working environment in mind. Brayley boxes are not considered effective by some states. The laptop (MDT) configuration was found to be more flexible compared to Brayley boxes.

• In implementing SDM systems, communication costs should be taken into account.

• It became clear that the level of available wireless communication services varied greatly between states. Coastal states like Delaware, Connecticut, and Rhode Island generally had CDPD services available statewide, while larger, inland states like New York and Pennsylvania often lacked such coverage in large portions of their state. Furthermore, alternative analog services available in these areas were expensive and not reliable. Unfortunately, the technical consultant to the SDM project made the initial assumption that CDPD coverage would be available to almost all areas of the seven participating states, which turned out to be incorrect. In response, New York is continuing to explore other wireless options, such as CDMA, that is showing some promise upstate. Attempts to deploy and test an alternate system in Pennsylvania using satellite communications were not successful.

• Early on, the SDM project was envisioned to test out various wireless technologies beyond CDPD once it became apparent that adequate coverage was not available in all the involved seven Eastern States. Unfortunately, the technical consultant never demonstrated adequate knowledge of the alternatives, including the use of analog wireless and satellite wireless. In many cases, the states were left to solve their own technology deployment issues, after the consultant made the initial technology selection for them.

• For larger states, the issue of providing ISS type data for all carriers—both interstate and intrastate—also was identified. States view all carriers the same but only interstate carriers are under the jurisdiction of USDOT. Inspectors need to be able to have real-time access to safety and credentialing data for all carriers, but most systems developed by USDOT to date have provided this for only interstate carriers. FMCSA appears to understand this issue and is trying to address it.

Directions for Future Research

The customization or adaptation of computer systems to the roadside working environment, noted in the focus groups and interviews, are important indicators of the degree to which inspectors are accepting the technology. Firsthand observations or accounts of such user adaptations, if analyzed in greater detail, may provide clues to not only the degree to which inspectors are invested in the technology, but also the practical, operational needs the inspectors face in day-to-day operations.

The integration of safety information exchange technologies with electronic screening systems could produce significant benefits by focusing enforcement efforts on high risk carriers. This will result in fewer crashes involving unsafe trucks and drivers. However, research is need to find the best ways to use the safety information to identify trucks and drivers that represent the biggest risks.

Satellite communication may offer an alternative for wireless exchange of data to and from the roadside. While initial and operating costs seem high, and data transfer rates are relatively low, satellite communication may provide states a way to avoid the substantial cost of building, deploying, and maintaining new statewide infrastructure for existing wireless technologies such as CDPD.

Future research should also explore the ratio of time to information that is at the center of the inspection system. The time spent in conducting and reporting on an inspection using paper and computer-based systems could be compared and analyzed, as could the amount, accuracy, and timeliness of information available to decision-makers resulting from both ways of conducting inspections.

The effect of computer-based inspection technologies on the motor carrier companies and the truck drivers themselves could be explored. The tests discussed in this report were more concerned with the adoption of the technology among the inspector community. It can be assumed that changes in inspection practices will lead to adaptations among drivers and operating companies. Many of the same tests used to gauge inspector attitudes and opinions, such as interviews, focus groups, and observations, plus more quantitative measures of compliance and highway safety, could also be applied to the motor carrier community.

Software and Systems at a Glance

The following are some of the most important software applications and intelligent transportation systems in use in commercial vehicle operations and enforcement today. A list of abbreviations and their definitions appears at the end of this report. More information can be found in a CVISN glossary prepared by Johns Hopkins University (1998).

Aspen: A pen-based roadside inspection system that allows commercial vehicle inspection data to be electronically transferred to SAFETYNET, either via AVALANCHE or the CVIEW/SAFER Data Mailbox System.

AVALANCHE: Serves as a communications handler and preprocessor for inbound vehicle inspection reports coming from the ASPEN inspection software.

BLIZZARD: A software system for managing exchanges of inspection data between SAFER Data Mailbox, CVIEW, and SAFETYNET.

CDLIS (Commercial Driver’s License Information System) A software system that serves as a pointer to the complete record kept by the state issuing the license. The system is intended to provide states with the ability to check a nationwide information system for possible duplicates or for a suspended license before issuing a commercial driver’s license to an applicant.

CVIEW (Commercial Vehicle Information Exchange Window) A state-based system that provides carrier, vehicle, and driver safety and credential information to fixed and mobile roadside inspection stations.

CVISN (Commercial Vehicle Information Systems and Networks) The collection of state, Federal, and private-sector information systems and communications networks that support commercial vehicle operations. When fully deployed, the system will enable the delivery of electronic services to states and carriers in areas such as safety, credentials, and electronic clearance.

ISS (Inspection Selection System): A software algorithm that prioritizes carriers using SAFER snapshot data.

NCIC (National Crime Information Center) A national, computerized central index operated by the FBI and linking documented files of local and State criminal justice agencies for real-time inquiries.

PC*MILER: A commercially available point-to-point highway routing, mileage, and mapping software application, offered by ALK Associates, Inc. (Princeton, NJ). Provides latitude/longitude routing, route optimization, leg and cumulative mileage, time and cost estimates, detailed driving instructions, etc. The system is used by both motor carriers and state safety investigators.

PIQ (Past Inspection Query) A module of Aspen that retrieves information on past inspections of a specific vehicle (by license plate number) and driver from the SAFER/driver-vehicle system. The PIQ system requires landline or wireless communications between the roadside and a central database system.

SAFER (Safety and Fitness Electronic Record) An on-line nationwide data network that, when fully deployed, is intended to return a standard carrier safety fitness record to the requestor in a few seconds.

SafeStat (Safety Status Measurement System) A summary measure of a motor carrier’s safety performance and history.

SAFETYNET: A distributed system for managing safety data on both interstate and intrastate motor carriers and for the federal and state offices to electronically exchange data on interstate carriers with MCMIS.

SIE (Safety Information Exchange) The electronic exchange of safety data and supporting credential information regarding carriers, vehicles, and drivers involved in commercial vehicle operations. These decisions would be based on the ready availability of historical safety performance information.

Source: Johns Hopkins (1998, 2000)

FOTs that involved the development and deployment of safety information exchange technologies for use by state safety enforcement personnel include the SAFER Data Mailbox (SDM) FOT, involving six eastern states, and three safety-related FOTs (FOT 7, FOT 9, and FOT10) sponsored by I-95 Corridor Coalition Commercial Vehicle Operations (CVO) Working Group. In particular, FOT 7 tested a wide range of technologies for use by roadside enforcement personnel.

The SDM system uses a variety of advanced database and electronic communication technologies to provide up-to-date motor carrier and vehicle-specific safety information to enforcement officers at the roadside. The SDM FOT was designed to demonstrate the feasibility of using SDM technology to help enforcement staff identify commercial vehicles and drivers that violate out-of-service (OOS) orders. In part, this initiative was an outgrowth of several activities undertaken by states and the Federal Highway Administration (FHWA) in the mid-1990s to ensure that serious commercial vehicle safety violations were corrected before these operators returned to the nation’s highways.

The Corridor Coalition’s FOT 7 is closely linked to the SDM project, partially because it involves some of the same states, but, more importantly, because they share the same objectives. Both FOTs use the same communication links to help focus enforcement resources on high‑risk carriers and drivers and to evaluate the broader impacts of safety information exchange technology.

The SAFER Data Mailbox Field Operational Test

In 1997, the states of Delaware, Maryland, New Jersey, New York, Pennsylvania, and Virginia agreed to participate in a FOT to evaluate the performance, costs, and benefits of SDM. Connecticut, a state with extensive experience using laptop computers in its commercial vehicle enforcement program, joined the project in early 1998. However, Connecticut’s participation in the SDM evaluation was not funded through the Eastern States coalition grant. The FOT was divided into two phases. In Phase 1, SDM provided the capability to send electronic inspection reports from the roadside to the national SAFER database immediately after an inspection is performed. Phase 2 tested the ability to retrieve past inspection results on specific vehicles. These are the key features of SDM that allow enforcement officers to identify violators of OOS orders. The FOT was officially completed by the end of January 1999. However, most states planned to expand the deployment of SDM and related technologies after the test was completed. Descriptions of SDM and its key components are provided in Chapter 2. Because all of the states participating in SDM were members of the I-95 Corridor Coalition it was agreed to continue the deployment of SDM under the I-95 FOT program.

In addition to the seven Eastern states, participants in the SDM project included the Johns Hopkins Applied Physics Laboratory (SAFER development), SAIC (SAFER operations and maintenance), RSIS (SDM support contractor), and the FMCSA (formerly the Office of Motor Carriers of the FHWA), the funding agency and developer of the ASPEN software system. Battelle, prime contractor for ITS Program Assessment Support to the ITS Joint Program Office, and Battelle’s subcontractor, Castle Rock Consultants, were responsible for SDM evaluation.

The I-95 Corridor Coalition’s Safety-Related FOTs

The I-95 Corridor Coalition is a partnership of the major public and private transportation agencies, enforcement agencies, toll authorities, and industry associations that serve the Northeast Corridor of the United States, from Maine to Virginia. The Coalition places a high priority on commercial vehicle operations because of the significant role that motor carriers play in moving goods and people throughout the region. The goal of the Coalition’s CVO program is to enhance the safety and economic well-being of the I‑95 Corridor. To accomplish this goal, the Coalition funded FOTs in four areas: inspection procedures and technologies that target high-risk carriers (FOT 7), electronic registration (FOT 8), electronic screening (FOT 9), and safety management (FOT 10). Because of its concurrent role in the evaluation of CVISN and SDM, Battelle was contracted to serve as the independent evaluator for the safety-related FOTs (FOTs 7, 9, and 10).

At the time that this evaluation got under way, Virginia, the state participating in FOT 9, was reassessing its approach to conducting the electronic screening test. For this reason, the Coalition’s CVO Program Track Safety Subcommittee directed the evaluation team to defer plans to evaluate FOT 9.

The objective of FOT 10, Coordinated Safety Management, was to move toward a performance-based motor carrier safety compliance and management program that would reduce highway accidents and incidents in the I-95 corridor. The Coalition funded two projects. The state of Maine was to implement a modification to its state databases to allow use of a single U.S. DOT number for interstate, and some intrastate, vehicle credentialing. Participants in the second project included the states of Connecticut, New York, and Pennsylvania, and the ATA Foundation. This project was to study best practices in CVO enforcement and motor carrier safety compliance programs. The FOT would result in a CVO enforcement “toolbox” and a motor carrier safety “toolbox,” as well as educational materials and recommendations for outreach. This project complemented and extended the other safety-related tests, and therefore did not require a separate evaluation. Instead, results from of FOT 10 were to be used as input to the evaluation of FOT 7.

Because of the delayed status of FOT 9, and the interrelationship between FOT 7 and FOT 10, the Subcommittee directed the Battelle evaluation team to focus only on FOT 7. FOT 8, which dealt with electronic credentialing, was being evaluated separately.

FOT 7

The Coalition requested letters of intent for states to participate in FOT 7 in early 1997. The purpose of the FOT was to test the implementation of procedures and technologies that enable state inspectors and enforcement officers to focus roadside inspections on high-risk motor carriers. Six states were awarded funds to participate: Connecticut, Maryland, Massachusetts, New York, Pennsylvania, and Rhode Island. The project was designed to:

• Accelerate the deployment of pen-based and laptop computers [initiated under the Motor Carrier Safety Assistance Program (MCSAP)] and provide uniform training in their use to roadside inspectors and enforcement officers throughout the Corridor.

• Use these computers and specialized decision-support software (developed by the Volpe National Transportation Systems Center and others for the FHWA) to assist inspectors and enforcement officers in the selection of carriers for roadside inspection.

• Use these computers and specialized data entry software (developed by the FHWA) to streamline inspection procedures and reporting.

• Establish roadside communication links to the SAFER system (developed by the FHWA) so that inspectors and enforcement officers have real-time access to motor carrier safety performance records, and

• Pilot test the SAFER data mailbox system so that inspectors and enforcement officers have immediate access to regional and national data on vehicle and driver out-of-service orders and recent motor carrier inspection reports.

Battelle and its subcontractors were also responsible for evaluating FOT 7. Chapter 2 includes details on states’ approaches to implementing this test.

Coordination of SDM and FOT 7 Results in this Final Report

The approach developed to evaluate the SDM project was described in the SAFER Data Mailbox Evaluation Plan (March 1999). The plan presented the evaluation goals and hypotheses to be tested and described the variety of data collection and/or analysis efforts proposed to answer the study questions. A similar plan described the evaluation approach to the I-95 Corridor Coalition Safety-Related Field Operational Tests, primarily focusing on FOT 7 (Draft Evaluation Plan, April 1999). There was substantial overlap in the evaluation objectives of both FOTs, as well as in the number of states participating. Connecticut, New York, Maryland, and Pennsylvania were participants in both programs. Many states combined funds from the SDM project and FOT 7 to accomplish the same objectives. The evaluation plans highlighted the close coordination among the SDM project, the Corridor Coalition’s FOTs, and a third project, the CVISN Model Deployment Initiative (MDI) involving ten prototype and pilot states (Maryland, Virginia, Washington, Oregon, California, Colorado, Minnesota, Michigan, Kentucky, and Connecticut).

As evaluation contractor for all three efforts, Battelle coordinated data collection and analysis activities in order to make maximum use of available evaluation resources and to reduce the “evaluation burden” on states participating in multiple projects using the same technology.

In recognition of the shared interests of the two projects, this report presents the results of both the SDM demonstration and FOT 7. While SDM focused more narrowly on out-of-service violators, there are common issues related to time, cost, and institutional concerns raised by both projects, leading to closely related conclusions.

Organization of this Document

In the remainder of this document, we provide an overview of safety information exchange technology deployment and the approaches employed by the participating states (Chapter 2). Chapter 3 presents the evaluation goals, measures to be tested, and hypotheses or study questions originally posed for the SDM and FOT 7 projects, as well as the combined four goals addressed in this report. In Chapter 4, we describe the technical approach to the evaluation with a brief synopsis of the primary data collection and analysis efforts undertaken. Chapter 5 presents our findings, and conclusions are stated in Chapter 6. References and a list of abbreviations are also included.

Appended to this report are several documents that expand on or provide background for the information and results:

• Appendix A presents a summary and analysis of quantitative and open-ended responses to a survey of motor carrier inspectors, along with detailed tabulations of numerical answers and transcripts of verbal responses.

• Appendix B presents results from in-person interviews and focus groups conducted with motor carrier inspectors.

• Appendix C gives the results of a roadside screening assessment study in Connecticut.

• Appendix D is summary a tabulation of safety information system deployment plans as reported by nine Eastern states.

• Appendix E presents the results of a study of costs and institutional issues related to SDM technology deployment.

2. OVERVIEW OF SAFETY INFORMATION EXCHANGE

TECHNOLOGY DEPLOYMENT

A challenge in evaluating SDM and FOT 7 was the variation among states in commercial vehicle enforcement practices and in the degree to which they have adopted safety information exchange technology. The uses of technologies such as ASPEN, the Inspection Selection System (ISS), SAFER, and related systems vary with both individual characteristics of the inspectors and with the characteristics of the administrative systems in which they operate. The following provides a brief overview of current commercial vehicle enforcement practices in general and the specifics of deployment in the participating states. Appendix D summarizes responses from nine states to a questionnaire on current commercial vehicle enforcement practices.

Current Commercial Vehicle Enforcement Practices

All participating states have fixed sites and mobile units for conducting commercial vehicle enforcement, ranging from