3. Data Sources and Analysis Methods
3.1 Review Information and Collect Existing Data
A literature search was conducted to identify existing data on the impacts (real and perceived) of CVISN deployment on motor carriers. The literature search included a targeted review of documents and databases known to contain CVISN-related information and a more general search engine-based approach using resources associated with highway transportation, advanced technologies, and the motor carrier industry.
The targeted review involved mining CVISN-related documents and databases for information associated with both perceived and measured economic and business benefits of CVISN deployment. Sources of CVISN information included:
- The CVISN Self-Evaluation benefits and lessons-learned reports now being completed and updated by many participating CVISN states
- Published reports posted on the FMCSA and ITS Electronic Document Library web sites
- Published reports on the CVISN web site (maintained by the Johns Hopkins University Applied Physics Laboratory)
- The independent evaluation report on the CVISN Model Deployment Initiative (published in 2002)
- Various state-sponsored and federally sponsored evaluations and case studies of different aspects of CVISN deployment
- Information in the web-based unit costs and benefits tracking databases maintained by the ITS Joint Program Office
- Public information posted by states on their respective transportation and law enforcement web sites
- Data being collected by other organizations on separate task orders, and original research through contacts with selected motor carriers and other stakeholders.
In conjunction with the targeted review, internet-based research was conducted to identify any other existing data that may have been published on the impacts of the CVISN deployment program. This work included keyword searches of technical databases and search engines.
Sources for the generalized literature search were identified through computer-based library resources. Keywords included the following: commercial motor vehicle, CVISN, weigh station, screening, preclearance, PrePass, Norpass, credential, registration, permit, inspection, truck, safety, efficiency, cost savings. These terms were used in various combinations using Boolean AND/OR/NOT logic. The search focused on documents published from 2001 to the present.
Researchers reviewed abstracts of journal articles or reports identified through technical databases and search engines to isolate the most relevant results. Full-text versions of all relevant items were obtained. The bibliographies of any relevant articles identified were reviewed for additional relevant references. All relevant results were documented and compiled in a reference list including bibliographic citation, brief description, and notes on where and how the information could be used in the business case.
3.2 Establish Contacts with Motor Carriers
Because this was a targeted survey—intended to collect information primarily from larger motor carriers who were using CVISN technologies and who were based in active CVISN states—it was not meant to be statistically representative of any particular population of motor carriers. Instead, we set out to collect in-depth economic information from between 20 and 50 motor carriers. A larger-scale, more representative survey was recently completed, under a separate FMCSA/FHWA task order (BA34007), the CVISN National Evaluation. That survey attempted to contact more than 1,800 motor carriers, and yielded more than 800 completed interviews. Results are currently in preparation for publication in late 2008.
The following sections summarize the development of the telephone calling lists and the calling process for the CVISN motor carrier business case. Further details on the telephone interview process and data collection for this study are presented in Appendices A and B.
Calling List Development. Two lists of motor carrier companies to contact were developed. The primary source for the first calling list was the FMCSA Motor Carrier Management Information System (MCMIS) census file, as of September 30, 2006. The MCMIS list was chosen as the most current, complete source of contact information on all motor carriers subject to federal regulation. It was hoped that the MCMIS census file would give the research team a fair cross-section of the motor carrier industry.
To collect sufficient information from carriers who were actually using CVISN technologies, the state CVISN self-evaluation reports were used to identify those states that were highly active in making CVISN available to the carriers based in their states. For example, the percentage of IRP transactions processed electronically and the number of electronic screening sites within a state were taken as reasonable surrogates of a state’s activity in deploying CVISN. Carriers from these states were believed to be more likely to be using CVISN technologies:
- Arizona
- Indiana
- Kansas
- Kentucky
- New Mexico
- Oklahoma
- Tennessee
- Virginia
Twenty-five carriers that were listed in MCMIS as operating more than 20 power units, and coded as being a company in the for-profit motor freight business were selected from each of these eight states. The list of 200 carriers was augmented with approximately 50 carriers from the PrePass and Norpass web site lists of enrolled carriers, and another approximately 40 carriers from a national commercial business directory. The 20-truck minimum company size was chosen for this investigation because it was thought that these larger carriers (a) would be more likely to have staff dedicated to the credentials and safety/screening functions affected by CVISN, (b) would be more likely than the smaller carriers to have staff available to participate in telephone interviews, and (c) would be more likely to have deployed some aspect of CVISN, which represented the main target population for the survey. The vast majority of motor carrier companies operate very few trucks; however, the top 20 percent of U.S. carriers by company size are responsible for approximately 80 of all drivers, trailers, and tonnage in the industry (Murray 2007). The implications of this focus, and opportunities for expanding the coverage of CVISN technologies to smaller motor carriers, are discussed in Section 7.2 below.
The second calling list was developed via a multi-tiered process that collected carrier contact data by sector, geography, and proclivity to be engaged in CVISN. Specifically, a proprietary trade association list of carriers in core CVISN states was developed, augmented by carrier “tax & registration” committee lists from state and national trucking associations. The two lists were compared to prevent callers from contacting the same company.
An interview guide, in the form of a telephone calling script (Appendix A), was drafted, with the goal of balancing the need for collecting useful, detailed economic and business information with the need to keep the interview length and thus the respondent burden as low as possible. Telephone research staff developed the interview guide, in consultation with FMCSA and the ITS Joint Program Office Program Assessment Customer Service Survey Review Team. The targeted length of the interviews was approximately 20 minutes.
Four pilot-test conference calls were pre-scheduled and made between February 23 and March 9, 2007, using a preliminary draft of the interview guide. These pilot calls tended to last approximately 30 to 45 minutes each, including some time for introductions, background, discussion of the survey methods, and refinement of terminology. The data from the motor carriers and service bureaus invited to take part in the pilot calls were included in the economic analysis. Minor modifications to the calling script were made based on the responses to the pilot calls. If a respondent requested it, the interview guide was also e-mailed, faxed, or sent to the company by U.S. Mail. As described elsewhere, some of the contacts began with an e-mail message, instead of a telephone call.
A blanket letter of authorization was prepared and signed by the FMCSA Task Order Manager, and this letter was sent to a few respondents who requested documentation of the purpose and legitimacy of the interview process. Respondents were told that the information collected would be reported anonymously.
Calling Process and Response Rates. Industry contacts using the final interview guide took place between April 2 and June 8, 2007. Table 1 shows the response rates.
|
Calls Attempted |
Calls Completed (%) |
|---|---|---|
Pilot Calls |
4 |
4 (100%) |
Study Calls |
268 |
34 (13%) |
TOTAL |
272 |
38 (14%) |
Table 2 shows the numbers of initial contacts made by telephone or e-mail/fax, compared with the number of surveys completed by both media. As shown in Table 2, approximately 20 percent of initial contacts were made by e-mail, and the bulk of completed surveys were taken by phone.
|
Number of Initial Contacts |
Number of Completed Surveys |
|||
|---|---|---|---|---|---|
Phone |
Phone |
Fax |
|||
Pilot Calls |
4 |
0 |
4 |
0 |
0 |
Study Calls |
215 |
53 |
23 |
6 |
5 |
Response rates were somewhat lower than expected. For the callers using the MCMIS list, it was found that many of the companies on the MCMIS census file had phone numbers that were either out of service or now belonged to a different company or individual, or the company simply did not respond to the initial phone and/or email messages. The low response rate for these calls indicates that the MCMIS census file may not be the ideal source for identifying motor carrier companies to contact to request responses to a lengthy, impromptu telephone survey dealing with potentially sensitive carrier business information.
In contrast, callers using the trade association listI had a much higher response rate when calling carriers, possibly due to their ability to better target interested parties, and access to more up-to-date and accurate contact information than available through the MCMIS census file.
Callers had more success in gaining an interview when the purpose of the study was presented simply. Callers found that saying something brief—for example, “We’re working on a study for FMCSA”—made more sense to respondents than going through the entire introductory script (see Appendix A) before asking for the appropriate party.
In the follow-up to the pilot calls, it was agreed that callers could use some latitude or conversational discretion in introducing themselves and the project, while generally keeping the language of the survey itself—especially the wording and flow of the individual questions—intact.
The introduction was often shortened or re-worded, with the insertion of the two CVISN technologies into the introductory paragraph and an explanation of the two technologies only if the respondent was not familiar with them (virtually never). It was emphasized that the caller was interested in the respondent’s experience with credentialing systems, particularly with respect to business impacts (cost savings or increases). In all cases interviewers emphasized that 1) participation was voluntary, 2) responses were confidential, and 3) FMCSA was the investigating organization.
Other questions were delivered verbatim, but the surveyor’s ability to offer probing questions was enhanced as the survey progressed and carriers provided examples of additional costs and benefits they had encountered. In many instances, the range of survey content questions ultimately required the participation and responses from two to three different individuals within each responding company. On average, the total time needed to complete the surveys was estimated at approximately 30 to 35 minutes.
Job titles and duties varied widely among the respondent companies, so when the caller did not have a specific contact name, the caller requested to speak with the person in charge of operating permits, safety, and/or weigh station clearance. It was found that “credentialing” is not a widely used term. Instead, “operating permits” was a more widely recognized term among the carriers contacted in this survey.
Table 3 shows the percent of calls completed, and breaks down the prevailing types of nonresponses faced by the research team.
|
Approximate Percent of |
|---|---|
Company responded and call completed |
14 |
Company did not respond to repeated contact attempts |
64 |
Phone number out of service or company closed |
12 |
Company initially agreed to participate, but did not respond to |
7 |
Company declined and gave a reason |
2 |
Company declined without giving reason |
1 |
TOTAL |
100 |
Among participating companies, the following types of reasons were given:
- Expressed particular interest in the topic
- Offered multiple benefits and disadvantages of the technologies
- Saw survey as an “opportunity for industry’s voice to be heard.”
Among companies that were contacted but declined to participate, the following types of reasons were given:
- Trucks are leased from another company, so the company has nothing to do with licensing, registration, etc., or else owner-operators take care of their own credentials and dispatchers order miscellaneous permits as needed
- Company policy prohibits responding to surveys or questionnaires
- Company does not deal with the issues described in the survey
- Company does not deal with weigh stations, etc., because it just distributes products locally.
As noted above, the response rate was much higher when callers contacted companies by phone, compared to making the initial contact by e-mail. The trade-off was that calling individual companies and making contact with the respondent at a convenient time for an interview proved to be very time consuming. Conducting the telephone interviews took much longer than estimated as well; interviews almost always took 30 minutes, but sometimes stretched to as much as an hour due to the detailed nature of the interview. Via e-mail, research staff were able to distribute many more surveys in less time, but saw a lower rate of response.
Many carriers preferred to be faxed or e-mailed a copy of the survey, which they could then fill out at their convenience rather than complete the survey over the phone. While this deprived researchers and respondents of the ability to ask clarifying questions, the advantages of contacting a greater number of respondent companies and allowing time for respondents to more thoroughly consider questions requiring cost or number estimates was a favorable trade-off.
Interviews became easier to conduct as time went on and the researchers gained a better understanding of the types of credentialing systems used by carriers. Familiarity with credentialing was extremely useful for developing rapport with respondents. Generally, respondents for the pre-clearance section of the interview were less forthcoming with implementation details, and tended to be more “operationally” focused.
In juxtaposing the complexity of the trucking industry with the detailed, multivariate survey objectives, it quickly became apparent that the survey was targeting different, very disparate audiences. With some exceptions, preclearance issues are divided between safety and technology/maintenance functions. Credentialing application and management functions was considered a “back-room” function—in contrast to dispatching and operations. Consequently, credentialing staff often had little information on trip and operational impacts. Furthermore, carrier size often determined the level of familiarity and cross-cutting of data and information; the larger the carrier, the more likely that credentialing management, financing, and operations were managed as separate functions within the carrier’s business plan. Alternatively, small carrier staff had greater understanding of the full range of issues and impacts, but lacked the internal sophistication to quantify the full range of CVISN costs and benefits.
The pilot calls—made to companies purposefully selected and scheduled in advance—achieved their goal of allowing the research team to refine the survey instrument and the methods of eliciting information. However, the pilot calls did not fully prepare the team for the eventual difficulty in completing surveys when initially contacting motor carriers on the calling lists. None of the four companies on the pilot calls indicated any concerns about the length, intrusiveness, or complexity of the survey. In retrospect, the complexity of the CVISN survey/interview methodology was almost certainly the leading factor in explaining the relatively low response rate as well as posing challenges in rectifying responses within and across interviews.
3.3 Develop Economic Parameters and Model
To establish a framework for systematically examining the benefit and cost elements outlined in the previous section, this study relies on a number of economic parameters and assumptions to determine the relevant return on investment (ROI) ratios, net benefits estimates, and payback periods (Table 4). Based on a preliminary review of CVISN studies, the analysis time horizon was established at 10 years, a period that is based on the expected economic life of CVISN equipment. The discount rate established for compressing streams of benefits and costs into present value terms was established at 7 percent, consistent with the recommendations of the Office of Management and Budget (U.S. OMB 1992). This rate serves as a proxy for the after-tax rate of return to private capital.
Parameter |
Assumption |
Basis of Assumption |
|---|---|---|
Discount rate |
7% |
OMB Circular A-94 |
Analysis base year |
2007 |
|
Annual PPI inflation |
2.6% |
Average annual change in the Producer Price Index |
Annual growth in truck registrations |
3.0% |
American Trucking Association, U.S. Freight Transportation Forecast to 2008 |
Analysis time horizon |
10 years |
|
The base year of the analysis is 2007, and all monetary values are presented in constant 2007 dollars unless otherwise noted. The average annual change in the Producer Price Index (PPI) is estimated at 2.6 percent based on growth in the PPI, as estimated by the BLS from 2001 to 2005. Annual growth in the number of heavy trucks operating in the US is estimated at 2.98 percent based on the American Trucking Association’s (ATA) US Freight Forecast to 2008.
These parameters are combined with industry data in an economic model to determine ROIs, net benefits estimates, and relevant payback periods. The Microsoft Excel-based economic model designed for this study contains multiple output or results pages and a single input or assumptions page. Designing the model in this manner enabled the analyst to enter inputs and change study parameters, including those related to discount rates and the analysis base year, without the need to examine detailed study data or possess any foreknowledge of the model’s design.
For the purposes of estimating the startup and recurrent costs associated with electronic credentialing, mean or median values were not used because the data were highly variable and many respondents either were unsure what costs had been incurred or indicated that the costs were negligible. Since statistical means and medians were of limited usefulness, the research team assigned what it considered to be reasonable values based on its evaluation of the survey responses.
The original intent was to attempt to segment the industry by region, company size, type of operation, and other demographic variables. However, the relatively small sample size and the homogeneity of respondents (e.g., all operating interstate, and mostly across the 48 continental states) meant that segmentation would have left very small numbers of carriers in each group, reducing the representativeness of the results. Further, not all of the responses received could be used to support the economic analysis, thus reducing the overall sample size for many of the questions. Unless otherwise indicated, all analyses in this report consider the respondents as a single group.