8.0  Conclusions and Recommendations

This section presents conclusions based on the evaluation findings and recommendations for making further improvements to the ITD Road-Weather Integrated Data System.  The objectives and hypothesized impacts originally proposed in the Evaluation Plan are revisited in light of the data and findings.

8.1  Evaluation Hypotheses and Findings

The evaluation hypotheses of the ITD RWIS Integration project were identified and discussed in the Evaluation Plan.(1)  The evaluation investigated each of the hypothesized impacts.  Table 8-1 presents each of the hypotheses and summarizes the evaluation findings relevant to each.  Overall, many of the hypothesized impacts were fully or partially supported by the evaluation findings, indicating that the project was largely successful.  Non-supported impacts consist of those related to quantifiable roadway safety and maintenance resource utilization gains.  The RWIS integration project was, and is, intended to promote safety and resource utilization efficiency.  However, from the outset the ITD believed that project impacts in these areas probably could not be quantified, primarily due to the many exogenous variables.  The interviews with ITD maintenance personnel identified a number of other factors related to the basic nature of winter maintenance decision-making and practices that contribute to the difficulty in identifying project safety and resource utilization benefits.

8.2  Conclusions

This section summarizes the major conclusions of the evaluation of the ITD Road-Weather Integrated Data System.  More detailed and comprehensive treatments of findings and conclusions associated with each evaluation analysis have been provided in the previous sections of this report.  The conclusions in this section are organized in three categories:

• RWIDS as an ITD Maintenance Information Resource
• RWIDS as a Traveler Information Resource
• RWIDS as a Data Integration Platform

 

8.2.1  RWIDS as an ITD Maintenance Information Resource

• RWIDS represents an important evolutionary, but not a revolutionary, advance for ITD winter maintenance decision-making.  The RWIDS project has been very successful in its objective of making available a much more convenient, consolidated source of ITD RWIS data, and a vast wealth of additional weather data, to ITD maintenance personnel.  Personnel with good access to the Internet in areas where RWIDS coverage is good find the site very useful.  However, even the RWIDS proponents view it as “just another tool in the tool box” that has incrementally enhanced their decision making, rather than as a revolutionary improvement.  This perspective is shaped significantly by the fact that many of the strongest advocates of RWIDS were proactively seeking out and using similar (in some cases identical) data from primary sources prior to the RWIDS website implementation.  Further, a number of these personnel continue to use those other sources because they provide data not available on RWIDS or in a different format than on RWIDS (e.g., forecast data from ITD ESS through subscription to RWIS vendor).  These considerations do not diminish the success of the project but rather serve to put those successes in perspective.
• The usefulness of the current version of RWIDS varies widely for ITD maintenance personnel throughout the state, and can be enhanced.  A number of factors impact utilization and perceptions of usefulness.  Expectedly, the system is much less useful in areas with few or no sensor stations, and there are still significant portions of the state with no stations.  Also, although other organizations’ weather sensors do provide some useful information, and are better than nothing, there is a preference for ITD stations.  Presumably, one of the most powerful motivations for any maintenance user to consult RWIDS would be to consult ITD stations in their area.  Other factors that impact usage and valuation of RWIDS include lack of high-speed Internet access in many areas, and even more fundamentally, the scarcity of maintenance shed personal computers.  Finally, the level of personnel prior experience using Internet weather information and particularly ESS data in maintenance decision-making is an important factor.  To some extent this factor is related to the variation across Foreman Areas and Districts in overall attitudes toward technology and new practices.
• ITD maintenance personnel are generally comfortable synthesizing discrete weather data to draw conclusions and make decisions.  Only a couple of personnel indicated that it was difficult to make sense of the wide range of RWIDS data and that perspective is probably primarily a result of their overall lack of experience with using the RWIDS website and other Internet information.  Most maintenance personnel that were interviewed expressed little interest in expert systems for maintenance decision support.  They don’t feel that the technology is sufficiently sophisticated to effectively deal with what they perceive as a very complex and changing decision-making context.  They also expressed concerns that ESS coverage and reliability are not sufficient to support such systems now or in the foreseeable future.  It was clear that the interviewees tend to associate more synthesized information and suggested actions with an “automated” system.  Therefore, their distrust of decision support does not necessarily imply that they would be likely to reject more synthesized information.  In fact, several maintenance personnel cited the synthesized weather summaries prepared by NWS employees as among the most useful type of weather information to help coordinate shift changes.
• Most ITD maintenance personnel are interested in a wide range of data, but are most interested in the same basic information popular with the public.  Many maintenance personnel utilize a wide range of the RWIDS data, including some of the less direct information like Pacific Loop radar and Jet Stream images.  However, the RWIDS information consulted most frequently and which seems to be most valued is the same information most appreciated by the general public and commercial vehicle operators: cameras and basic current and forecasted weather information.
• RWIDS might not generate quantifiable improvements in roadway safety or the utilization of ITD winter maintenance resources.  There was no statistically significant change in statewide winter weather-related accident rates during the one-year period after implementation of RWIDS.  The absence of measurable, traceable improvements in safety is not unexpected given the many exogenous factors; the modest, incremental improvement in decision-making associated with RWIDS; the fact that not all maintenance personnel utilize RWIDS; and the inherently conservative and relatively inelasticnature of many ITS winter maintenance resource commitment decisions.
• Although demonstrable safety improvements are lacking, travelers perceive that their safety is enhanced through the use of the RWIDS webpage.  Eighty percent of on-line survey respondents who used RWIDS agreed that the information helped them better prepare for road-weather conditions and 76 percent of the respondents indicated that the information helped them drive more carefully. 

8.2.2  RWIDS as a Traveler Information Resource

• The addition of the RWIDS webpage accounts for a sizable percentage of the overall increase in Road Report website utilization.  Utilization of the Road Report website in general has risen consistently and dramatically in the year following implementation of the RWIDS webpage (169%).  About half (47%) of the new user sessions included RWIDS viewing.  Further, 85 percent of all sessions that included RWIDS viewing featured only viewing of RWIDS content.  A full 25 percent of post-RWIDS Road Report sessions feature viewing of only RWIDS information.
• The traditional Road Report information, fed by ITD maintenance foreman reports, is not “replaced” by the RWIDS website.  Although “RWIDS-only” user sessions account for a significant portion of the overall growth in Road Report website utilization (about 47%), most of the growth, and most of the total current utilization is for traditional Road Report information (the road closure and conditions information provided by maintenance foremen).
• RWIDS has probably not reached its full potential as a public information resource, since it has not been marketed or integrated.  The ITD has not marketed the RWIDS webpage.  Although significant levels of usage have occurred as result of users happening across the new webpage, and presumably through some word-of-mouth, it seems likely that marketing could significantly increase usage.  Also, since it is clear that users still value—and separately consult—the traditional Road Report closure and condition information, that overall Road Report usage might be further enhanced if the RWIDS information was integrated with the traditional information. 
• Public RWIDS users, including commercial vehicle operators, are most interested in camera images and basic current and forecasted weather information.  Camera images are very popular and are the most frequently accessed RWIDS data.  Like the ITD maintenance personnel, respondents to the on-line survey and commercial vehicle operators feel that additional cameras would be very useful.

8.2.3  RWIDS as a Data Integration Platform

• RWIDS has successfully leveraged a wealth of data from other organization’s ESS.  The RWIDS webpage provides access to approximately 120 ESS operated by other organizations, including other state department of transportation ESS.  Most maintenance personnel find these other stations useful, although they prefer ITD operated ESS.
• Other organization’s ESS supplement but do not necessarily replace ITD ESS.  Data from other organizations’ stations provide a low-cost means to greatly enhance data coverage.  However, even in the case of Idaho which acquired access to a large number of such stations, such stations do not necessarily address all, or even the most critical (hot spot), data coverage gaps.  Further, although they provide some value, other organizations’ stations do not provide a standard, full compliment of information.  For example, very few non-transportation department stations provide pavement data.  Non-transportation agency stations are typically not ideally sited to support roadway maintenance-related data collection.  Finally, other organizations may not maintain their stations at levels needed to support public traveler or highway maintenance information needs.
• RWIDS has allowed ITD to pursue more cost-competitive ESS procurements.  Since completion of the RWIDS data platform, two sets of new ITD ESS have been successfully integrated into the system.  Both of the procurements featured different brands of stations, and both brands are different than either of the two pre-RWIDS (“incumbent”) brands composing the ITD road-weather information system.  Before RWIDS, when considering new stations, ITD was essentially restricted to one of the two incumbent brands.  Installing equipment of any other make would have necessitated a third, separate proprietary user interface.  By providing access to multiple brands of ESS via a single user interface, RWIDS has allowed the ITD to select from among a wider, more cost-competitive range of offerings and thereby significantly reduce the costs of new stations.  The ITD estimates that the cost to deploy four sites using one of the two new brands of stations is about 12 percent lower than the bid submitted by the primary “incumbent” RWIS vendor.  In that particular procurement, the flexibility provided by RWIDS resulted in a savings of about $30,000, or about $7,500 per site.

8.3  Recommendations

A number of recommendations have been developed based on the findings of the evaluation.  They are presented in the spirit of suggestions for consideration not only for ITD but also for other state transportation departments who may be considering RWIS deployment or integration.

• Continue to fill gaps in ESS coverage.  Although the addition of the approximately 120 other organizations’ ESS to the ITD RWIS provided an immediate improvement in data coverage, many critical gaps remain.  In some cases, even if there is another organization’s station in a given general area, the specific placement or type of information gathered may not fully meet winter maintenance decision making needs.  Overall, although helpful, other organizations do not completely eliminate the need for additional ITD ESS.  The ITD may also consider adding certain sensors to existing ESS operated by other organizations, most notably pavement sensors, which are not included on most non-ITD stations.  Based on the fact that the ITD continues to implement their own ESS it is clear that they agree that continued investment in ITD stations is a necessary part of the overall RWIS strategy.
• Improve the reliability of ESS.  A number of maintenance employees expressed concerns about the reliability of both ITD and non-ITD stations, noting that particular stations have been inoperative for extended periods.  Such unreliability obviously significantly limits the usefulness of the RWIDS website and it is recommended that the ITD continue to work to improve the reliability of stations.  In cases where other organizations’ stations located in key locations are unreliable, consideration may be given to implementing ITD stations in those locations. 
• “Level the playing field” statewide with respect to personal computers and Internet access.  Even more fundamental to the use and usefulness of RWIDS than ESS coverage is that maintenance sheds have personal computers and Internet access; preferably high-speed access.  It is strongly recommended that ongoing ITD efforts in this area continue and be accelerated if possible. 
• Provide additional, focused training to ITD maintenance personnel.  Most of the maintenance personnel that were interviewed expressed interest in receiving a second round, or more, of training on the RWIDS webpage.  Many of them also indicated that it’s important that the training be hands-on, with computers at each training station.  It is recommended that the follow-up training provide instruction on overall information gathering strategies and use of various types of information, including the various RWIDS data, in support of specific types of winter maintenance decisions as well as the use and benefits of road weather information to make proactive decisions about treatments (such as anti-icing).  Such information represents a natural evolution in ITD headquarters efforts to encourage efficient and effective winter maintenance and was specifically requested by a couple of the maintenance personnel.  It is also recommended that training on the use and benefits of road weather information to make proactive decisions about treatments (such as anti-icing) be considered.
• Market RWIDS to the public.  Although a substantial number of Road Report users have “found their way” to the RWIDS page in the absence of any marketing, it would seem that marketing of the RWIDS webpage could significantly increase utilization.
• Consider more fully integrating RWIDS and the remainder of the Road Report.  The general public and commercial vehicle operators are interested in the “traditional” Road Report closure and conditions information as well as information on the RWIDS webpage.  It is recommended that ITD consider integrating these information types, including possibly using RWIDS data to enhance the traditional roadway-specific Road Report advisories.  Such integration could occur as part of the planned ITD implementation of a comprehensive traveler information system.
• Develop system performance monitoring capabilities and monitor the utilization and effectiveness of RWIDS.  Currently, ITD does not have effective mechanisms for monitoring the utilization of the RWIDS webpage by either the public or ITD maintenance personnel.  In fact, limitations in this regard, such as the inability to clearly distinguish all ITD RWIDS user sessions, impacted this evaluation.  In order to support a continuous quality improvement process it is recommended that the ITD implement mechanisms that will allow for tracking of RWIDS utilization and that they routinely analyze that data.  For example, it is possible to configure the RWIDS server logs so as to capture information on the use of RWIDS webpage data that appears in pop-up windows, accessed from another organization’s server.  This analysis could not quantify such usage because the logs were not configured to preserve such data.  Likewise, there are various options for tracking RWIDS usage by individual maintenance sheds.  Such geographic resolution was not possible in this analysis because so many of the maintenance personnel chose not to use the employee log-in feature.  The log-in access included an obvious “back door” that allowed them to effectively utilize the site without logging in.
• Monitor broader national developments in road weather information sharing and position ITD to capitalize on advances.  Road weather information sharing and utilization of road weather information in maintenance decisions are topics being considered by transportation agencies throughout the United States.  The ITD should be aware of FHWA activities to promote technology transfer of the Federal MDSS prototype, which integrated road weather and resource information to provide route-specific treatment recommendations.  In addition, the USDOT Clarus Initiative is one of the most important activities that ITD should be aware of, and if possible, participate in.  That initiative is focusing on integration of all weather and pavement condition observations from transportation agencies and shares the same philosophy in data integration with this ITD earmark project.  The specific RWIDS project approach to leveraging data from the MesoWest consortium has been successful, but monitoring and participation in the Clarus Initiative Coordinating Committee can help keep ITD in a position to benefit from information processing and dissemination that may result as a part of that initiative.
• Continue to consider opportunities to further synthesize road weather information for winter maintenance personnel.  When asked about their interest in software tools that could provide recommendations for maintenance actions based on RWIS and other data, several maintenance personnel reacted negatively, seeming to interpret any sort of computerized decision support with a move toward minimizing or eliminating their role in decision-making.  Also, most of the maintenance interviewees are generally content with synthesizing a wide range of discrete data themselves.  However, this does not suggest that maintenance personnel would not benefit from, and adopt if properly introduced to, a Maintenance Decision Support System.  Maintenance Decision Support Systems use state-of-the-art weather forecasting and data fusion techniques and merge them with computerized winter road maintenance rules of practice.  The result is a set of guidance for maintenance managers that provides a precise forecast of surface conditions and treatment recommendations customized for specific routes.  These systems do not make decisions, but rather make recommendations which personnel can choose to implement.  It is recommended that ITD keep an open mind to the possibility of introducing maintenance decision support.

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS