CHAPTER 13: GOAL 9, EVALUATE THE PERFORMANCE OF THE HARDWARE, SENSORS, AND DATA COLLECTION SYSTEM USED IN PHASE II

DATA ANALYSIS OVERVIEW

The 100-Car Study was originally planned as a pilot test to prepare for a Phase IV large-scale naturalistic data collection effort. As such, the purpose of Chapter, 13, Goal 9 is to evaluate the hardware, sensors, and data collection systems used in the 100-Car Study and, based on system performance and results, determine the best data collection configuration for a large-scale effort. The definition of what constitutes a large-scale effort has not yet been precisely defined, but, for the purpose of these analyses, the assumption is that 5,000 vehicles will be on the road for a period of two years each.

When reviewing this chapter, it is important to note that the results are very closely tied with the results of the Goal 10 Report (separate report) in making suggestions for a future Phase IV effort. This research goal describes hardware, sensor, and data collection system performance throughout the data collection effort in terms of its reliability, maintainability, and sensitivity to event detection. In other words, this research goal focuses on the aspects of data collection that are important to a large-scale study while the Goal 10 Report (separate report) focuses on the aspects of data reduction important to a large-scale study.

For Chapter 13, Goal 9, different analyses were used to address each of the three questions. To address Question 1, which included the assessment of hardware failure rates, frequencies of equipment failures were tallied based on repair records. Downtime for these failures was also estimated to determine failure rates for different system components.

Question 2, which assessed cost/benefit issues, was addressed by determining the net benefit of valid triggers obtained versus all of the cost factors associated with the triggers (including sensor costs, installation time, etc.) in a large-scale study.

Question 3 focused on determination of the criteria for data triggers in a large-scale effort. The assumption was made that it will not be possible to collect continuous data for 5,000 vehicles as was possible with 100 vehicles. Therefore, a scheme must be developed to trigger data collection for a large-scale effort that minimizes misses and false alarms in the dataset. To address this issue, discriminant analyses and logistic regression were used to determine logical trigger settings that contributed maximum event detection without incurring a large number of invalid events. Within the discriminant analysis approach, various techniques were tried, including stepwise selection of variables and the use of costs for particular classification errors.

Data Included in the Analyses

The analyses described in this chapter used data from vehicle repair logs to determine system and component downtime. In addition, the database of reduced events was used to obtain the efficacy of the various sensors and to determine suitable sensors and their corresponding trigger settings for inclusion in a future large-scale data collection effort.

---

Question 1. What were the failure rates for each sensor type and/or hardware equipment?

---

There are several pieces of information needed to address failure rate for each piece of hardware, including:

• When was each vehicle on the road?
• What lessons were learned during the data collection period that improved detection and repair time and when were these lessons learned?
• What kind of failures occurred?
• When did failures occur?
• When were failures detected?
• What were repair times for failures as well as the failure rate for components?

The answers to these questions are not independent. For example, the repair time for each failure was greatly dependent on the lessons learned during the data collection period that improved the efficiency with which repairs were detected and made. The time to make a repair changed depending on when it occurred during the course of the study. In addition, other factors affected the estimation of reliability since, for example, a failure was not detected the moment it occurred. Instead, a failure was detected via three different methods that occurred at different times during data collection. Thus, reliability must be estimated and the questions that were considered to make reasonable reliability estimates are considered in more detail below.

When was Each Vehicle on the Road?

The first vehicle was put on the road during January 2003. From that point on, vehicles were phased into the study until June 2003. At this point, all 100 vehicles were on the road collecting data. In January 2004, the phase-out process began with the vehicles that had been in service for one year. The phase-out process concluded in July 2004, after private vehicles had been on the road for a minimum of one year and leased vehicles had been on the road for a minimum of 13 months. The following table shows the number of vehicles on the road for each month from January 2003 to July 2004.

Table 13.1. Number of vehicles on the road for each month of the study.

Month	Cars on the Road (@ end of month)
January, 2003	6
February, 2003	12
March, 2003	33
April, 2003	65
May, 2003	100
June, 2003	100
July, 2003	98
August, 2003	100
September, 2003	100
October, 2003	99
November, 2003	98
December, 2003	95
January, 2004	96
February, 2004	91
March, 2004	85
April, 2004	67
May, 2004	50
June, 2004	26
July, 2004	0

What Lessons Were Learned During the Data Collection Period That Improved Detection and Repair Time and When Were These Lessons Learned?

A very important factor in determining the severity of a particular hardware failure is the amount of total downtime for the failure. The total downtime is a function of:

1. The time it took to detect the failure;
2. The time to obtain a replacement part(s); and
3. The time to actually perform the repair needed within the vehicle.

Throughout the 100-Car Study, lessons were learned that reduced these times and made data collection more efficient. The lessons learned to reduce the time to obtain a replacement part and to perform repairs are presented in this subsection. Efforts to reduce the time it took to detect a failure are discussed in a different subsection (see the subsequent question in this section called “When were failures detected?”).

For the three times affecting a failure described above, the greatest variable tended to be the time required to perform the repair. This timeliness factor was considered an important area in reducing downtime throughout the data collection effort, and was therefore the focus of many timesaving measures. The time to acquire repair parts was also minimized to the extent possible within the study. The lessons learned while developing these timesaving measures were initially discussed in Chapter 4, Lessons Learned, but are expanded here to provide a more detailed framework to support the assumptions made later in the section to calculate estimated failure rates for various components.

A lesson learned from the beginning of the study was that an effective way to minimize repair time was to have replacement parts on-hand at all times. A key lesson learned, however, was the number of spare parts needed in inventory. For example, it was not anticipated that a larger number of radar antennas would be needed due to damage in rear-end crashes. Throughout the study, the time required to acquire a part was eliminated by having a sufficient number of spare parts and/or spare systems available when they were needed. Throughout the data collection effort, there was not a single repair instance in which downtime was increased due to the unavailability of a particular part.

The replacement time was the most variable and lengthy aspect of the failure downtime. For an estimated 50 percent of the duration of the study, the downtime for failures was approximately 2 weeks, since bi-monthly trips were scheduled for technicians to travel from Blacksburg to northern Virginia to fix all failures that had been identified up to that point in time. This downtime period applied to all failures types: catastrophic, major, and minor. During this time period, methods to increase cost-effectiveness, decrease replacement downtime, and reduce the number of active service requests were being explored.

For the remainder of the study, two important measures were taken to reduce the replacement time for inoperative hardware. First, chase vehicle drivers (or data “downloaders”) who were already in close contact with the cars were trained to perform minor maintenance procedures after gaining access to the car. Up to this point, the downloaders were only required to notify any perceived faults with the equipment. While effective for fault detection, it was felt that this approach was time-inefficient and under-used downloader abilities. Therefore, for the second half of the study, downloaders were also required to watch video from each vehicle when they downloaded it and fix any cameras that were inoperative. This approach reduced the response time to minor failures from 2 weeks, up until that point, to a maximum of 1 week. Second, an on-site (i.e., a northern Virginia area) technician was hired to respond to any catastrophic or major failures. In addition, the on-site technician was supplied with replacement systems that could be “line-replaced” to decrease overall system downtime. These actions reduced the response time to catastrophic and major failures from two weeks to a maximum of 1 week for the second half of the study.

A logical question is why these lessons took several months to be implemented. Recall that vehicles were phased into the study from January 2003 to June 2003. Until most of the vehicles were on the road, it was possible to make repairs quickly. However, once most of the vehicles were on the road, the full burden of repairs became apparent. It was at this time that new measures to address repairs were adopted.

These lessons learned can be used to estimate system downtime. For approximately the first half of the study, the maximum system downtime (combining fault detection and repair) was less than three weeks. For the final part of the study, the maximum system downtime was reduced to less than two weeks. These downtime distributions will be considered when estimating failure rates for the DAS hardware.

What Kind of Failures Occurred?

Components of the data collection system can be classified into two general categories: data gathering and data storage (in-vehicle). Both of these functions had several distinct hardware items contained within the data acquisition system. For example, each sensor component had three parts: the sensor itself, its associated cable, and the component inside the system box that controlled its data collection (see Chapter 2, Method). The failure of any of these components resulted in a sensor failure. All of these hardware components were subject to failure within the data collection period. However, the failure of a particular hardware component was only as significant as the type of data that was lost due to the failure.

Based on the type of data lost, three different classifications for the observed failure can be considered: catastrophic, major, or minor. Catastrophic failures resulted in the loss of data already stored within the system and the cessation of further data collection. Major failures caused the loss of a substantial number of data streams without loss of data already stored within the system. For example, the system stopped collecting video due to a malfunction in the video board. However, other data (e.g., throttle position, radar) continued to be collected. Finally, minor failures caused the loss of a small number of data streams without loss of data already stored within the system. For example, the face camera view was not acquired, but the remaining camera views were available within the data stream.

There were no catastrophic and major failures requiring a complete system overhaul or replacement of the DAS during the course of the study. Without exception, these failures were repaired through the replacement of one or two parts of the DAS, which was then put into service again. System remove/replace operations to fix these failures occurred in 45 instances. The replacement process entailed the removal of the data acquisition system from the vehicle and installation of a different DAS. The malfunctioning part within the unit was then repaired offline and the refurbished unit was subsequently used as a replacement.

Based on repair logs and communications between technicians, it is estimated that approximately 45 catastrophic or major failures occurred within the data collection period. Causal factors for each of these failures are presented in Table 13.2, along with the sensor or subsystem affected and the frequency of occurrence. Note that the frequencies add to more than 45 because in many instances failures occurred on more than one sensor or subsystem at the same time.

Table 13.2. Causal factors for catastrophic and major failures.

Failure	Instances	Sensor/Subsystem Affected
System fuse blown	5	Power Control Battery Backup
Car battery draining Car failed to start System fuse pulled to prevent battery drainage	28	Power Control Battery Backup
System shuts down during a trip Empty data files	34	Acquisition Software
Hard drive failure Hard drive becomes corrupt	33	Acquisition Software
Download cable failures resulting in incomplete downloads	17	Remote Download
Video problems – No video	22	Real-time Video

Minor failures were, for the most part, constrained to sensors, since loss of data acquisition capability entailed a catastrophic or major failure. A total of 268 minor failures were recorded in repair logs or communications between technicians. These failures, their corresponding sensor or subsystem, and their number of occurrences, are listed in Table 13.3.

Table 13.3. Causal factors for minor failures.

Failure	Instances	Sensor/Subsystem Affected
Improper camera orientation	34	Real-time Video
Cameras falling from mount	21	Real-time Video
Video problems – Single view (other views available)	42	Real-time Video
VORAD – Radar unit	8	Headway Detection
VORAD – Board / Cable	37	Headway Detection
Network box	43	Vehicle Network
Lane tracker	46	Lane Tracker
Cell phone antenna	8	Remote Vehicle Tracking
Other cables	3	---
Internal (backup) battery	6	Power Control Battery Backup
Broken License Plates	20	---

When did Failures Occur?

Extensive logs detailing the timing of fault detection and fault repair were maintained throughout the study. Assuming that fault detection took a set amount of time (i.e., a week), these logs can help establish the timing of the repairs. No particular differences between catastrophic and major or minor failures were observed in terms of timing. Table 13.4 shows the relative number of failures detected during each month of the study. The first failures were detected in March, 2003, and the last failures were detected a year later. The month with the largest percentage of failures was October 2003. Also note that the percentages increased initially as the number of vehicles on the road ramped-up, and decreased as the vehicles were removed from the road after January 2004.

Table 13.4. Percentage of failures for each month of the study.

Month	Percentage of failures
March, 2003	2.3
April, 2003	3.6
May, 2003	6.6
June, 2003	9.1
July, 2003	8.4
August, 2003	7.9
September, 2003	9.1
October, 2003	19.5
November, 2003	7.9
December, 2003	13.7
January, 2004	4.6
February, 2004	3.8
March, 2004	3.6

When were Failures Detected?

The time needed to detect a failure was reduced by requesting this information from various sources. Data downloaders were required to identify, via a checklist, any aspects of the installation that were visually askew as they downloaded data from the vehicle. Furthermore, participants were also instructed throughout the study to call the study contact person when they believed problems existed with any part of the data acquisition system or with the system’s interaction with their vehicle. In addition, as data was downloaded and transferred to the storage server, a data reductionist was required to observe a subset of the data and point out any problems noted. It is estimated that these three failure identification methods resulted in a failure occurrence to a failure identification average lag time of one week. Once the problem was identified, total downtime became a function of the availability of replacement parts and the replacement time.

What was the Repair Time for Failures and the Failure Rate for Components?

Out of the 45 estimated catastrophic or major failures outlined above, it is estimated that 20 occurred during the first half of the study (response time less than three weeks) and 25 during the second half (response time less than two weeks). Throughout the study, a total of 4,554 vehicle-weeks were collected from a total of 102 vehicles. Thus, the first half of the study resulted in 60 vehicle-weeks of downtime (20 instances X 3 weeks downtime) while the second half resulted in 50 vehicle-weeks (25 instances X 2 weeks downtime). As a result, the total data collection downtime due to catastrophic or major failures was 110 vehicle-weeks. This represents an overall catastrophic of major failure rate of 2.4 percent (110 vehicle-weeks downtime / 4,554 total vehicle-weeks). The 110 vehicle-weeks of downtime represent, based on an assumed weekly mileage rate for the study of 404.0 miles/vehicle-week (assuming 1.84 million VMT for the study and 4,554 vehicle-weeks), a total of 44,444.4 miles of data that were not collected due to catastrophic or major failures.

Catastrophic and major failure rates per sensor or subsystem were derived from Table 13.2 and are shown in Table 13.5. Sensors and subsystems not mentioned in the table did not exhibit any catastrophic or major failures. A total of 4,554 vehicle-weeks of data collection were used in the calculations. In addition, three weeks of downtime is assumed. This assumption is based on adding estimates for the time required to detect a failure (~1 week) and estimates for the time to perform a repair (~2 weeks). This estimate is somewhat conservative, since in many instances it took fewer than 3 weeks to detect and repair a fault, especially in the latter part of the study. Thus, the failure rates presented in this section represent a ceiling for the hardware used in the study.

Table 13.5. Catastrophic or major failure rates by sensor or subsystem.

Failing Sensor/Subsystem	Instances	Failure Rate (%)
Power Control Battery Backup	33	2.2
Acquisition Software	67	4.4
Remote Download	17	1.1
Real-time Video	22	1.4

Minor failure rates per sensor or subsystem, compiled from Table 13.3, are shown in Table 13.6. An assumption of three weeks downtime is used, along with a total data collection period of 4,554 vehicle-weeks. These 268 minor failures represent 804 vehicle-weeks of incomplete data. This means the overall minor failure rate (assuming independent failures and the downtime assumptions used before) was 17.7 percent. A total of 324,816.0 miles of data were incomplete, based on the assumed weekly mileage rate for the study of 404.0 miles/vehicle-week. In some cases, this data could still be used in data reduction because a redundant source of data was available.

Table 13.6. Minor failure rates by sensor or subsystem.

Failing Sensor/Subsystem	Instances	Failure Rate (%)
Power Control Battery Backup	6	0.4
Real-time Video	97	6.4
Headway Detection	45	3.0
Vehicle Network	43	2.8
Lane Tracker	46	3.0
Remote Vehicle Tracking	8	0.5

Overall, all failure rates were relatively low. None of the component specific rates were larger than 10 percent and the majority of the rates were smaller than 5 percent. In addition, many subsystems (i.e., accelerometer, critical incident button, gyroscope, GPS) had overall failure rates of zero.

Assuming that all failures resulted in loss of data, the maximum number of miles lost due to failures would have been near 370,000 miles (the sum of miles lost due to all types of failure). There are data for 1.37 million miles, with an estimated total number of miles of possible data available equal to 1.80 million, a difference of approximately 430,000 miles. The difference between the miles lost due to failure (370,000) and the estimated total miles lost (430,000) is 60,000 miles. This discrepancy is probably due to log discrepancies and a few outlier cases in which the failure detection took an inordinately long time for one reason or another. Overall, however, this comparison justifies the validity of the rates calculated in this section and serves as an error check for any repairs that might have been missed in the logs. While it is possible that some repairs were never entered, the agreement between miles calculated using different approaches supports that the number of these missing repairs would be relatively small (i.e., 60,000/1,800,000 x 100 = 3.3%).

It is important to note that minor failures were also assumed to result in loss of data. However, certain types of failures, such as vehicle network or network box problems, resulted in data that were not analyzed because the trigger criteria relied upon the data generated (e.g., speed). This means that a significant portion of this data can potentially be recovered in the future by estimating the associated parameters post hoc.

The failure rates discussed in this section can now be combined with the benefits that would be expected from each of the subsystems within a large-scale naturalistic study. These benefits are discussed in Questions 2 and 3 from two different perspectives. Question 2 discusses the benefits that could be expected from different sensors, hardware components, and data collection components when compared to their failure rates. Question 3 addresses how these systems can be optimized so that the least amount of hardware produces the best possible results.

---

Question 2. What is the relative cost/benefit for each sensor type and/or hardware component?

---

A large naturalistic study with 5,000 cars on the road for two years would raise a series of unique and important issues. The larger scale of the project makes many of the support mechanisms available during the 100-Car Study not feasible due to: (1) the large geographical area in which a study like this would need to be conducted, and (2) the sheer number of vehicles and drivers that would have to be tracked.

Perhaps the most important of these issues is that chase cars with downloaders would not be feasible. Maintaining a fleet of chase cars and hiring the personnel required would be too costly and time-consuming. Thus, it is foreseen that cars would be released on the road for a period of six months. At the end of six months of data collection, vehicles would return to have the data collection system removed. At that time, the data stored on the vehicle during the data collection period would be downloaded and backed up. Note that there would be an exception to this scheme when a crash was detected. It is envisioned that automated crash detection would be available as part of a 5,000-car system and when a crash was detected, the data would be immediately downloaded.

This approach poses several challenges. First, the DAS within the car must be highly reliable and resistant to data corruption. Otherwise, it would be highly possible that data from many vehicles would be lost due to system failure. Second, the sensors and hardware associated with the system must have low failure rates to increase the opportunity of acquiring the most data possible during the study period. If any of the sensors is failure-prone, then it will be more likely to fail during the course of the study. Since there would be no interaction with the DAS for six-month intervals, sensor failures would not be detected and any events occurring after the sensor failure would be lost permanently. Third, the number of sensors and other hardware components should be reduced to the largest extent possible in order to minimize failure rates and in-vehicle data storage needs. A greater number of sensors would increase the likelihood that the DAS would run out of in-vehicle storage space before the end of the data collection period.

In-vehicle data storage capacity needs can also be addressed by considering the amount of data that are actually collected. Since it was considered to be a pilot study, the 100-Car Study used continuous data collection. In this case, continuous collection was needed to observe all driving behaviors and actions in order to determine which of those actions merited more detailed consideration from a traffic safety perspective. For a large-scale study, this would not be feasible or needed. Thus, data collection would only occur with specific triggers. The development of these triggers and their effectiveness is the focus of Question 3 of this goal and discussion of this issue will be deferred until then. The effectiveness of the sensors and hardware suggested for use on a large-scale study at the end of this section have been verified by the results presented in Question 3.

The number of sensors and other hardware should also be reduced to decrease system failure rates and installation requirements. It is logical to expect that the fewer sensors and other hardware components that are needed, the fewer opportunities there will be for the system to break down and stop collecting data. In addition, fewer hardware components also imply fewer wires and fewer control boards within the DAS, which implies easier installation and perhaps even a smaller DAS form factor. Fewer hardware components also mean that installation might be possible on a larger variety of vehicle makes and models, an essential aspect of a 5,000-vehicle study.

DAS components can be split into three categories for this discussion: in-vehicle data storage, sensing, and other hardware (e.g., video). The benefits for each of these components have to be considered separately. In some cases, the benefits can only be presented subjectively, while in other cases, quantification of the benefits is possible.

In-vehicle data storage was an essential part of the 100-Car Study and would also be a central aspect of any large-scale study. The goal of any of these studies is to obtain data from drivers in a naturalistic setting. If these data are lost, the study has been unsuccessful. Thus, the benefits of data storage are quite large, albeit difficult to quantify, and efforts should be directed toward improving the reliability of this system to the degree possible. For the 100-Car Study, catastrophic and major failures occurred at a rate of 2.4 percent. For a 5,000 vehicle study, this would imply that 120 cars would lose their ability to collect or store data throughout the study. Thus, improvement in this figure is probably desirable.

The next important component of any large-scale DAS is the array of sensors included within the system. The benefit for each of these sensors relies on the number of triggers that the sensor is able to support, as well as the effectiveness of these triggers. Maximizing these aspects of the DAS is the focus of Question 3; however, these aspects are initially discussed here because they are directly relevant to determining the benefit provided by each of the sensors that could be used on a large-scale naturalistic study.

Each of the events (i.e., crashes, near-crashes, and incidents) that were collected for the 100-Car Study was selected as a function of one or more triggers. These triggers, in turn, depended on the sensors or subsystems that collected the data against which the triggers were contrasted. This section examines the effectiveness of each trigger type (and thus each underlying sensor or subsystem type) in correctly identifying a valid event. More details about each of the triggers can be found in the discussion for Question 3.

The database of reduced events included 69 crashes, 761 near-crashes, and 8,295 incidents. Table 13.7 lists each of the sensors or subsystems used in the data acquisition system and the percentage, by severity, of the valid events detected by each trigger. These parameters are indicative of the first measure of benefit for a trigger, which is the proportion of valid events that would be detected. In a large-scale study without continuous data collection, a failure of a trigger to activate for an event would mean the loss of the event. Thus, the higher the percentage of valid events that a trigger captures, the better that particular trigger is considered to be. Note that each sensor is independent in the number of events that are detected, meaning that the percentages are not additive for any column.

Table 13.7. Sensor or subsystem benefits in terms of valid events captured.

	Severity
Sensor/Subsystem	Crashes Detected % (Instances)	Near-crashes Detected % (Instances)	Incidents Detected % (Instances)	Valid Events Detected * % (Severe Instances) (All instances)
Accelerometer (Lateral)	18.8% (13)	3.2% (24)	0.6% (48)	3.5% (85) (316)
Accelerometer (Long.)	58.0% (40)	61.9% (471)	37.6% (3,089)	44.7% (3,600) (4,078)
Critical Incident Button	18.8% (13)	16.0% (122)	5.2% (434)	8.4% (569) (762)
Range/Range Rate Detection – Fwd.	24.6% (17)	42.8% (326)	57.5% (4,768)	56.5% (5,111) (5,158)
Range/Range Rate Detection – Rear	2.9% (2)	7.6% (58)	4.4% (363)	4.6% (423) (424)
Gyroscope (Yaw Rate)	24.6% (17)	25.0% (190)	17.2% (1,423)	21.7% (1,630) (1,983)
Lane Tracker	0.0% (0)	0.5% (4)	0.1% (5)	0.6% (9) (82)
Radar – Side	0.0% (0)	0.3% (2)	3.0% (251)	3.1% (253) (280)
*Severe instances include crashes, near-crashes, and incidents. All instances also include non-conflict incidents. The percentage provided is based on all instances.

Only the forward headway detection sensor was able to identify more than 50 percent of the valid events (56.5%), with the longitudinal acceleration sensor identifying the second largest percentage of valid events (44.7%). When event severities are considered, some could be predicted more than 50 percent of the time by the data from some sensors. The longitudinal accelerometer sensor performed best in crashes, correctly marking as an event 58.0 percent of all of the crashes in the dataset. This sensor was also the most effective in identifying valid near-crash events (61.9%). The forward headway detection sensor correctly triggered for 57.5 percent of all incidents. Finally, all three severity levels had reasonably high detection rates triggered from the gyroscope sensor.

It is important to note that the lane tracker and side radars are not fairly represented in Table 13.7. The side radars were only present on one fifth of the fleet (20 leased vehicles) for 6 months. Thus, their rate should be less than 10 percent of any of the other sensors. However, the inclusion of side radars for a full-scale 5,000 vehicle fleet would probably be impractical due to cost and installation requirements.

However, the lane tracker may present a different case. The lane tracker is software-based and uses the same forward camera already present for the study. While it does increase computer processing requirements, the cost is fairly minimal. In addition, due to the focus of the current study on rear-end crashes, there was not a great deal of effort made to determine the feasibility of using the lane tracker as a trigger. Early attempts did show that the lane tracker signal was noisy both for reasons of road marking visibility and driver behavior (i.e., many drivers exceeded lane boundaries on purpose or relaxed standards in the absence of other traffic). Therefore, a lane tracker may turn out to be a very valuable addition to a study that is more broadly focused and when more time and resources are available to improve the signal filtering.

Another measure of benefit from a trigger relates to its ability to capture only valid events, which in turn indicates the level of noise (i.e., invalid events) that will exist within the stored data in a large-scale study. Triggers that perform poorly in this regard will overload the in-vehicle data storage equipment with useless data and could compromise the capacity of the system to store important events occurring after the in-vehicle data storage unit is full. Table 13.8 shows the rate at which invalid events were found for each trigger. A lower rate indicates better trigger performance for this measure. In addition, the catastrophic/major and minor failure rates were noted. Note that each sensor is independent regarding the number of events detected; thus, the percentages are not additive for any column. Also note that for the side radar, the same failure rates used for headway detection are used, given that the same technology was applied.

Table 13.8. Sensor or subsystem benefits in terms of proportion of invalid events.

Sensor/Subsystem	Invalid Events Detected % (Instances)	Failure Rate (%) (Catastrophic & Major / Minor)
Accelerometer (Lateral)	91.3% (3,325)	0 / 0
Accelerometer (Long.)	66.4% (8,047)	0 / 0
Critical Incident Button	69.9% (1,773)	0 / 0
Headway Detection – Fwd.	83.4% (25,833)	0 / 3.0
Headway Detection – Rear	59.9% (633)	0 / 3.0
Gyroscope (Yaw Rate)	91.1% (20,217)	0 / 0
Lane Tracker	96.1% (2,532)	0 / 3.0
Radar – Side	96.5% (13,808)	0 / 3.0

 

These higher sensitivities to valid events were not necessarily accompanied by good negative predictive ability. The forward headway detection sensor had a rate of invalid events of 83.4 percent, near the top of the list for this category. The longitudinal acceleration sensor fared better, with 69.9 percent of all the events that it triggered being classified invalid (second from the bottom of the list for this category). The rear headway detection sensor had the lowest rate of invalid events, at 59.9 percent.

Perhaps the single most important sensor present in the 100-Car Study data collection system was video. The benefit and value of video is very large, given the aid it provides to data reductionists in determining the validity of triggered events. Given that invalid events seem unavoidable based on current sensor technology (see the discussion for Question 3 for more on this issue), it seems that some video views would be absolutely necessary in a large-scale naturalistic study to aid in the data reduction process. This issue is discussed in more detail under the Goal 10 Report (separate report). However, note that even with a failure rate of 6.4 percent, video is considered an important piece of hardware to be included on a large-scale study due to its data validation benefit. A logical concern would be that with this relatively high failure rate, video might become problematic to include in a large-scale study. While this concern is justified, the failure rate can be reduced by including only camera views that are absolutely necessary (which may also reduce in-vehicle data storage requirements). Justifications for the elimination of certain camera views are presented in the Goal 10 Report .

Other subsystems not shown in Tables 13.7 or 13.8 were not used to collect any trigger information or validate events. Rather, they served to support the data collection function or the remote vehicle tracking function. These functions contributed to the success of the overall system and their absence in this analysis simply indicates that they had no direct bearing on the selection of particular events for further analysis, which is the main source of benefit information. For example, RF and glare sensors were installed in vehicles, but their data were not considered essential or useful in triggering for events. These sensors are part of a category that can be deemed optional, whose use depends on the specificity of the data desired within any larger-scale study. Their usefulness and corresponding benefit lies in providing the ability to characterize events better than by simply looking at a video. However, this ability has to be weighed against their cost, expected effectiveness, and required maintenance. Whereas the effectiveness for the sensors considered above in this section can be determined based on their performance, the effectiveness of optional sensors is more subjective, and depends on the value of the information to the stakeholders for a particular data collection effort. Costs of maintenance did not seem to be high, based on their absence from repair logs. However, some of these sensors can be noisy, and care should be taken in ensuring that the quality of data from any of them included in a large-scale study is sufficient to justify their expense in terms of cost and of in-vehicle data storage.

The data available for all of these in-vehicle data storage components, sensors, and hardware components are limited in terms of benefits, since most of the benefit gathered from the components is difficult to quantify. The costs, in terms of failure rate (i.e., required maintenance and repairs), in-vehicle data storage needs, and classification effectiveness, while somewhat more quantifiable, are also subject to some degree of interpretation.

When benefits and costs are observed as a whole, several technologies stand out for inclusion in a large-scale naturalistic study. Accelerometers, yaw rate sensors, and range/range rate sensors (particularly forward) seem essential to the real-time classification of valid events for a larger-scale study using a trigger-based data collection system. Video is also necessary to allow for screening the invalid events that will inevitably be collected. If these technologies are combined using algorithms that aggregate their data (as discussed in Question 3), they should be able to collect an acceptable number of valid events while minimizing the number of invalid events that are stored and have to be eliminated manually.

Previous | Next | Top | Table of Contents

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS