Project: CONTXT02 — Context-Related Reliability of Automation and Automation Failure Detection

Partner: NSERC
Dates: January 2003 - August 2004
Participants: Jamieson, G., Bagheri, N.
Description: We examined the question of how automation reliability affects human performance in a multi-task environment. Subjects are asked to perform two manual tasks using a flight simulation software (The MAT battery): a tracking task, and a fuel management task. At the same time, subjects had to monitor and detect occasional failures of an automated system that controls four gauges. An eye-tracking system, ERICA, was used to determine how participants allocated their attention across the different tasks.
The goals of this research were twofold.
The first goal was to further investigate the issue of complacency or over trust in automation. To date, most of the literature on complacency had used the detection rate of automation failures as an indication of the monitoring behavior. As automation failures started to go undetected by participants, they were said to show sign of complacency. Our first study replicated one of the studies where complacency was observed, but this time with eye movement recording. As in previous studies, monitors of constant high-reliability automation performed poorly compared to participants in other conditions. However, whereas this performance might previously have been attributed to complacency, eye movement recordings showed that, following an initial lag in sampling of the automated task, participants in the constant high reliability condition began sampling more frequently – eventually matching sampling patterns of other participants. This observation tends to argue against complacency as an explanation for the poor monitoring performance. That no differences in self-reported trust in the automation between groups were observed further discounts the complacency/overtrust attribution.
The second goal of the study was to investigate the effect of context-related automation failures on operator reliance. To date, most of the existing literature on human-automation interaction dealt with random failures of the automation. Furthermore, participants were usually not given any information about how and why the automation might fail. However, in real settings, certain phases of operation or particular environmental conditions are more or less likely to be associated with automation failures, and operators usually know it. Thus, in our second study, participants were given information about the automation reliability, and why it might fail. In contrast to the first the first study, the performance of monitors of constant high reliability automation was indistinguishable from that of all other groups. Moreover, eye tracking data showed that the monitoring behavior of these participants was more consistent and more frequent than in the first study. These observations suggest that providing operators with information about the context sensitive nature of automation reliability might alleviate the monitoring performance decrement associated with highly reliable automation.

View PDF Bagheri, N., & Jamieson, G. A., "The impact of context-related reliability on automation failure detection and scanning behaviour," Proceedings of the IEEE 2004 International Conference on Systems, Man and Cybernetics, pp. 212-217, 2004.

View PDF Bagheri, N., Mohite, S., Junta, D., & Jamieson, G. A., "Influence of context-related reliability on automation failure detection performance," Proceedings of the 22nd European Annual Conference on Human Decision Making and Control, pp. 41-47, 2003.

« View all projects