Human Factors Research from the Center for Operator Performance

COMPLETED PROJECTS

In what began as an ad hoc discussion of the need to address the human factors in process control at a National Petroleum Refiners Association meeting in the fall of 2005, the Center for Operator Performance (COP) has identified and funded numerous pre-competitive research projects to enhance the safety and environmental performance of process industries through the improvement of operator performance.

Topic: GRAPHICS

  • Display Content projects (200903 and 201204) investigated how the information operators need to make decisions should be mapped to operating graphics in order to make recommendations for display redesign.  Results have been successfully utilized by at least two COP members to identify content for Level 1 and Level 2 displays.
  • Background Color for HMI (201501) performed a literature review and analysis on outstanding questions regarding how to select an effective pairing of background color and palette colors for HMI graphics. The study provided an experimental design with specific variables to be tested, including glare/ambient lighting, and defined the testing approach. A phase II project is planned to conduct the testing.
  • Color Use Projects (200702 and 201003S) suggest that while misuse of color can slow down information processing, the specific presentation of the information given the decisions to be made directly affects performance, and probably more dramatically than color. However, the use of red for stopped and warning showed that it took 25% longer to find alarms when red was also used to indicate motor/valve status as when it was used for alarming alone.
  • Console Operator Situation Awareness is critical to knowing “Is my process OK?” and “Is it running at the expected targets?” The COSA Overview Display Design (201206) project identifies the core characteristics required to design an effective situation awareness overview display for an operator’s scope of responsibility. The study also recommends proven methods for determining display content, structure, and navigation. The design guidelines provide a means to create a COSA overview display that enables the console operator to answer these two critical questions in four (4) seconds or less.Learn more about this project in the September 2015 Project Summary Console Operator Situation Awareness (COSA): Designing an Effective COSA Overview Display
  • Data Entry Devices (201306S) assessed the accuracy and speed of response for entering data through two different devices (traditional keyboard and mouse versus integrated mouse) and three different input methods (manual data entry, ramp keys, slider bar) under time pressured and unpressured scenarios. The goal was to determine if errors could be minimized and speed of response improved with a particular device or method.
  • Display Content I (200903) investigated how the information operators need to make decisions should be mapped to operating graphics in order to make recommendations for display redesign.
  • Display Evaluation Toolkit I (201001) analyzed human performance of two interfaces at different levels of workload. The analysis examined previously established metrics of speed and accuracy and additional measurements that influence performance (situation awareness, eye movement, and subjective workload ratings). The work continued in Display Evaluation Toolkit II (201201) that tested several human factors techniques on old and new style graphics to show how performance differences can be measured. Some of the techniques are simple paper-and-pencil, while others utilize the Center’s eye tracking software.
  • Display Type Impact (201406S) This student project assessed the impact of information format and task complexity on performance. An experiment was conducted in which subjects were asked to conduct tasks of varying complexity with object oriented, emergent features, and numerical displays.  Consistent with other studies, the emergency feature displays (analog bars) produced faster recognition of process variance than did the object displays (radar/spider plots).
  • Vector Arrows (201505S) This student project assessed the impact of using vector arrows that show rate of change information versus trends. An experiment was conducted in which students were asked to identify changes to process variables over various rates of change, various time periods, and different conditions (stable vs. unstable). Vector arrows were best when averaging over a long time period (2 hours) with an angle of 10 degrees.
  • Trend Performance (201506S) The intent of this project was to assess the performance of trends in identifying process changes. Subjects were asked to detect changes in process variables with either numeric data or process trends. Although some of the results were inconclusive, trends produced lower mental workload and better results only when the rate of change was large.
  • Optimum Pump Symbol Size (201105S) suggests maximum discrimination and information transfer are obtained with a symbol 0.48” for normal viewing distances.

Topic: PROCEDURES AND JOB AIDS

  • Procedure Warnings (201503) The nature of a diverse international and linguistic workforce argues for the use of symbols rather than text for conveying warnings in procedures. The study examined different symbols and warnings for individuals for whom English was their first language and those for whom it was their second. While the symbols had little impact in perception of hazard, those for whom English was not their first language performed worse than for those whom English was their first language. Failures were at a minimum of 10% across both groups. 
  • Decision Aids (201002S) found that, the use of checklists, procedures, and/or line demarcation can reduce the probability of an improper valve line-up from near 50% to less than 5%.
  • Knowledge Management Projects (200901P and 201004) developed software that can read procedures and convert into a tabular format. The Procedure Analyzer can be used for procedure automation or creating easier to use procedure modules. Learn more about these projects in the July 2015 Chemical Processing articles Bring Your Procedures into the 21st Century and Plants Grapple with Graying Staff.

Topic: WORKSTATION

  • Operator Workspace (201502) A systematic approach to design of workstations was developed. Key steps in such a process were tested on the HMI at a pulp and paper mill. The results showed opportunities to reduce the number of monitors and improve screen layout. The level of effort for the analysis was deemed to be too high for use on all workstations.
  • Hand-helds (201303) assessed the presentation of intelligent information on mobile devices and the applicability of this technology in process control. The alarm response study found use of mobile devices can be effective: mobile devices were easier to use and provided better interaction modality, operators effectively controlled while maintaining situational awareness of the process information, and tasks were completed in significantly less time in the mobile system versus the traditional system.
  • Large Screen Survey (201405) A survey of both member and non-member companies was conducted to determine how large screens are or are not being effectively used in the petrochemical industry and the reasons why. In addition to size and number of monitors, the survey elicited input on the intent of the monitors, the formatting of the data, and suggestions for use. This survey was open to any operating facility and company, with results provided to all who participate.
  • Large Screen Performance projects (201203S and 201205S) found that information on a larger screen placed further away can result in longer response times than when the data is closer. Use of large screen monitors that have no more information than smaller monitors at the operator’s workstation should be avoided.

Topic: ALARMS

  • Alarm Tones (201604P) A pilot project to understand the auditory requirements of a control room, enable basic spectral measurement of the various alarms’ acoustic spectral emissions in narrow bands, and establish audio recording of the alarms.
  • Highly Managed Alarms (HMA) Survey (201605) A survey that covered multiple industries and continents found that there is high variability in the use of highly-managed alarms as defined in ISA-18.2. Some companies were still developing how they would handle safety related alarms that would normally be considered HMA. Other companies said they intentionally avoided the term HMA although they did have a special class of alarms for safety related alarms. 
  • Alarm Formatting (201302) investigated current industry HMI alarm environments through plant site and vendor visits, concluding that the same fundamental technology is universally used throughout industry. Additionally, there is no apparent emerging new technology to change the current paradigm. The study suggests some concepts for consideration in designing future HMI alarm environments.
  • Alarm Rates Projects (200801 and 200904) suggest novice operators can handle 10 alarms in ten minutes as easily as lower rates. Expert operators respond to alarms twice as fast as novices, but only at high (20 alarms / 10 minutes) rates. At low rates, their performance is similar to the novice. Learn more about these projects in the December 2011 Chemical Processing article How Many Alarms Can an Operator Handle?

Topic: AUTOMATION

  • Event Prediction Projects (200902, 201103, and 201304) provided identifying precursors that lead up to an unplanned event and suggested guidelines for developing displays and navigational aids to support operators. A pilot study included parallel coordinate plotting and demonstrated the ability to identify precursors to compressor surge. Subsequent research (201401) led to the development of a geometric analysis tool that could be used to clearly display the approaching abnormal event to board operators. Using historic data, the system they developed was able to identify a flare event two hours prior to the event occurring. Additional work was done using historic compressor surge data, which resulted in the surge being identified 60 minutes prior to surge. The final deliverable of the research is a software tool that, using their algorithm, can be used to identify the variables and produce the related radial plot diagram. Learn more about these projects in the July 2014 News Release New event detection tool 90% faster than traditional

Topic: SKILLS/KNOWLEDGE

  • Expert Operator (200601P): Nature of expertise in control of continuous processes suggested automation can diminish expertise by inflicting three levels of damage: (1)disable the expertise of those who are already skilled, (2) slow the rate of learning, and (3) teach dysfunctional skill that will actively interfere with the ability to achieve expertise, by (a) limiting operators ability to understand relationships and identify data shifts, (b) hindering the ability to understand how process works by making it invisible, (c) not requiring the operator to form their own assessment by removing them from the process and only providing recommendations and alerts, and/or (d) hindering the ability to spot anomalies and patterns by removing variance from representations.
  • Experts Use of Display Cues Projects (201101P and 201104) found that use of trend data was critical to expert decision making, raising issues of those locations that do not use trends.
  • Incident Database Mining (201402) analyzed public data related to incident reports on catastrophic events in the Oil, Gas, and Chemical Industries, attempting to identify operator actions related to human performance issues reported as factors contributing to the incidents. The data mining technique, using word cloud analysis, was successful in identifying the reported human errors associated with incidents. Unfortunately, the categories used for human error factors did not clearly indicate root causes in terms of human performance issues.
  • Mental Models-ARA (201102P): Causal Mental Models of Plant Operations demonstrated graphically how experts use deeper layers of mental model to troubleshoot process.

Topic: TRAINING

  • Effectiveness of Scenario-Based Training and the DMX Train the Trainer Video. If you want operators to be good decision makers, then they need to practice making decisions. That is the purpose of Decision Making Exercises (DMX) (200703) adapted from the military for console operators. The exercises can be done in an hour, allowing training to occur before or after the normal shift, not requiring the operators to come in on a day off.  DMX Manual (201504) A user’s manual has been created on how to develop and apply decision making exercises. A tutorial video is included that shows how to facilitate the exercises. 
  • ShadowBox Training (201305P) assessed the feasibility of using the ShadowBox method to capture the tacit knowledge of experts and be an effective training method for process plant operators. Using training scenarios designed for console operators, this pilot study demonstrated that the ShadowBox method can be successful in capturing SME tacit knowledge and transferring that knowledge to less experienced trainees. The project recommended continued study of the ShadowBox method to determine if generalized scenarios, focused on cognitive skills, can be effective across companies and industry segments, particularly for developing new console operators more quickly. Learn more about this project in the July 2015 Chemical Processing article Plants Grapple with Graying Staff.
  • Simulator Survey (200701): Effective Simulator Usage Research has shown that neither (1) high fidelity nor (2) high user favorability ratings guarantee transfer to actual job/task.
  • Training Methods (201301P) assessed training processes in the petrochemical industry and compared to DoD standards. The focus was on training programs for console operators to help manage turnover by training in the most effective and efficient manner. The project included recommendations for fundamental training, team performance, expertise, and training strategies oriented to organizational culture and priorities. Learn more about this project in the October 2014 Project Summary Process Control Operator Training: Identifying Needs and Research Priorities

Topic: JOB DESIGN

  • Fatigue Data Mining (201005) found that, at eight continuous days worked, the probability of making a mistake doubles. At 11 days in a row, it increases by orders of magnitude.

Complete reports and raw data available in the Members-Only section

CURRENT RESEARCH

  • Background color II (201702): Michael Murdoch and S. P. Farnand of Rochester Institute of Technology will be improving the understanding of principles involved in selecting display panel color palettes to increase operator effectiveness.
  • Data to Information II (201403): Mark Nixon will lead an internal group to continue work on the previous project, Display Design Handbook.
  • Mental Models (201602): Dr. Gary Klein, Joey Borders and John Schmidt of MacroCognition will be investigating whether or not an operator’s mental model can be assessed to identify gaps or errors that can be corrected and why gaps in mental models occur.
  • Operator Training Benchmark (201704): A working subgroup of COP members and representatives from other companies will be working together to create a benchmark of the current state of initial operator training in our industries.
  • Procedure Modularization (201703): Dave Strobhar and Kim Elkins of Beville Engineering will be working to determine if the use of modularized emergency procedures makes it easier to review and/or revise procedure content for accuracy and applicability.
  • ShadowBox Train the Trainer (201601): Dr. Gary Klein, Dr. Corinne Wright, Joey Borders and Emily Newsome of ShadowBox, LLC will be developing a comprehensive user guide for trainers and supervisors to independently create and deliver cognitive training techniques such as Decision Making Exercises (DMX) and ShadowBox.
  • ShadowBox Training Method II (201404): Gary Klein, Ph. D. of MacroCognition LLC will be continuing research into the use of ShadowBox training techniques in training console operators. In this phase he will be focusing on using ShadowBox training methods to help new console operators gain necessary cognitive skills more quickly as well as how to use ShadowBox training in conjunction with existing training techniques, particularly the use of simulators.
  • ShadowBox with Low Fidelity Simulation (201701P): Dr. Gary Klein and Joseph Borders of MacroCognition LLC will be evaluating if the Shadowbox technique can be combined with low-fidelity simulation to improve operator decision making in an experiential environment.
  • Training Systems (201603): Dr. Beth Blickensderfer of Embry-Riddle Aeronautical Institute will be assessing current training practices in process control operator training and compare those practices with the current training standards in aviation and the military.

RESEARCH PROPOSALS

  • Console Workload
  • Event Prediction Tool User Manual