Human Factors Research from the Center for Operator Performance

COMPLETED PROJECTS

In what began as an ad hoc discussion of the need to address the human factors in process control at a National Petroleum Refiners Association meeting in the fall of 2005, the Center for Operator Performance (COP) has identified and funded numerous pre-competitive research projects to enhance the safety and environmental performance of process industries through the improvement of operator performance.

Topic: GRAPHICS

• Data to Information II (201403) resulted in a 200 page guide that can be used for ideas on how to alter current displays or as a template for an entire refinery display system. Examples of how to represent major pieces of equipment for different levels of a display hierarchy should be useful for any process. Examples are provided of how to combine the major equipment as part of a hierarchy for each major refinery unit.  
• Display Content projects (200903 and 201204) investigated how the information operators need to make decisions should be mapped to operating graphics in order to make recommendations for display redesign.  Results have been successfully utilized by at least two COP members to identify content for Level 1 and Level 2 displays.
• Background Color for HMI (201501) performed a literature review and analysis on outstanding questions regarding how to select an effective pairing of background color and palette colors for HMI graphics. The study provided an experimental design with specific variables to be tested, including glare/ambient lighting, and defined the testing approach. A phase II project is planned to conduct the testing. The experimental design from the first project was tested in Background Color for HMI II (201702), varying lighting conditions, background color, foreground color, and target color. The selection of color schemes can make the difference between operators rapidly recognizing and interpreting information on the display and failing to efficiently detect important indicators, which, if missed, can result in off-spec product, unscheduled downtime, equipment damage, or worse.  A phase 3 project is planned to that will expand potential foreground colors and create a tool for color selection.
• Color Use Projects (200702 and 201003S) suggest that while misuse of color can slow down information processing, the specific presentation of the information given the decisions to be made directly affects performance, and probably more dramatically than color. However, the use of red for stopped and warning showed that it took 25% longer to find alarms when red was also used to indicate motor/valve status as when it was used for alarming alone.
• Console Operator Situation Awareness is critical to knowing “Is my process OK?” and “Is it running at the expected targets?” The COSA Overview Display Design (201206) project identifies the core characteristics required to design an effective situation awareness overview display for an operator’s scope of responsibility. The study also recommends proven methods for determining display content, structure, and navigation. The design guidelines provide a means to create a COSA overview display that enables the console operator to answer these two critical questions in four (4) seconds or less.Learn more about this project in the September 2015 Project Summary Console Operator Situation Awareness (COSA): Designing an Effective COSA Overview Display
• Data Entry Devices (201306S) assessed the accuracy and speed of response for entering data through two different devices (traditional keyboard and mouse versus integrated mouse) and three different input methods (manual data entry, ramp keys, slider bar) under time pressured and unpressured scenarios. The goal was to determine if errors could be minimized and speed of response improved with a particular device or method.
• Display Evaluation Toolkit I (201001) analyzed human performance of two interfaces at different levels of workload. The analysis examined previously established metrics of speed and accuracy and additional measurements that influence performance (situation awareness, eye movement, and subjective workload ratings). The work continued in Display Evaluation Toolkit II (201201) that tested several human factors techniques on old and new style graphics to show how performance differences can be measured. Some of the techniques are simple paper-and-pencil, while others utilize the Center’s eye tracking software.
• Display Type Impact (201406S) This student project assessed the impact of information format and task complexity on performance. An experiment was conducted in which subjects were asked to conduct tasks of varying complexity with object oriented, emergent features, and numerical displays.  Consistent with other studies, the emergency feature displays (analog bars) produced faster recognition of process variance than did the object displays (radar/spider plots).
• Vector Arrows (201505S) This student project assessed the impact of using vector arrows that show rate of change information versus trends. An experiment was conducted in which students were asked to identify changes to process variables over various rates of change, various time periods, and different conditions (stable vs. unstable). Vector arrows were best when averaging over a long time period (2 hours) with an angle of 10 degrees.
• Trend Performance (201506S) The intent of this project was to assess the performance of trends in identifying process changes. Subjects were asked to detect changes in process variables with either numeric data or process trends. Although some of the results were inconclusive, trends produced lower mental workload and better results only when the rate of change was large.
• Optimum Pump Symbol Size (201105S) suggests maximum discrimination and information transfer are obtained with a symbol 0.48” for normal viewing distances.

Topic: PROCEDURES AND JOB AIDS

• Procedure Modularization (201703) set out to determine if the use of modularized emergency procedures makes it easier to review and/or revise procedure content for accuracy and applicability.  The project was unable to re-create the original procedures as intended due to the high degree of variability in those procedures as far as the individual steps. While the modularization of the specific procedures under the specific protocol of this project could not be achieved, procedure modularization still appears to be a viable means to improve procedure consistency and ease of updating.
• Procedure Warnings (201503) The nature of a diverse international and linguistic workforce argues for the use of symbols rather than text for conveying warnings in procedures. The study examined different symbols and warnings for individuals for whom English was their first language and those for whom it was their second. While the symbols had little impact in perception of hazard, those for whom English was not their first language performed worse than for those whom English was their first language. Failures were at a minimum of 10% across both groups. 
• Decision Aids (201002S) found that, the use of checklists, procedures, and/or line demarcation can reduce the probability of an improper valve line-up from near 50% to less than 5%.
• Knowledge Management Projects (200901P and 201004) developed software that can read procedures and convert into a tabular format. The Procedure Analyzer can be used for procedure automation or creating easier to use procedure modules. Learn more about these projects in the July 2015 Chemical Processing articles Bring Your Procedures into the 21st Century and Plants Grapple with Graying Staff.

Topic: WORKSTATION

• Operator Workspace (201502) A systematic approach to design of workstations was developed. Key steps in such a process were tested on the HMI at a pulp and paper mill. The results showed opportunities to reduce the number of monitors and improve screen layout. The level of effort for the analysis was deemed to be too high for use on all workstations.
• Hand-helds (201303) assessed the presentation of intelligent information on mobile devices and the applicability of this technology in process control. The alarm response study found use of mobile devices can be effective: mobile devices were easier to use and provided better interaction modality, operators effectively controlled while maintaining situational awareness of the process information, and tasks were completed in significantly less time in the mobile system versus the traditional system.
• Large Screen Survey (201405) A survey of both member and non-member companies was conducted to determine how large screens are or are not being effectively used in the petrochemical industry and the reasons why. In addition to size and number of monitors, the survey elicited input on the intent of the monitors, the formatting of the data, and suggestions for use. This survey was open to any operating facility and company, with results provided to all who participate.
• Large Screen Performance projects (201203S and 201205S) found that information on a larger screen placed further away can result in longer response times than when the data is closer. Use of large screen monitors that have no more information than smaller monitors at the operator’s workstation should be avoided.
• Alarm Sounds (201802) All environments where alarms are used are governed by two key elements: what event is making the sound, and the urgency of the situation being signalled.  Typically there is some interdependency of these factors. This was a follow on project from Alarm Tones (201604P) under Alarms below.  The result of this project was an Auditory Mapping Protocol that provides a way of developing and planning a consistent and rational auditory alarm design and implementation approach for control rooms of varying size.

Topic: ALARMS

• Alarm Tones (201604P) A pilot project to understand the auditory requirements of a control room, enable basic spectral measurement of the various alarms’ acoustic spectral emissions in narrow bands, and establish audio recording of the alarms.  Follow on project is Alarm Sounds (201802), see above under Workstation.
• Highly Managed Alarms (HMA) Survey (201605) A survey that covered multiple industries and continents found that there is high variability in the use of highly-managed alarms as defined in ISA-18.2. Some companies were still developing how they would handle safety related alarms that would normally be considered HMA. Other companies said they intentionally avoided the term HMA although they did have a special class of alarms for safety related alarms. 
• Alarm Formatting (201302) investigated current industry HMI alarm environments through plant site and vendor visits, concluding that the same fundamental technology is universally used throughout industry. Additionally, there is no apparent emerging new technology to change the current paradigm. The study suggests some concepts for consideration in designing future HMI alarm environments.
• Alarm Rates Projects (200801 and 200904) suggest novice operators can handle 10 alarms in ten minutes as easily as lower rates. Expert operators respond to alarms twice as fast as novices, but only at high (20 alarms / 10 minutes) rates. At low rates, their performance is similar to the novice. Learn more about these projects in the December 2011 Chemical Processing article How Many Alarms Can an Operator Handle?

Topic: AUTOMATION

• Alarm and Automation Impact (201901): A COP working subgroup and researcher worked to examine the effects of level of automation and the quality of the alarm systems on operator and system performance.
• Event Prediction Projects (200902, 201103, and 201304) provided identifying precursors that lead up to an unplanned event and suggested guidelines for developing displays and navigational aids to support operators. A pilot study included parallel coordinate plotting and demonstrated the ability to identify precursors to compressor surge. Subsequent research (201401) led to the development of a geometric analysis tool that could be used to clearly display the approaching abnormal event to board operators. Using historic data, the system they developed was able to identify a flare event two hours prior to the event occurring. Additional work was done using historic compressor surge data, which resulted in the surge being identified 60 minutes prior to surge. The final deliverable of the research is a software tool that, using their algorithm, can be used to identify the variables and produce the related radial plot diagram. Learn more about these projects in the July 2014 News Release New event detection tool 90% faster than traditional

Topic: SKILLS/KNOWLEDGE

• Expert Operator (200601P): Nature of expertise in control of continuous processes suggested automation can diminish expertise by inflicting three levels of damage: (1)disable the expertise of those who are already skilled, (2) slow the rate of learning, and (3) teach dysfunctional skill that will actively interfere with the ability to achieve expertise, by (a) limiting operators ability to understand relationships and identify data shifts, (b) hindering the ability to understand how process works by making it invisible, (c) not requiring the operator to form their own assessment by removing them from the process and only providing recommendations and alerts, and/or (d) hindering the ability to spot anomalies and patterns by removing variance from representations.
• Experts Use of Display Cues Projects (201101P and 201104) found that use of trend data was critical to expert decision making, raising issues of those locations that do not use trends.
• Incident Database Mining (201402) analyzed public data related to incident reports on catastrophic events in the Oil, Gas, and Chemical Industries, attempting to identify operator actions related to human performance issues reported as factors contributing to the incidents. The data mining technique, using word cloud analysis, was successful in identifying the reported human errors associated with incidents. Unfortunately, the categories used for human error factors did not clearly indicate root causes in terms of human performance issues.
• Mental Models (201602) was an effort to determine 1) if it was possible to formalize a panel operator’s mental model(s), 2) detect potential deficiencies, and 3) how the mental models impacted the operator’s ability to troubleshoot process problems.  The initial results revealed that poor mental models were not the cause of trouble-shooting difficulty but the inability to activate the model when needed. The follow-on project is Cognitive After Action Review Guide for Observers (CAARGO) (see 201803 under Training below).
• Mental Models-ARA (201102P): Causal Mental Models of Plant Operations demonstrated graphically how experts use deeper layers of mental model to troubleshoot process.

Topic: TRAINING

• CAARGO Video (201903) was developed (Cognitive After-Action Review Guide for Observers (CAARGO)) as a training video. This video illustrates a trainer employing CAARGO methods and can be used to both introduce CAARGO to trainers as well as provide exercises for trainers to test their knowledge of CAARGO concepts.
• Cognitive After Action Review Guide for Observers (CAARGO) (201803) was developed to assist trainers in understanding trainee mental models, helping them discover how they arrived at their decisions and understanding any errors.  The guide was tested during White Board sessions, Simulator training and Tabletop exercises.  This was the continuation of the Mental Models project (see 201602 above under Skills/Knowledge).  This project resulted in a nine-page guide, a one-page summary sheet and a wallet card with highlights.  The production of a CAARGO Video (201903) has been approved as a follow-on project.
• Effectiveness of Scenario-Based Training and the DMX Train the Trainer Video. If you want operators to be good decision makers, then they need to practice making decisions. That is the purpose of Decision Making Exercises (DMX) (200703) adapted from the military for console operators. The exercises can be done in an hour, allowing training to occur before or after the normal shift, not requiring the operators to come in on a day off.  DMX Manual (201504) A user’s manual has been created on how to develop and apply decision making exercises. A tutorial video is included that shows how to facilitate the exercises.
• Evaluation of TeamSTEPPS (201905) reviewed the TeamSTEPPS material, a CRM related training program, to determine if the training could apply to our industries and if so, whether or not we would require an outside consultant to help develop the training.
• Fixation Workshop (202001) Fixation or falling victim to diagnostic errors is one of the cognitive barriers we described in the original Mental Models effort (see 201602 above under Skills/Knowledge). This barrier is especially problematic when operators attempt to troubleshoot a novel situation. Fixation occurs when someone forms an initial assessment or hypothesis of what is going on and then either holds onto this hypothesis without testing it or in light of contrary evidence. When someone is fixating on an incorrect explanation for a problem, there should be anomalies that don’t fit this explanation. However, these anomalies often go unnoticed or one’s fixation usually drives them to explain the anomaly away.
• Operator Training Benchmark (201704) found that while almost all participating companies were generally pleased with the product of their training programs (i.e. trained operators), wide variance exists in how it is achieved.  The data came from multiple companies and multiple locations within a company.  The final report covers total time to complete all training, initial (basic) training and on-boarding, position training, advanced position training (up-skilling), console training and training system.  The project recommended three follow-on projects to cover training next generation of operators, operator of the future and a training program assessment.   
• ShadowBox Training (201305P) assessed the feasibility of using the ShadowBox method to capture the tacit knowledge of experts and be an effective training method for process plant operators. Using training scenarios designed for console operators, this pilot study demonstrated that the ShadowBox method can be successful in capturing SME tacit knowledge and transferring that knowledge to less experienced trainees. The project recommended continued study of the ShadowBox method to determine if generalized scenarios, focused on cognitive skills, can be effective across companies and industry segments, particularly for developing new console operators more quickly. Learn more about this project in the July 2015 Chemical Processing article Plants Grapple with Graying Staff.
• ShadowBox Train the Trainer (201601) created a comprehensive user guide for trainers and supervisors to independently create and deliver cognitive training techniques, such as Decision Making Exercises (DMX) and ShadowBox, which accelerate and enhance novice operator abilities to identify critical problems during troubleshooting scenarios. Included in the guide are strategies for knowledge capture and transfer, suggested methods to integrate ShadowBox and DMX training into an organization’s existing training modules (e.g., on-the-job training, simulation training), and the organization and distribution of expertise scenario-based methods.
• ShadowBox Training Method II (201404) answered questions regarding the flexibility and effectiveness of ShadowBox training for companies seeking to use ShadowBox to accelerate expertise.  Integration of ShadowBox with high fidelity simulation improved the operators’ ability to troubleshoot problems by over 25%. A web-based tool for Shadowbox was developed, but unable to be tested due to constraints at the operating company.
• ShadowBox with Generic Simulation (201701P) deployed ShadowBox with a generic simulation of a process unit: trainee scores improved by almost 10%.  The operators responded positively to the training and specifically made note of the value they felt the exercises held for both novices and experienced operators.
• Simulator Survey (200701): Effective Simulator Usage Research has shown that neither (1) high fidelity nor (2) high user favorability ratings guarantee transfer to actual job/task.
• Training Methods (201301P) assessed training processes in the petrochemical industry and compared to DoD standards. The focus was on training programs for console operators to help manage turnover by training in the most effective and efficient manner. The project included recommendations for fundamental training, team performance, expertise, and training strategies oriented to organizational culture and priorities. Learn more about this project in the October 2014 Project Summary Process Control Operator Training: Identifying Needs and Research Priorities
• Training NextGen Ops (201902P) evaluated and determined the training strategies, medium, and curriculum to be considered for the next generation of operators.
• Training Systems (201603) was a survey of industry training professionals and operators from across the industry. Fourteen training trends were identified, with process control operator training focusing most often on technical skills and relying primarily on observation-based learning with hands-on practice using operational systems. Possible training gaps that may be an obstacle to preparedness include relatively inexperienced operators training new operators, a low use of simulation-based training, and low emphasis of non-technical skills (e.g., teamwork).

Topic: WORKLOAD

• Console Workload (201801P) A key goal of the project is to develop a red line of operator workload that will predict when performance will break down because of overload, typically occurring during times of crisis or system failure.  In this pilot effort, an approach was developed that described how to predict and validate the red line, employing human factors findings from other industrial domains and tailoring those to the refinery/pipeline industry.  

Topic: JOB DESIGN

• Fatigue Data Mining (201005) found that, at eight continuous days worked, the probability of making a mistake doubles. At 11 days in a row, it increases by orders of magnitude.

Topic: ORGANIZATION

• Board Field Collaboration (201806P) assessed the current status of communication and collaboration between board operators and field operators.

Complete reports and raw data available in the Members-Only section

CURRENT RESEARCH

RESEARCH PROPOSALS

• Background Color II.25
• Crew Resource Management (CRM) Needs Analysis
• Event Prediction Tool User Manual
• Safety Climate Benchmark