Mike Nartker
NS&D Monitor
3/28/2014

A set of self-assessments of safety conscious work environment (SCWE) performed by Department of Energy sites and contractors suffered from several weaknesses that may reduce their overall usefulness, according to a report released last week. The DOE Office of Health, Safety and Security performed an evaluation of the self-assessments and found several issues, including inadequate guidance, inconsistent application of assessment methodologies and poor communication of results to senior management. “The overall approach ultimately used to self-assess SCWE across the complex did not provide for consistent application of assessment methodologies and was not designed to ensure validity and credibility,” the report says. “The wide variation in the quality of methodologies and analysis of results significantly reduces the confidence in the conclusions of many of the self-assessments. Consequently, caution should be used in drawing firm conclusions about the state of SCWE or safety culture across the entire DOE complex based on a compilation of results from all the site self-assessments.” DOE did not respond to requests for comment on the report this week.

The self-assessments were performed as part of DOE’s response to a formal recommendation issued by the Defense Nuclear Facilities Safety Board in 2011 concerning the safety culture at the Hanford Waste Treatment Plant. The self-assessments were intended to help DOE determine if safety culture concerns existed at other sites. Among the problems HSS found with how the self-assessments were performed, though, was a lack of adequate guidance that led to many sites and organizations developing their own data collection approaches with “widely varying results.” The report says, “Examples of problems in the guidance include the lack of established criteria or qualification standards for a safety culture SME, data collection and administration protocols, and data analysis expectations and documentation. In some cases, data analysis was biased or influenced by earlier data streams (using survey results to identify and focus interviews), and in other cases the documentation of data analysis in reports was too limited to provide credibility to the results.”

HSS also found that many sites and organizations “tended to positively bias the results” of the self-assessments to senior management and, as a result, tended to play down concerns over perceived retaliation or retribution. “In many of the self-assessment reports, the overall conclusions did not accurately reflect the information in the data and analysis sections. In some cases, negative results were presented with a statement rationalizing or minimizing the issue, rather than indicating a need to find out more about the issue and resolve it. In other cases, although data and/or analysis reflected potential problems, those problems were not mentioned in the conclusions or executive summaries, which senior management is most likely to read,” the report says.

The report goes on to state, “Senior managers across the complex need a balanced set of results to understand the overall SCWE and safety culture health of the organizations and need to acknowledge and accept those results as existing perceptions among their organizations. Providing overly positive conclusions that do not fully reflect the potential concerns identified in the assessment gives the  perception of compromising the results for the sake of appearance and is not indicative of learning organizations committed to acknowledging and addressing problems in safety culture.”

‘All Sites Benefitted From The Experience’

Despite the problems found with the self-assessments, the exercise was not completely without merit, according to HSS. “Although the overall effort in performing SCWE self-assessments varied, all sites benefitted from the experience,” the report says. “The most obvious result of the SCWE self-assessments across the complex was an increased awareness, knowledge, and understanding of safety culture concepts, particularly SCWE. The self-assessments provided an opportunity for organizations to learn and improve their overall culture. Many of the organizations learned things they did not know before or obtained more clarity on known issues. Many sites integrated the self-assessment into existing assessment processes and/or modified existing processes to include safety culture attributes, thereby making the self-assessment a more sustainable and ongoing indicator of the health of the site culture.”