Collection and Analysis of Safety Data
Any data about aviation operations that can be used to learn how to improve safety, can be considered "safety data." HP insights gathered from safety data can provide information on how the humans in the system contributed, both positively and negatively, to the various outcomes, and importantly – why they acted the way they did. This information could suggest important improvements to the design of technology, procedures, and training, as well as help identify regulatory gaps. Moreover, such information is critical in the identification of systemic issues and could lead to significant system-level improvements beyond minor adjustments to procedures or training.
Learning to improve safety, and therefore also improve effectiveness, efficiency and overall system performance, should draw on all possible sources of operational data. Operational data can and should be collected from day-to-day operations to identify possible hazards and to improve mitigations or work practices. Collection of this kind of safety data requires engaging with frontline personnel to identify issues and fix them collectively. This allows organisations to engage in very proactive safety management activities, such as risk assessing their compliance audit findings and performing hazard identification workshops. However, more often, detailed safety data are only collected after some safety occurrence, whether that safety occurrence is an accident, an incident, or a close call (aka a "near event"). This section focuses mainly on the collection of operational safety data and its analysis and interpretation from an HP perspective.
Further information on HP considerations when collecting and analyzing data, whether following a safety occurrence or not, can be found in the HP Manual for Regulators (Doc 10151).
1. What are some sources of people-generated safety data?
Mandatory reporting systems
A mandatory reporting system supports a process in which operational personnel are required to report accidents and certain types of incidents to enable organizational learning and to be able to mitigate potential safety deficiencies. The provision in Chapter 8 of ICAO Annex 13 requires the States to establish mandatory occurrence reporting systems to facilitate the collection of information on actual or potential safety deficiencies. States are expected to adopt laws to adhere to the provisions outlined in Chapter 7 of Annex 19 for the protection of safety data, safety information, and related sources.
Voluntary reporting systems
In a voluntary reporting system, the reporting person, without any administrative or legal obligation to do so, submits a voluntary report. A voluntary reporting system can be considered in both reactive and proactive terms. The voluntary reporting of a safety occurrence can be considered reactive, because the report is contingent on an event happening first. The lessons learned from data collected about that safety occurrence may be used proactively to prevent similar safety occurrences. The voluntary reporting of an issue or a hazard for the purposes of avoiding a potential safety occurrence is purely proactive.
Chapter 5 of ICAO Annex 19 further requires States to establish a voluntary safety reporting system to collect safety data and safety information not captured by mandatory safety reporting systems. In addition to meeting the Annex 19 requirements for the protection of safety data, safety information and related sources, confidentiality should be ensured in voluntary reporting systems.
Post-occurence interviews
Conducting interviews immediately after a safety occurrence is an important process that helps investigators gather details about the occurrence. It is important to recognize that interviewees who have experienced a serious event or even a near miss are likely to be strongly affected. Due to limitations on attention and perceptual mechanisms, those involved in an accident or serious incident cannot process all the information at the time of the event. It is also the case that people process incoming information in a biased way. A simple example would be seeing what you expect to see or hearing what you expect to hear when the expectation does not match reality.
The objective of interviews should be to maximize the retrieval of information about the occurrence and not focus on finding fault with the actions taken or decisions made. Understanding the context is as important as identifying the 'chain of events' that led to the safety occurrence. HP principles must be considered to facilitate an interview environment in which the interviewee benefits from positive safety culture. From a safety perspective, taking systemic remedial actions is more beneficial than punishing an individual, particularly when their behavior can be understood in terms of the HP principles and the operational context.
Interviewers, and investigators in general, must remember that most of the time people try to do a good job and do what makes sense to them at the time without knowledge of the outcome. Rather than trying to find out how somebody "could be so stupid," the interview should focus on understanding why the actions taken made sense at the time.
2. What influences enable people to report or prevent people from reporting safety concerns?
Because having data is a prerequisite for learning and for any safety improvements, and because people-generated data provides the "HOW and WHY information," it is critical that people do generate such data. Whether organizations or individuals are willing to report their experiences and errors is largely dependent on the perceived benefits and risks associated with reporting. The key to this is maximizing the ease of reporting while minimizing the anxiety of reporting.
Anonymity and Confidentiality
Safety reporting systems may be either anonymous, confidential, or both. In an anonymous reporting system, a reporter does not provide their identity. In this case, there is no opportunity for further clarification of the report’s contents, and there is no possibility to provide direct feedback. However, there is very little risk to the reporter. In a confidential reporting system, any identifying information about the reporter is known only to a designated custodian. Again, this minimizes the personal risk to the reporter but allows for further clarifications when needed. If individuals who report safety issues are protected, have a clear, accessible policy on safety reporting to follow, and are treated fairly and consistently, they are more likely to disclose such information and work with the regulator or management to effectively mitigate the identified safety risks.
Organizational/Safety Culture
An organization's culture affects how safety is perceived, valued, and prioritized by management and operational personnel. Individuals are more likely to report their experiences and errors in an environment where they will not be judged or treated unfairly by their peers or their employer. This speaks to the culture of the organisation and whether the focus is to learn from a safety occurrence for the benefit of all or to apportion blame. If organizational leaders and managers are truly committed to learning from safety occurrence data, members of the organization will come forward with such data and with important insights into the operation.
An increase in confidential reports and a decrease in anonymous reports is usually indicative of the organization's progress towards a positive safety culture. A large number of reports does not necessarily mean there is a safety issue but could be an indication of a sincere commitment to identify hazards early.
Regulators and organizations should consider that a mandatory reporting system might yield reports that supply the bare minimum of information, and that people work hard not to be put in a position to have to report if the consequences are likely to be negative for them. More useful information may be offered in a voluntary reporting system, especially in a cultural environment where reporters trust the information reported will be managed to make positive local and systemic changes.
Ease of reporting
Safety reporting is essential for a service provider to be able to identify and understand the risks associated with HP and, where possible, to be able to develop mitigations for them. A regulator should assess both the safety reporting system that a service provider established as well as the outputs of that system. A regulator should not focus just on how many safety reports a service provider receives, but whether the service provider has considered the variety of factors that might be influencing why and when people in their organization report. Furthermore, the regulator should assess what the service provider does with the data and how the data is used to improve and promote safety.
Standardized report forms must strike a balance between requiring enough information to understand when further investigation is needed and deterring people from reporting because to report is too onerous. This challenge inherently limits the number of HP-specific questions that can be asked on such a form. Also, the method used to submit reports should be assessed: Do all personnel have equal access to the method used (computer / application / office)? Can it be used discreetly? The form and the method for submission should encourage and facilitate reporting.
The role of feedback
People are much more likely to report when they receive meaningful feedback and when they see that their reporting leads to meaningful action. There is no point in collecting data if nothing is being done with it. When frontline personnel have the impression that reporting is only solicited to meet a regulatory requirement, very few reports are submitted. To truly encourage the open submission of safety reports, frontline personnel should be involved in both the analysis of the data and in determining the follow-up actions, and not simply seen as “data providers”. The data provided needs to be valued, analyzed carefully, and followed with clear action.
3. HP insights gained from analysing safety occurrence data
When analyzing the human contribution to safety outcomes, particular care must be exercised to avoid viewing the outcome as more predictable than it really was (hindsight bias). It must be remembered that in most cases, there are numerous people involved in the lead up to a safety occurrence, that people do what makes sense to them at the time, and that these people do not know what the outcome will be. Analysts should pay close attention to the HP Principles and to understanding the operational context when seeking to understand human performance and the choices people made during what later turned out to be a “safety occurrence.” Human error is a term usually assigned post-occurrence to a behaviour, even if that behaviour made sense at the time. Labeling acts as human errors and counting how many were made does little to improve safety.
Single event vs multiple events analysis
Considering the HP principles during an in-depth analysis of a single occurrence assists the regulator and the provider in better understanding why humans throughout the system behaved or responded in the way they did. The regulator needs to take into account the dynamic aspects of the situation and the factors most likely to have influenced peoples' actions at the time. This understanding is necessary to be able to identify and accurately describe any systemic issues or latent conditions so that the most appropriate remedial actions can be taken. Such actions may include the need for comprehensive or updated guidance; further safety promotion activities; adjustment to regulatory requirements; or enforcement action in those rare cases where necessary to maintain minimum safety standards. When participating in an in-depth analysis of a single occurrence, a regulator should have sufficient understanding of HP to recognize their own knowledge gaps and to determine whether special HP expertise is needed.
To monitor performance and support policymaking, analyzing aggregated safety data across multiple occurrences, hazards or other performance indicators is necessary. Such data may be obtained through collections of in-depth analyses of single occurrences as well as from occurrence reporting systems, hazard reporting systems, and in-service operational data. Identifying common HP issues even when they occur in different environments and result in different types of outcomes enables regulators to identify the most effective preventive or remedial actions. Therefore, when paired with reviews of individual occurrences and with dialogue with service providers and operational experts, analyses of aggregated data can be particularly useful in identifying latent conditions, underlying common causes, and systemic issues.
Taxonomies
A taxonomy provides the organizing framework for categorizing (or coding) information to allow statistical analysis of data. Common taxonomies and definitions establish a standard industry language, thereby facilitating information sharing and exchange. The aviation communities' capacity to focus on safety issues is greatly enhanced by sharing a common language. However, it is important to recognize that all taxonomies have some limitations.
A taxonomy that is too narrowly focused, or that contains a mismatch of the scope of codes, may not help describe or identify common problems. Such a mismatch of codes can arise when circumstances, events, and outcomes are mixed in a taxonomy.
Particular care must be exercised when using a taxonomy that:
- Focuses on identifying "deficient individuals" rather than on identifying hazardous conditions which may make it difficult for a person to function well
- Allows the same item to be coded differently by different coders, or under multiple categories. Such a taxonomy creates the possibility of drawing incorrect conclusions about what hazards exist resulting in an inability to identify appropriate mitigations.
Limitations in the analyses of HP-related data
Objective information is quantifiable and not based on opinion, while subjective information reflects a person's experience or perspective.
- Quantitative data is expressed in numbers. Subjective and objective information may both be expressed quantitatively. Contextual information that can provide HP insights may already be quantitative (e.g., the date, the time of day, light levels, the number of hours on duty, the amount of fuel used for landing, or the temperature), or be coded to be expressed quantitatively (e.g., subjective rating scales).
- Qualitative data provides a description of a person's experience or perspective in a narrative. While many accident and incident forms ask for subjective narrative about causes, contributing factors and circumstances, this information may still need to be coded for analysis. Qualitative data is necessarily subjective.
Both qualitative (subjective) and quantitative (objective and subjective) data offer useful HP insights. However, it is important to note that there are limitations in the analysis both data types, and these limitations must be taken into account. As the saying goes: Not everything that can be counted matters, and not everything that matters can be counted.
Qualitative data is usually gathered through open ended questions on questionnaires, and through interviews and case studies. This sort of data, expressed from an individual's viewpoint, can be very detailed and contextually rich, offering insights to why people acted in the way they did. However, this data carries the risk of biases and misinterpretations due to the subjective nature of the information provided, as well as how that information is interpreted by the receiver of the information. Moreover, such studies require a significant amount of time and often rely on a small sample size with results that may not be generalized to a larger population.
To gather subjective data that is expressed quantitatively, researchers often develop a set of questions with a very limited number of specified response alternatives that require the respondent to categorise their response (e.g., indicating how they felt on a scale of 1 to 5; indicating agreement or disagreement with a statement; selecting applicable responses from a list). This essentially codes subjective information into numbers, making it easy to analyse multiple responses to the same question. It also allows commonalities to be identified across multiple safety occurrences. From an HP perspective, however, much detail can be lost in translating complex processes or characteristics into numbers. It's also critical to remember that representing qualitative data in numbers does not make the data objective.
Simple numbers may also fail to capture the complexity of the operational context. For instance, a study attempting to identify and compare the percentage of hazards from two different airports may conclude from the statistical analysis that there are more frequent hazards at airport A than at airport B. However, it's possible that airport A is suffering from budget cuts that may lead to less operational personnel being available to perform certain ground handling tasks and thus lead to more mistakes being made by operational staff. It's also possible that airport A has more but simpler hazards while airport B has fewer but more serious hazards. Alternatively, one airport might have different weather conditions than the other, handle different type and volume of traffic, or be situated in a very different terrain. Referring to percentages alone can lead to an inability to set correct priorities or identify and implement appropriate mitigation strategies because the reasons behind the hazards are not fully understood.