Purpose and characteristics of “NOTECHS for HEMS” project
This project is required by Helicopters Italia and is concerned with the design and the possible validation of a questionnaire regarding the NOn-TECHnical (NOTECHS) for the HEMS flight crew members with the aims of monitoring and operating safely.
Macro-phases of the project
The project intended to develop a psychometric instrument using a quali-quantitative methodology, in order to measure the NOTECHS constructs. These are defined as cognitive and social competences that help to promote the correct application of the working procedures in aerospace. The design process of the scale occurred in a secondary review (about specific and scientific literature): a series of interviews performed on technical staff chosen by systematic sampling to define the procedural practises, the development of items for the instrument that will be based on the results of the previous phases. The definition of each step of the project follows in detail.
Considering what has already emerged in the specialist and scientific literature about the NOTECHS, we selected scales already validated and used internationally for the measurement of those constructs and similar – for instance, Decision making Competence, Extra-Role Performance, but also other useful constructs in order to prove the divergent validity of the instrument. The great number of questionnaires developed in English made it necessary to have specialist translation in addition to the procedures considered scientifically based ( translation-and back translation), in order to maintain the content validity of the original item. During the literature review phase we considered those constructs and those scales that actually apply, where available, to the contest subject of the study. We also considered to perform a meta-analysis or an analysis of the multiple matches in order to have a comprehensive understanding of the most relevant constructs according to the previous and latest applied research in this field.
Panel and Focus group
Simultaneously, the research group, with the help of a series of panel and focus group, using semi-structured interviews on selected samples of technical and professional staff, checked the most common issues occurring during operations. The team research classified these errors (slips, trips) or potential errors (near-miss) developing a tree diagram in accordance with the NOTECHS model.
That consistency allowed to prepare a scale in accordance with the NOTECHS systematisation, consisting of principles, rules and behavioural markers (Figure 1) which defined these four principal macro-constructs (Co-operation, Situation Awareness, Leadership and managerial skills, Decision-Making).
With the stakeholders involved, in this step we developed the specific items for the considered working properties in order to preserve the ecological validity of the instrument.
Figure 1. NOTECHS descriptive framework.
Item and scoring definition
The final part of the project is concerned with the development of the questionnaire in accordance with the previous phases. A battery of items has been created and administrated to a sample for a pilot survey with the aim of testing comprehensibility and clarity. Later on the response scale (Likert) of the instrument has been developed. A series of descriptive statistics have been performed as a result of a second administration, aimed at verifying the reliability of the scales (inter-correlation measures, McDonald’s Omega, Guttman-Chronbach Alpha, ecc.) 10-17. Taken out the items with low fit indexes, the complete version of the instrument has been provided with indications for compiling, scales for measuring the convergent and divergent validity, objective and descriptive measures concerning personal data and work data (near-missing rate, slips and trips reported recently by the subject, etc.). Furthermore, the research group established the manual for scoring and a format for the presentation of the findings.
Validation and publication of the scale and dissemination of results
In accordance with the international procedures of validation, the research team tested the reliability of the instrument. That operation has been carried out after a significant number of compilations in order to statistically test the model through a series of confirmatory analyses. This phase aims at creating a measuring scale that satisfies the scientific requirements of reliability and validity. The research group aimed to publish the validation of the instrument on specialist journals in this field. That step provides benefits to the instrument in terms of reliability and statistic validity, as well as an international approval by the scientific and professional community. The research team promoted a meeting through a seminar that aims at the dissemination of the research results.
Consultancy in the programming training
In the final phase, we provided technical assistance for the implementation of training modules. Basing on the findings of the previous phases, and on the more recent methods suggests by the educational organizational psychology, the research group participated at the programming exercise and at the human resource development, by proposing useful instruments to evaluate the learning efficacy based on the NOTECHS model.
Flin, R. et al. Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety 3, 97-120 (2003).
Mishra, A., Catchpole, K. & McCulloch, P. The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Quality and Safety in Health Care 18, 104-108 (2009).
Weller, J., Ceschi, A. & Randolph, C. Decision-Making Competence Predicts Domain-Specific Risk Attitudes. Frontiers in Psychology 6, 540 (2015).
Koopmans, L. et al. Development of an individual work performance questionnaire. International Journal of Productivity and Performance Management 62, 6-28 (2012).
Robertson, E. R. et al. Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PloS one 9, e90320 (2014).
Goeters, K.-M. Validation of CRM training by NOTECHS: Results of the PHARE ASI project. Aviation psychology: Practice and research. Aldershot, UK: Ashgate (2004).
Avermate van JAG, K. E. NOTECHS: the evaluation of non-technical skills of multi-pilot air crew in relation to the JAR-FCL requirements. (Research Report for the European Commission, 1998).
Davis, P. Human factors contributing to slips, trips and falls. Ergonomics 26, 51-59 (1983).
Van der Schaaf, T. W., Lucas, D. A. & Hale, A. R. Near miss reporting as a safety tool. (Butterworth-Heinemann, 2013).
Sartori, R. & Bortolani, E. Approccio ideografico e approccio nomoteico alla persona: Il caso dei test psicologici. Giornale italiano di psicologia 33, 107-118 (2006).
Sartori, R. Response bias e response style negli strumenti self-report. Di-PAV Quaderni 20, 57-64 (2007).
Sartori, R. Psicologia psicometrica. (LED, Edizioni universitarie di lettere economia diritto, 2009).
Sartori, R. & Pasini, M. Indicatori e costrutti in psicometria: validità dei test. Sezione monografica: GLI STRUMENTI DI AUTOVALUTAZIONE IN PSICOLOGIA 5, 25 (2007).
Sartori, R. Considerazioni sull’analisi dei dati come fase del processo di ricerca in psicologia. DiPAV-QUADERNI (2008).
Sartori, R. Il questionario autosomministrato: considerazioni critiche. DiPAV-QUADERNI (2004).
Sartori, R. Validità, attendibilità, sensibilità e sensatezza dei metodi e delle misure in psicologia: questionari e test. DiPAV-QUADERNI (2004).
Sartori, R. & Pasini, M. Quality and Quantity in Test Validity: How can we be Sure that Psychological Tests Measure what they have to? Quality & quantity 41, 359-374 (2007).