Best Practices in Pilot Selection

DOT/FAA/AM-19/6

Office of Aerospace Medicine Washington, DC 20591

Best Practices in Pilot Selection

Dana Broach David Schroeder Kevin Gildea

February 13, 2019

Final Report

NOTICE

This document is disseminated under the sponsorship of the U.S. Department of

Transportation in the interest of information exchange. The United States Government assumes

no liability for the contents thereof. ___________

This publication and all Office of Aerospace Medicine technical reports are available in full-text

from the Civil Aerospace Medical Institute's publications website:

1. Report No.

2. Government Accession No.

DOT/FAA/AM-19/6

4. Title and Subtitle

Best Practices in Pilot Selection

7. Author(s)

Dana Broach, Kevin Gildea, & David J. Schroeder

9. Performing Organization Name and Address

Aerospace Human Factors Research Laboratory AAM-500 P.O. Box 25082 Oklahoma City, OK 73125

12. Sponsoring Agency Name and Address

Office of Aerospace Medicine Federal Aviation Administration 800 Independence Avenue, S. W. Washington, DC 20591

15. Supplementary Notes

Technical Documentation Page

3. Recipient's Catalog No.

5. Report Date

February 13, 2019

6. Performing Organization Code 8. Performing Organization Report No.

10. Work Unit No. (TRAIS)

11. Contract or Grant No.

13. Type of Report and Period Covered

Final

16. Abstract

Demand for new civilian pilots continues to grow as the world's aviation system expands. U.S. mainline and regional air carriers will need about 1,900 to 4,500 new pilots per year (U.S. Government Accountability Office, 2014). Selection of the new pilots is a critical human resource management challenge for operators. We review civilian pilot selection procedures relative to a set of seven best practices: 1) Conduct a job analysis; 2) Define measurable, observable job performance metrics; 3) Identify and use reliable and valid predictors; 4) Conduct an appropriate validation study; 5) Determine cut-scores (pass/fail) on tests based on predicted job performance; 6) Evaluate the fairness of tests and cut-scores; and 7) Document the analyses. We reviewed 15 pilot test batteries and selection processes for 2 U.S. and multiple foreign airlines based on publicly available information. Overall, adherence to best practices as defined by the relevant legal and professional guidelines, standards, principles, and practices for the development, validation, and use of employee selection procedures, was inconsistent. Relatively few of the test batteries were based on a recent or current job analysis. There is evidence of psychometric reliability and useful validity for the test batteries. However, there is no evidence on the reliability or validity of the interviews used in pilot selection. The primary job performance criterion is success or failure in training. No data are available on the comparative selection (pass) rates by demographic groups on the test batteries. Technical reporting and documentation on pilot test batteries is better in Europe than in the U.S. Overall, we agree with the 2012 conclusion reached by the International Air Transport Association (IATA) that pilot selection programs worldwide and in the U.S. do not appear to have a strong scientific basis. We make two recommendations to improve the state-of-the-art in civilian pilot selection.

17. Key Words

18. Distribution Statement

Pilot

Document is available to the public

Personnel Selection Job Performance

through the National Technical Information Service, Springfield, VA 22161

Job Analysis

Ability Tests

19. Security Classif. (of this report) 20. Security Classif. (of this page)

21. No. of Pages

22. Price

UNCLASSIFIED

UNCLASSIFIED 73

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

ACKNOWLEDGMENT Research report in this paper was conducted under the Air Traffic Program Directive/Level of Effort Agreement between the Human Factors Division (ANG-C1), FAA Headquarters, and the Aerospace Human Factors Research Division of the FAA Civil Aerospace Medical Institute.

i

TABLE OF CONTENTS

Acknowledgment ........................................................................................................................................... i Table of Contents..........................................................................................................................................ii Table of Figures ........................................................................................................................................... iv Table of Tables ............................................................................................................................................. v Introduction................................................................................................................................................... 1

Organization of report............................................................................................................................... 4 Terminology.............................................................................................................................................. 4 Chapter 1: Best practices in personnel selection processes .......................................................................... 7 Conduct a job analysis .............................................................................................................................. 7 Define job performance measures............................................................................................................. 8 Identify predictors ..................................................................................................................................... 9 Evaluate relationship of KSAO measures to job performance criteria ..................................................... 9 Set cut-scores and/or decision rules ........................................................................................................ 10 Evaluate the fairness of the selection process ......................................................................................... 10 Assess selection process utility ............................................................................................................... 11 Challenges in applying these best practices ............................................................................................ 12 Chapter 2: Pathways for Becoming a Pilot ................................................................................................. 15 Pathway 1: University-based pilot training programs............................................................................. 15 Pathway 2: Independent flight schools/flight academies ........................................................................ 16 Pathway 3: Former military pilot ............................................................................................................ 17 Pathway 4: Multi-Crew Pilot License (MPL) ......................................................................................... 17 Pathway 5: "Flow-through" agreements ................................................................................................. 18 Chapter 3: Civilian Pilot Selection.............................................................................................................. 20 Previous reviews of (civilian) pilot selection practices........................................................................... 20 Review of pilot aptitude test batteries..................................................................................................... 20

Best practice #1: Conduct a job analysis............................................................................................. 21 Best practice #2: Define job performance measures........................................................................... 27 Best practice #3: Identify predictor measures..................................................................................... 27 Best practice #4: Evaluate relationship of KSAO measures (predictors) to job performance criteria. ............................................................................................................................................................ 27 Best practice #5: Set cut-scores and/or decision rules. ....................................................................... 28 Best practice #6: Evaluate the fairness of the selection procedures.................................................... 30 Best practice #7: Assess selection procedure utility. .......................................................................... 30 Innovations in pilot aptitude assessments ............................................................................................... 30 Personality assessment. ....................................................................................................................... 30 Social skills or competencies. ............................................................................................................. 30 Assessment Centers. ........................................................................................................................... 32 Interviews................................................................................................................................................ 33 Chapter 4: Screening in Other Transportation-related Occupations ........................................................... 35 Air traffic control specialist (ATCS; FG-2152) ...................................................................................... 35 Railroad dispatcher ................................................................................................................................. 39 Comparison of selection of pilots with selection in other transportation occupations............................ 40 Chapter 5: Summary, Recommendations, and Research Questions ........................................................... 41

ii

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download