CONTENT VALIDITY REPORT



CONTENT VALIDITY REPORT

FOR

ENTRY-LEVEL FIREFIGHTER

READING ABILITY TEST

PREPARED BY FIREFIGHTER SELECTION, INC.

193 Blue Ravine Road, Suite 270

Folsom, California 95630

1997

1. PROBLEM AND SETTING

The purpose of this study was to revise the 1996 Biddle & Associates’ Entry-Level Firefighter Test Preparation Manual and Reading Ability Test (hereinafter referred to as the “TPM Test”) for use as a selection procedure for Entry-Level Firefighters.

A total of 30 subject-matter experts were selected to participate in the validation study. The subject-matter experts represented a diverse ethnic/gender pool that had knowledge of the firefighter job and training academy. Ten (10) subject-matter experts participated each day.

2. JOB ANALYSIS: CONTENT OF THE JOB

A job analysis of the Entry-Level Firefighter classification was conducted using ten subject-matter experts on October 16th from 8:00 a.m. to 4:00 p.m.

The job analysis method used was called GOJA™ (Guidelines Oriented Job Analysis). A draft firefighter job description (developed by Biddle & Associates, Inc., from work with numerous other fire departments) was submitted to the subject-matter experts for their independent review. The draft job description was added to, subtracted from, and modified by the subject-matter experts to accurately represent the duties, knowledge, skills, abilities, physical and other characteristics that are required for successful performance of the firefighter job.

The final Job Description reflected the opinions of at least seven of the ten subject-matter experts that participated on October 16th.

The work behaviors and their associated tasks (called duties) were grouped into common domains. When the duties resulted in work products, they were described in the duties.

The duty domains identified were:

Domain A Station Duties

Domain B Apparatus and Equipment Maintenance

Domain C Readiness and Training

Domain D Inspections

Domain E Driving and Positioning

Domain F Rescue and First Aid

Domain G Deploying Hose and Pumping

Domain H Laddering

Domain I Ventilation and Forcible Entry

Domain J Extinguishing Fire

Domain K Salvage and Overhaul

Domain L Probationary Period

Domain M Special Assignments

The knowledge, skills, abilities, and other characteristics needed to perform the duties were identified in Domain N as Knowledge, Skills, Abilities, and Other Characteristics. The physical characteristics needed to perform the duties were identified in Domain O.

For each duty, subject-matter experts identified:

A) The FREQUENCY the duty was performed (listed as daily to weekly, monthly to yearly, quarterly to yearly, less than yearly, and not performed). Subject-matter experts identified one of the above selections.

B) The IMPORTANCE level of the duty. A scale of 1 - 5 (1 = not required / 5 = extremely critical) was used.

C) Whether the duty was PERFORMANCE DIFFERENTIATING

(i.e., whether or not the duty represented an aspect of performance which was likely to differentiate among levels of job performance).

A scale of 1 - 4 (1 = not performance differentiating / 4 = clearly performance differentiating) was used.

The operational definitions of knowledge, skills, and abilities were included in Domain N.

• Each knowledge was defined in terms of a body of learned information.

• Each skill or ability was operationally defined in terms of the duties which the skill or ability was needed.

• In addition, each knowledge was linked to the duties for which the knowledge was needed. Other characteristics were identified as to the reasons they were needed and/or the duty area to which they were linked.

Domain N also contained information on the degree of importance of each knowledge, skill, ability, or other characteristic, whether or not it was needed at the time of hire, and whether it was performance differentiating.

3. SELECTION PROCEDURE AND ITS CONTENT

The Test Preparation Manual (TPM) approach uses concrete measurement (testing) of a candidate’s reading, comprehension, retention, and recall ability which is used as a necessary prerequisite to critical, observable work behaviors and/or products. This is

accomplished by providing all candidates with a representative work sample of information in booklet form (TPM) taken directly from critical materials required and used by incumbents on the job.

The testing structure of allowing candidates to study and prepare themselves for the test has several benefits:

A) It replicates and simulates the process applied on the job of learning information and being tested on that information;

B) It measures (in a fair, job-related format) a candidate’s ability to read, comprehend, retain, and recall technical, job-related information;

C) It allows for educationally disadvantaged candidates to compensate for lack of academic skills by extra study, as they would be able to on the job; and

D) It allows candidates exposure to the type of reading materials they will be learning on the job.

TEST PREPARATION MANUAL (TPM)

The Test Preparation Manual (TPM) was originally written in 1984 under the direction of Biddle & Associates by qualified subject-matter experts working in fire departments. Information in the TPM represents a carefully selected sample of job and academy materials frequently used by firefighters on the job. In order to sample a broad range of reading material types (different sentence structures, contexts, and formats) read by firefighters on the job and in the academy, 115 pages in six (6) chapters were included in the TPM representing various types of literature samples read on the job and in the academy.

The six chapters in the TPM were:

Chp. 1 Fire Chemistry

Chp. 2 Hoses and Ladders

Chp. 3 Tools and Equipment

Chp. 4 Ventilation/Overhaul/Salvage

Chp. 5 First Aid and Rescue

Chp. 6 Fire Prevention

The TPM was updated in 1996, with an additional chapter added to the manual. The information included within the new chapter was selected by qualified subject-matter experts. The chapter was entitled “General Fire Ground Operations” and extended the TPM to134 pages in seven (7) chapters.

The candidates are given between four and six weeks to study the TPM prior to taking the exam. From Biddle & Associates’ past testing experience, this has been shown to be an adequate time for studying the TPM. Firefighters (in the City in which this study was performed) were required to read, study, and learn over 2000 pages of reading material during the first year on the job, and over 3000 pages in approximately 6-10 weeks of the academy.

The TPM has been carefully designed in light of the Uniform Guidelines on Employee Selection Procedures (1978). The TPM is designed to REPLICATE THE TYPE of materials that firefighters read on the job and to SIMULATE THE PROCESS used to read, comprehend, retain, and recall firefighter job and academy reading materials.

Surveys were given to gather evidence to support that the TPM Test measures the reading ability required for successful job performance and that it is a representative sample of the reading materials that are read on the job and academy (see Section 3).

TPM TEST

The TPM Test includes multiple-choice written test items constructed from the material in the TPM. The questions are designed to measure each candidate’s mastery of the body of information contained in the TPM. All the answers to the items on the test are in the TPM.

The TPM Test pool contains over 300 items and is currently split into two equal test forms--each containing 126 items (18 items representing each chapter in the TPM). Each item has four alternatives: one key and three distractors. Subject-matter experts reviewed all 300+ items.

The subject-matter expert item review included the following:

A) Reviewing the item distractors for incorrectness,

B) Reviewing the correctness of the key,

C) Reviewing page reference in the TPM,

D) Ensuring the uniqueness of each test item (duplicates, similar items, items which provided or pointed to key of the other items), and

E) Identifying the Minimum Expected Passing (Angoff rating).

Items that did not meet the above criteria were either deleted or revised during the workshop in order to meet the criteria. Items that were revised were re-rated during the workshop.

The behaviors and abilities measured by the TPM and TPM Test were identified and confirmed by subject-matter experts on the Job Description and other test validation forms discussed throughout this report (see Section 3).

Subject-matter experts identified and confirmed that the TPM Test measures and is a representative sample of the reading ability required for successful job performance.

The definition read as follows:

Read, retain, recall and appropriately interpret technical documents in

English, including diagrams, fire fighting manuals, “chem-tech” books, codes, ordinances, laws, polices, and procedures.

The above definition was taken from the Job Description.

Subject-matter experts identified the reading ability required on the job as:

A) EXTREMELY CRITICAL - Necessary for the performance of the job (subject-matter experts assigned it an importance level of five on a scale of one through five),

B) NOT LEARNED IN A BRIEF ORIENTATION OR THROUGH MORE EXTENSIVE TRAINING ON THE JOB, and

C) PERFORMANCE DIFFERENTIATING.

Subject-matter experts agreed and confirmed that the TPM Test measures a candidate’s reading ability.

4. RELATIONSHIP BETWEEN THE SELECTION PROCEDURE AND THE JOB

Each component of the TPM Test (TPM and TPM Test) was evaluated for content validity.

TPM AND READING ABILITY COMPONENT

Twenty seven (27) subject-matter experts compared the reading ability and process of reading required on the job and academy to that of the TPM and Test. This was done in order to evaluate if the content and use of the TPM were similar and related to the content of the reading material used on the job and in the academy. The subject-matter experts completed a TPM Validation Survey from which these conclusions on job relatedness were made.

The following are summaries, averages, and conclusions from the survey:

A) Subject-matter experts’ average opinion was that they spent 4.11 hours reading during each “on” day in the academy.

B) Subject-matter experts’ average opinion was that they spent 4.5 hours reading during each “off” day in the academy.

C) All 27 subject-matter experts stated that they could not have passed the academy without the ability to read, comprehend, retain, and recall written information.

D) Twenty-six (26) of the subject-matter experts (96%) stated that they could

not adequately perform the duties of a firefighter without the ability to read, comprehend, retain, and recall job-related materials.

E) Twenty-five (25) of the subject-matter experts (93%) stated that they could not have passed the academy solely through listening to lectures, attending class discussions, and/or other learning experiences WITHOUT independently reading and studying assigned materials.

F) Twenty-five (25) of the subject-matter experts (93%) stated that they could not learn the necessary information to adequately perform the duties of a firefighter solely through listening to lectures, attending training sessions, and/or other learning experiences WITHOUT independently reading and studying job-related materials.

G) Twenty-five (25) of the subject-matter experts (93%) stated that the TPM

did, as a whole, REPRESENTATIVELY SAMPLE the different types of materials that must be read on the job and in the academy.

H) Twenty-five (25) of the subject-matter experts (93%) stated that the FORMAT (the general make-up and structure) of the materials in the Test Preparation Manual were similar to most of the required reading materials on the job and in the academy.

I) Twenty-six (26) of the subject-matter experts (96%) stated that the CONTEXT (sentence structure) of the materials in the Test Preparation Manual was similar to most of the required reading materials on the job.

J) All 27 (100%) subject-matter experts stated that the CONTEXT (sentence structure) of the materials in the Test Preparation Manual was similar to most of the required reading materials in the academy.

K) Twenty-six (26) of the subject-matter experts (96%) stated that the process of having the candidate read the materials in the Test Preparation Manual REPLICATES (duplicates or repeats) some of the required reading part of the job and in the academy.

Subject-matter experts were asked to describe the level of memorization needed of the reading materials on the job and in the academy. The results obtained are as follows:

A) Subject-matter experts’ average opinion was that 79.9% of the material in the Recruit Training Manual must be memorized for successful completion of the training academy, and 78.9% must be memorized for successful performance of the job.

B) Subject-matter experts’ average opinion was that 79.6% of the material in the Basic Training Manual must be memorized for successful completion of the academy, and 74.8% must be memorized for successful performance on the job.

C) Subject-matter experts’ average opinion was that 65.7% of the material in the textbook Prehospital Emergency Care must be memorized for successful completion of the training academy, and 65.5% must be memorized for successful performance on the job.

D) Subject-matter experts’ average opinion was that 39.1% of the material in the Policies and Procedures Manual Volume I must be memorized for successful completion of the academy, and 53% must be memorized for successful completion of the job.

E) Subject-matter experts’ average opinion was that 59% of the material in the Policies and Procedures Manual Volume II must be memorized for successful completion of the academy, and 59.4% must be memorized for successful completion of the job.

F) Subject-matter experts average opinion was that 37.5% of the material in the Policies and Procedures Manual Volume III must be memorized for successful completion of the academy, and 44.1% must be memorized for successful completion of the job.

READABILITY ANALYSES

Readability analyses were conducted on a sample of the City’s job and academy reading materials and the Test Preparation Manual.

As different readability analyses tend to produce different results, Biddle & Associates evaluated the job and academy materials and the TPM with four readability techniques: (1) the Flesch-Kincaid, (2) the Coleman-Liau, (3) the Bormuth, and (4) the FOG Readability Index. Each of these analyses evaluated a number of passages from both sets of the documents and produced a readability statistic expressed in a grade level. For example, a readability statistic of 11.0 indicates an 11th grade reading level. The average reading level associated with the job and academy reading materials in this City was 12.2; the average reading level of the TPM was 11.5. Therefore, the reading level of the TPM is slightly below, but well within the range, of the level required for the job.

EFFORTS TO REDUCE ADVERSE IMPACT

The TPM is based on a concept designed to minimize adverse impact. Candidates who have “educational disadvantages” may compensate for their disadvantages through extra study for Biddle & Associates’ TPM Test.

During 10 years of protective service testing, Biddle & Associates, Inc. has continually evaluated and compared passing rates for the TPM Test against other protective service written tests (with and without pre-test study guides). The TPM Test consistently has demonstrated less adverse impact than the other tests to which it has been compared.

Several public employers across the nation have used the TPM concept to reduce adverse impact, and it is a nationally recognized method among test publishers for reducing adverse impact.

OTHER VALIDITY EVIDENCE

Biddle & Associates’ TPM Test has been validated in numerous fire departments across the nation including other cities in California. In 1982, the Test was challenged and upheld for the first time in Federal Court where content validity was the major defense (although criterion-related validity was also presented for a small sample). In 1996 the TPM and TPM Test was challenged and upheld for a second time in Federal Court where content validity was the major defense. The TPM Test was found to be job-related and consistent with business necessity.

Both content validity and criterion-related validity studies have been conducted.

5. USES AND APPLICATIONS

The TPM Test is recommended to be used as a selection tool for hiring Entry-Level firefighters. It has been a policy of Biddle & Associates, Inc. to recommend that the TPM Test be used on a pass/fail basis in order to minimize adverse impact. However, data to support ranking on the test has been gathered and the Test can be used as a ranking device. See the end of Section 4 for the data gathered to support ranking.

METHOD OF CUTOFF DETERMINATION

In order to estimate a minimum cutoff level for the test, Biddle & Associates, Inc. used the Modified Angoff Technique. This method has been supported in Bouman v Block post judgment enforcement litigation. Subject-matter experts used Cutoff Data Entry Forms to confirm the test key, confirm the page reference in the TPM, and to give their opinions on “the percentage of minimally-qualified candidates who were likely to answer this item correctly” for each item on the test. All of this information was thoroughly described orally to the subject-matter experts and given to them in writing.

A “minimally-qualified candidate” was defined as: “one who possesses a competent level of the ability being measured to successfully perform the job.”

The concept of a “minimally-qualified candidate” was discussed at length with the subject-matter experts. Examples of minimally-qualified, below minimally-qualified, and above minimally-qualified candidates was discussed openly with the group until a consensus opinion of a minimally-qualified candidate was reached.

The information gathered from subject-matter experts assisted Biddle & Associates in developing a minimum cutoff level for the TPM Test.

After the workshop, the survey results were tabulated and summarized. The mean average of subject-matter experts’ opinions on the percent of “minimum-qualified candidates expected passing” for each test form constructed was 81%, or 102 of the 126 items answered correctly. In order to properly consider several human and statistical factors the mean Angoff average (81%) should be reduced by at least one (1) standard error of measurement (SEM). To calculate the SEM, the reliability and standard deviation of each test administration will be needed. The formula for calculating the SEM is: the standard deviation of the test times the square root of one (1) minus the test reliability (see formula below).

Standard Error of Measurement:

For example, if the standard deviation of the test was 14.0 and the reliability was .92 (typical statistics based on prior administrations), a standard error of measurement of 3.9597 would be derived. Using the SEM of 4.0, a cutoff using one SEM below the Angoff mean would be a score of 98 (81% Angoff mean times 126 items equals 102; 102 minus SEM (4.0) equals 98).

At least four statistical and human factors should be considered when determining whether to lower the Angoff mean by one (1), two (2), or three (3) standard errors of measurement. The factors are:

A) The possibility of sampling error in the study,

B) The consistency of the results (internal comparisons of panel results),

C) The supply of and demand for the position being tested, and

D) The racial composition of the job classification.

These four standards were adopted from U.S. v. South Carolina, 434 US 1026 (1978).

At least one (1) standard error of measurement should be used. After the application of the standard error of measurement adjustment defined above, the resulting score is one that reflects the minimum competency level of the test.

It is the policy of Biddle & Associates, Inc. to recommend a cutoff that minimizes adverse impact set above the minimum competency level of the test. Biddle & Associates recommends using both statistical significance and practical significance tests when analyzing tests for adverse impact.

RANKING AND SCORES ABOVE MINIMUM COMPETENCY

Evidence to support the use of ranking was gathered for the test.

Subject-matter experts identified reading ability as performance differentiating on the Job Description.

Of all the duties previously linked to reading ability as defined in the Job Description, 74% were said to be performance differentiating by the subject-matter experts. Of the remaining duties linked to reading ability, 26% were said to be performance differentiating “to some degree.”

The Uniform Guidelines on Employee Selection Procedures (1978) Section 14C(9) stated the following regarding the use of ranking on employment related tests:

Ranking based on content validity studies. If a user can show, by job analysis or otherwise, that a higher score on a content valid selection procedure is likely to result in better job performance, the results may be used to rank persons who score above minimum levels. Where a selection procedure supported solely or primarily by content validity is used to rank job candidates, the selection procedure should measure those aspects of performance which differentiate among levels of job performance. (p.3803)

Section 5H of the Guidelines (1978) stated the following regarding ranking or high cutoff scores:

Where applicants are ranked on the basis of properly validated selection procedures and those applicants scoring below a higher cutoff score than appropriate in light of such expectations have little or no chance of being selected for employment, the higher cutoff score may be appropriate, but the degree of adverse impact should be considered. (p. 38298)

The Principles for the Validation and Use of Personnel Selection Procedures (1987) stated the following:

If a selection instrument measures a substantial and important part of the job reliably, and provides adequate discrimination in the score ranges involved, persons may be ranked on the basis of its results. (p. 24)

Selection techniques developed by content-oriented procedures and discriminating adequately within the range of interest can be assumed to have a linear relationship to job behavior. Consequently, ranking on the basis of such scores is appropriate. (p. 32)

The evidence obtained from the City’s subject-matter experts provided support for ranking and/or cutoff scores above minimum competency levels by means of the Job Analysis. The TPM and TPM Test process measures reading ability which is needed in the academy and on the job. Based upon past test administrations, the reliability of the test is consistently very high (between .90 and .94) and the standard deviation of the Test is consistency between 14.0 and 14.6, which shows an adequate discrimination in the score range.

Therefore, for reasons stated in the paragraph above, it is appropriate under content validity to rank-order scores above the minimum competency score and/or to select cutoff scores above the minimum competency score.

The validation study data shows that the minimum competency score of the test would be 98 given the sample standard deviation and reliability above. The City should not set a cutoff score less than 98 out of 126 items as passing. (Biddle & Associates recommends using only one (1) SEM below the Angoff average due to the protective service nature of the job, high test reliability, and other factors listed above.)

If ranking will be done on the test, Biddle & Associates recommends creating bands (rather than top-down ranking) using the standard error of measurement.

The data gives support for setting cutoff scores higher than the minimum competency level (see three references above showing that cutoffs may be set higher than normal expectations of minimum competency), if the data supporting the test shows it is differentiating or there are so many candidates that a lower score is not reasonable.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download