CRS EXAM STORY - AIRS



ANATOMY OF DEVELOPMENT OF NEW CRS EXAMINATIONS

On January 1st 2017, AIRS introduced three new examinations for the Certification for Resource Specialist (CRS) professional credential.

The process of creating new exams occurs about every 4 years, takes about 10 months to complete and follows the standards provided by the Institute for Credentialing Excellence. Here’s how it happens … with a process that involved no less than 67 volunteers.

Stage 1: The Job Task Analysis

The first and most crucial stage involves a Job Task Analysis (JTA). In order to create a testing instrument (i.e. an exam), you must know what needs to be tested. What does a Resource Specialist actually do? What skills and knowledge is needed to carry out those tasks competently? Which parts of the job are the most important?

Early in 2016, three teleconferences were held with separate groups of existing CRS holders to review the existing JTA and decide if it remained generally applicable and/or required additional changes.

Our guides through this and all of the subsequent stages were AIRS’ certification consultants, Michael Hamm and Dr. Gerald Rosen.

Michael Hamm managed the accreditation program for the National Organization for Competency Assurance (NOCA) for eight years and is an American National Standards Institute (ANSI) certified auditor for compliance determination. He is a national authority on certification and accreditation. In fact, he literally has written the book on it as he is the author of Fundamentals of Accreditation. He also served on an ANSI advisory body developing a new international accreditation program for professional certification bodies.

Our psychometrician, Dr Gerald Rosen, is a consulting psychologist specializing in the design and administration of testing programs, test validation, statistical analysis, and vocational testing. Dr. Rosen has more than 20 years of experience in testing and measurement, and has produced numerous professional publications on these topics. He has managed the development of many national certification and licensure programs.

The volunteers shared their experience and insights. Although the general content of the previous JTA remained applicable, many suggestions were made to reflect the changing nature of resource specialize work. The final draft established 6 Domains, 14 Tasks, 24 Knowledge Areas and 21 Skills Sets.

The Tasks were weighted in terms of their relative importance to the job and to the client (for example, “Reporting and Analysis” was determined to comprise 10% of the overall job.

Stage 2: JTA Validation

However, the initial JTA only represented the combined views of 16 individuals. Did those conclusions hold true for everyone else?

The next stage involved an online survey of more than 400 current CRS practitioners that yielded detailed feedback from 74 individuals at a statistically acceptable response rate of around 15%.

The results validated the initial draft and enhanced it through several minor word improvements and subtle adjustments to the weightings of the various tasks.

To view the final version, go to:



Stage 3: Assessing Existing Questions

So now that we knew what needed to be tested, how many of our existing questions were still relevant and how many new questions would be required and in what areas?

Another volunteer subject matter expert (SME) team was assembled.

The challenge for this group was to go through the entire CRS question database of more than 250 items and decide which questions should be kept and which should be deleted. Any retained items had to be assigned to a Domain/Task within the new JTA. At the end of this process, we were able to quantify how many new questions were needed and in which areas (for example, we needed more than 70 new questions that tested the understanding of database development and maintenance).

Only about 60% of the existing questions were deemed both still relevant and soundly constructed (although many of the rejected ones had not been in active use for a few years). However, all of the retained existing questions were subsequently revised/edited to the point of being virtually unrecognizable from their original form as the process continued.

Stage 4: New Question Development

We now needed to write new over 150 new questions. Cue another subject matter expert group.

Throughout the process, the various volunteer groups were chosen to reflect the diversity of I&R/A, both in terms of the type of work performed, the geographic region, the nature of the agencies (small and large, urban and rural), and in terms of the individuals themselves (from 2 to 20 years of I&R experience, different levels of education, and cultural backgrounds).

This was one of the most challenging stages and took place over a two-day session in Atlanta.

Writing good questions is hard. But what is really hard is writing good ‘wrong’ answers. This ideally involves coming up with three distracters for each question that are plausible without being misleading while still leaving one answer that is obviously correct (but only obvious to the individuals that properly understand the issue).

This resulted in a provisional bank of about 180 new questions. Each question was linked to a specific Domain/Task within the new JTA and had a clear source (that is, a documented reference to a particular part of the AIRS Standards or the AIRS I&R Training Manual that confirmed the accuracy of the content).

Stage 5: Cut Score Review

A cut score (or pass mark) is not a random number. It should represent the percentage of answers that most accurately reflects the ability of an individual who is competent in the issues being tested.

A cut score is never perfect. But it should be the score that eliminates as many false positives and false negatives as possible (that is, it tries to ensure that people who are competent, pass and that people who are not yet as competent, do not pass). Within this context, an exam theoretically might have a pass mark as high as 95 or as low as 40 if those marks represent the “border line” between someone with the desired amount of understanding and someone who has yet to reach that level.

The AIRS Certification Program, in common with many examinations, uses a methodology known as modified Angoff ratings to determine cut scores. Basically, this assigns a ‘degree of difficulty’ number to each question in the item bank. Technically, each exam can have a different cut score depending on the difficulty or otherwise of the questions within each exam. However, AIRS generates a mathematical formula to ensure that each exam contains the same balance of difficult and easy questions (that is, each exam has the same cut score).

In the fall of 2016, another group of SMEs spent several hours assigning Angoff ratings to each question.

As a natural part of this process, further changes were made to many questions to improve their clarity and some questions were eliminated.

Stage 6: Exam Creation

AIRS now had a database of more than 400 new and existing questions but no actual examinations.

Using mathematical models from the cut score analysis, three new draft exams were created with an identical (to within 2 decimal points) balance of difficult/easy questions that accurately reflected the weighting of the JTA (for example, 12% of the questions involved a knowledge of the issues surrounding Communication). By the end, each new exam had a cut score of 75.

Stage 8: Initial Exam Review

The next stage involved a group of SMEs ensuring that each exam was a fair test unto itself (for example, an exam did not contain two questions that addressed the same issue with similar words … or did not contain a question that inadvertently gave away the answer to another).

The process involved yet another opportunity to confirm and further enhance question clarity and accuracy. And even at this stage, a handful of questions were removed as the team verified that there is one and only one correct answer for each question.

Stage 9: Final Exam Review

This involved the largest number of volunteer SMEs as we needed about 10 existing CRS holders to take each of the three new exams – in a real situation (i.e. answering the questions in a ‘real’ exam situation.

The value of this stage is that it provided insights into any questions that acquired larger-than-normal ‘wrong’ answers and to see if some of the distractors were proving more popular than they should ... in the end this resulted in about 20 questions being changed.

In 2020, the process starts again!

AIRS would like to thank all of the following individuals who volunteered for this long, challenging and highly confidential process, and their organizations for allowing their participation.

| | | | |

|Name | | | |

|Organization Name | | | |

|City | | | |

|State or Province | | | |

| | | | |

|Alisha Barnes | | | |

|NJ 2-1-1 Partnership | | | |

|Whippany | | | |

|NJ | | | |

| | | | |

|Ann Matteson | | | |

|Detroit Public Library | | | |

|Detroit | | | |

|MI | | | |

| | | | |

|Anna Kirkpatrick | | | |

|Multnomah County Aging, Disability, and Veteran Services | | | |

|Portland | | | |

|OR | | | |

| | | | |

|Barbara Fisher | | | |

|United Way of Broome County | | | |

|Vestal | | | |

|NY | | | |

| | | | |

|Becky Varik | | | |

|Contact Community Services / 211CNY | | | |

|Syracuse | | | |

|NY | | | |

| | | | |

|Beth Diamond | | | |

|United Ways of Vermont/Vermont 2-1-1 | | | |

|Essex Junction | | | |

|VT | | | |

| | | | |

|Bruce Greenstein | | | |

|211 Palm Beach Treasure Coast | | | |

|Lantana | | | |

|FL | | | |

| | | | |

|Candace Poole | | | |

|United Way of the Chattahoochee Valley | | | |

|Columbus | | | |

|GA | | | |

| | | | |

|Carlos Garbutt | | | |

|Arkansas Crisis Center | | | |

|Springdale | | | |

|AR | | | |

| | | | |

|Christopher Castillo | | | |

|211 LA County | | | |

|San Gabriel | | | |

|CA | | | |

| | | | |

|Courtny Nesheim | | | |

|Helpline Center | | | |

|Sioux Falls | | | |

|SD | | | |

| | | | |

|Dave Erlandson | | | |

|Ceridian/211 Twin Cities | | | |

|Minneapolis | | | |

|MN | | | |

| | | | |

|David Smith | | | |

|United Way of Greater Houston | | | |

|Houston | | | |

|TX | | | |

| | | | |

|Dayann Harvey | | | |

|Heart of Florida United Way | | | |

|Orlando | | | |

|FL | | | |

| | | | |

|Daynell Marbury | | | |

|PA 2-1-1 Southwest - United Way of Southwestern PA | | | |

|Pittsburgh | | | |

|PA | | | |

| | | | |

|Donna Fowler | | | |

|Comprehensive Crisis Care | | | |

|Napoleon | | | |

|OH | | | |

| | | | |

|Donnie House | | | |

|Community Service Council/211 Helpline | | | |

|Tulsa | | | |

|OK | | | |

| | | | |

|Elizabeth Kargbo | | | |

|Info Line | | | |

|Akron | | | |

|OH | | | |

| | | | |

|Elizabeth Radcliff | | | |

|Indiana 211 Partnership | | | |

|Indianapolis | | | |

|IN | | | |

| | | | |

|Elly Stornebrink | | | |

|bc211 | | | |

|Vancouver | | | |

|BC | | | |

| | | | |

|Emily Ruckel | | | |

|United Way of Greater Toledo | | | |

|Toledo | | | |

|OH | | | |

| | | | |

|Gabriela Trujillo | | | |

|United Way of Merced County | | | |

|Merced | | | |

|CA | | | |

| | | | |

|Hannah Newton | | | |

|Crisis Clinic | | | |

|Seattle | | | |

|WA | | | |

| | | | |

|Jennifer Abels | | | |

|Helpline Center | | | |

|Sioux Falls | | | |

|SD | | | |

| | | | |

|Jessica Embury | | | |

|LifeWays Community Mental Health | | | |

|Jackson | | | |

|MI | | | |

| | | | |

|Jim Christie | | | |

|2-1-1 Big Bend | | | |

|Tallahassee | | | |

|FL | | | |

| | | | |

|Joey Schulte | | | |

|HandsOn Central Ohio | | | |

|Columbus | | | |

|OH | | | |

| | | | |

|John Allec | | | |

|Findhelp Information Services | | | |

|Toronto | | | |

|ON | | | |

| | | | |

|John Mitchell, C.R.S. | | | |

|Heart of West Michigan United Way | | | |

|Grand Rapids | | | |

|MI | | | |

| | | | |

|Julia Webb | | | |

|Frederick County Mental Health Association | | | |

|Frederick | | | |

|MD | | | |

| | | | |

|Kate Snow | | | |

|211 Nova Scotia | | | |

|Dartmouth | | | |

|NS | | | |

| | | | |

|Katie White | | | |

|Contact Community Services / 211CNY | | | |

|Syracuse | | | |

|NY | | | |

| | | | |

|Kim Lowery | | | |

|UnitedWay of Northeast Louisiana 2-1-1 | | | |

|Monroe | | | |

|LA | | | |

| | | | |

|Laura James | | | |

|211 LA County | | | |

|San Gabriel | | | |

|CA | | | |

| | | | |

|Laura McDonald | | | |

|Laura L. McDonald Consulting/CIOC | | | |

|Brampton | | | |

|ON | | | |

| | | | |

|LeAnn Sedgwick | | | |

|Clearfield County Area Agency on Aging, Inc. | | | |

|Clearfield | | | |

|PA | | | |

| | | | |

|Licinia Mesquita | | | |

|Information Hamilton | | | |

|Hamilton | | | |

|ON | | | |

| | | | |

|Lilli Land | | | |

|LIFE Senior Services | | | |

|Tulso | | | |

|OK | | | |

| | | | |

|Linda Tokarski | | | |

|2-1-1 Michigan | | | |

|Lansing | | | |

|MI | | | |

| | | | |

|Lindsay Paulsen | | | |

|United Way of the Midlands | | | |

|Omaha | | | |

|NE | | | |

| | | | |

|Lisa Owens | | | |

|211 Ontario - Northern Region | | | |

|Thunder Bay | | | |

|ON | | | |

| | | | |

|Lorraine Ryder | | | |

|211 Northern Region | | | |

|Thunder Bay | | | |

|ON | | | |

| | | | |

|Marioly Botero | | | |

|United Way of Greater Atlanta | | | |

|Atlanta | | | |

|GA | | | |

| | | | |

|Martha Chown | | | |

|Information Barrie, Barrie Public Library | | | |

|Barrie | | | |

|ON | | | |

| | | | |

|Martha Love | | | |

|United Way of Greater Cleveland | | | |

|Cleveland | | | |

|OH | | | |

| | | | |

|Mary Leininger | | | |

| | | | |

| | | | |

| | | | |

| | | | |

|Megan R. Velasquez | | | |

|LifeStream Services | | | |

|Yorktown | | | |

|IN | | | |

| | | | |

|Melissa Agustin | | | |

|Connect2Help | | | |

|Indianapolis | | | |

|IN | | | |

| | | | |

|Meredith Hansen | | | |

|Brown County Aging & Disability Resource Center | | | |

|Green Bay | | | |

|WI | | | |

| | | | |

|Mona Gobert-Cravins  | | | |

|232-HELP/Louisiana 211 | | | |

|Lafayette | | | |

|LA | | | |

| | | | |

|Pat Toles | | | |

|211 Texas United Way of Tarrant County | | | |

|Arlington | | | |

|TX | | | |

| | | | |

|Patricia Dailey | | | |

|United Way 2-1-1 South Central IN | | | |

|Columbus | | | |

|IN | | | |

| | | | |

|Polly McDaniel | | | |

|iCarol | | | |

| | | | |

|NY | | | |

| | | | |

|Richard Rolbiecki | | | |

|Milwaukee County Department on Aging | | | |

|Milwaukee | | | |

|WI | | | |

| | | | |

|Robert Spangler | | | |

|United Way of Greater Toledo, DGI Associates | | | |

|Dayton | | | |

|OH | | | |

| | | | |

|Robin Pokolski | | | |

|United Way of Greater St Louis | | | |

|St Louis | | | |

|MO | | | |

| | | | |

|Sarah Fleming | | | |

|United Way of Metropolitan Nashville | | | |

|Nashville | | | |

|TN | | | |

| | | | |

|Sarah Pagels | | | |

|Gryphon Place | | | |

|Kalamazoo | | | |

|MI | | | |

| | | | |

|Sarah Teague | | | |

|South Sound 2-1-1 | | | |

|Tacoma | | | |

|WA | | | |

| | | | |

|Steve Eastwood | | | |

|2-1-1 Arizona | | | |

|Phoenix | | | |

|AZ | | | |

| | | | |

|Susan Larkin | | | |

|bc211 | | | |

|Vancouver | | | |

|BC | | | |

| | | | |

|Teresa Lavinder Zeigler | | | |

|Senior Navigator | | | |

|North Chesterfield | | | |

|VA | | | |

| | | | |

|Terry Halverson | | | |

|Great Rivers 2-1-1 | | | |

|Onalaska | | | |

|WI | | | |

| | | | |

|Terry Reardon | | | |

|Fairfax County Government | | | |

|Fairfax County | | | |

|VA | | | |

| | | | |

|W Keith Lavery-Barclay | | | |

|Area Agency on Aging for North Florida | | | |

|Tallahassee | | | |

|FL | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download