Lzdkfvnlkfsbnslfkbndlkndlfbn lzdkfvnlkfsbnslfkbndlkndlfbn ...



CHAPTER 3

THE MULTIPLE-CHOICE COMPONENT

MULTIPLE-CHOICE – 80%

MULTIPLE-CHOICE QUESTIONS COMPRISE 80% OF THE ENTIRE FINANCIAL ACCOUNTING AND REPORTING (FAR) SECTION OF THE COMPUTER-BASED CPA EXAM. UNDER THE PENCIL AND PAPER-BASED EXAM, ONLY 50-60% OF THE QUESTIONS WERE TESTED VIA THIS FORMAT. AT FIRST GLANCE, MOST CPA CANDIDATES AND INTERMEDIATE ACCOUNTING STUDENTS WOULD PROBABLY TELL US THAT THE MORE MULTIPLE-CHOICE QUESTIONS, THE EASIER THE EXAM IS. WHY DO MOST STUDENTS PREFER A MULTIPLE-CHOICE EXAM? THEY RECEIVE SOME COMFORT FROM THE FACT THAT AN ANSWER IS GIVEN. THEY FEAR THE OPEN-ENDED ESSAY AND PROBLEM FORMAT, AS THE ENTIRE ANSWER IS UP TO THEM. THEY BELIEVE THAT WITH MULTIPLE-CHOICE QUESTIONS THEY AT LEAST HAVE A FIGHTING CHANCE. WHAT DO THEY CALL THESE QUESTIONS – MULTIPLE-GUESS? LET’S HOPE THE TYPICAL ACCOUNTING STUDENT ISN’T JUST GUESSING. I AM QUITE CONFIDENT THAT CANDIDATES AREN’T PASSING THE CPA EXAM BASED UPON THEIR GUESSING ABILITIES. LET’S FACE IT – IF CANDIDATES KNOW THE MATERIAL AND THE CONCEPTS, IT WON’T MATTER WHAT QUESTION FORMAT IS USED.

The purpose of this chapter is to provide intermediate accounting faculty with information about multiple-choice question design, presentation format, selection methods, and the scoring process. Again, not all the information is provided, as the AICPA continues to release details. Further developments will be communicated in the quarterly CPA Connection email updates, scheduled to begin April 2003.

What could possibly be new information? Aren’t multiple-choice questions a familiar question format? Yes, some of the details have not changed for over 70 years, but the computer-based exam is taking test question selection and scoring processes to new levels, utilizing such methods as multistage testing presented in distinct testlets while tracking test enemies. Don’t stop now. Read on to learn more about these changes.

Multiple-choice Question Design

Historically, the CPA exam multiple-choice format used a four-option answer, allowing candidates to select from four (4) choices marked with small letters “a” through “d” The examiners will continue to use this format on the computer-based test (CBT). Another carryover from the paper-based exam is the requirement that candidates select the “best” answer. This statement tells us that while there could be more than one right answer, the examiners believe there is clearly one answer that stands out above the rest. This tends to be especially true in the auditing section. Financial accounting and reporting questions on the other hand usually have just one, correct answer. It is the nature of the subject matter. Candidates compute the answer using a distinct formula or algorithm applied to the relevant information. The search for the best answer applies only when a theoretical question requires candidates to identify or define a concept. Set your students’ minds to rest – the financial accounting and reporting (FAR) area is primarily computational. It’s easy – learn the proper formula or algorithm, plug-in the numbers, and the correct response should be listed as an answer choice. Well, maybe not that easy.

None of the Above

What about the dreaded “none of the above” response? For over twenty years, the CPA exam has deleted this response from its list of acceptable answers. In a recent meeting of CPA Review providers, Gregory Johnson, AICPA Director, CPA Examination, stated that they will continue to see “none of the above” as an unacceptable answer choice. The examiners believe there should be a distinct answer. The answer response, “none of the above,” does not answer the question.

Concerned that the exam might be too easy if every multiple-choice question has one right answer? What makes multiple-choice questions difficult? One of the factors is the inclusion of extraneous data. Extra information adds greatly to the question difficulty. Put the bait out to see if it they will bite on it. Write the question to include more data than they need to use. An entry-level accountant who possesses the skill to sort through information and only select the pertinent data is in high demand in today’s complex working environment. Students will rise to the challenge. They see the question design as fair as long as a correct answer response is listed. The “none of the above” response increases test anxiety. Put your students fears to rest – avoid the use of “none of the above” as an answer choice. For more ideas about question design and content, see Chapter 6, “Developing Tests Linked to the CPA Exam”.

Multiple-choice Question Presentation

Similar to the pencil and paper exam, the CBT presents the multiple-choice question format first. That’s, however, where the similarity ends. The paper-based FAR exam typically asked sixty-five to sixty-eight questions all under the heading of “Problem Number 1.” Problem numbers 2 and 3 then followed using a question format referred to as “other objective.” The last problems were the open-ended responses to essays and problems. The format of the new FAR exam is quite different. It is five separate questions labeled as follows:

1. Testlet Number 1 – Fifteen to forty multiple-choice questions

2. Testlet Number 2 – Fifteen to forty multiple-choice questions

3. Testlet Number 3 – Fifteen to forty multiple-choice questions

4. Simulation Number 1

5. Simulation Number 2

Note, the above scenario holds true for the FAR section only. The remaining exam sections may contain a different number of testlets and simulations. See Chapter 4 for simulation details. This chapter describes the FAR testlets.

Testlets and the Computer-based Exam

What is a testlet? In general, a testlet is defined as small subset of items in a pool, preassembled to meet sets of content specifications.[1] The small subsets of items on the CPA exam consist of groups of fifteen to forty multiple-choice questions chosen from the AICPA Examination Content Specifications. The content specifications describe, in outline form, the technical content areas that could be tested. As stated in the AICPA, Information for Uniform CPA Examination Candidates, 17th edition, page 12:

“The content specifications provide the framework or “blueprint” for testing

knowledge and skills on the Uniform CPA Examination.”

Look at a testlet as just a group of questions. The topics, however, are not randomly selected. The examiners carefully write questions to match the items listed in the content specification outlines (CSOs).

Content Specification Outlines (CSOs)

The CSOs identify the areas, groups, and topics tested. For example, Roman numeral I, Concepts and standards for financial statements (17% - 23%) is then followed by four groupings entitled:

A. Financial accounting concepts

B. Financial accounting standards for presentation and disclosure in general-purpose financial statements

C. Other presentation of financial data (financial statements prepared in conformity with comprehensive bases of accounting other than GAAP)

D. Financial statement analysis

Within each grouping, a topic is listed. For example under group A, Financial accounting concepts, two topics are shown:

1. Process by which standards are set and roles of standard-setting bodies

2. Conceptual basis for accounting standards

View the complete content specification outlines (CSOs) at the cpa- website. The CSOs are based primarily on the results of periodic practice analyses and studies of various-size public accounting firms. The information gathered relates to the work performed by entry-level accountants. Feedback is also obtained from academicians and corporate professionals who serve as CPAs. A list of selected publications for candidates to study is provided at the end of each CSO.

The FAR exam CSO study references include all generally accepted accounting principles (GAAP) issued by the Financial Accounting Standards Board (FASB). Unfortunately, the CSOs are too general to help faculty narrow the great breadth of financial information given. One quickly notes that the CSOs are nearly all-inclusive. After reading the exhaustive list by area, group, and topic, a typical student and CPA candidate becomes overwhelmed. The exam content remains a mystery, as it appears that all intermediate accounting topics are listed. Students believe every topic presented in the 11th edition of Intermediate Accounting has an equal chance of being tested. Experienced professors know better. Refer to Chapter 5 for a discussion of the basic concepts tested. Help your students narrow their study list.

Don’t forget that the computer-based FAR exam tests 80% intermediate accounting topics and 20% governmental/nonprofit accounting. This is a new combination. Prior to the CBT, governmental/nonprofit accounting was tested along with the tax and managerial accounting topics. This new combination may lead to increased difficulty for those candidates who are weak in the governmental/nonprofit area. You know what students ask: “Professor, how many of the multiple-choice questions test financial accounting topics and how many questions test the governmental/nonprofit accounting area?” The answer is plain and simple – we don’t know.

Question Quantity and Controlling Time

Each testlet contains the same number of multiple-choice questions. For example, if testlet number 1 has 25 multiple-choice questions, then testlet numbers 2 and 3 will also contain 25 questions. The stated range for a testlet is anywhere from 15 to 40 questions. Given the breadth of the FAR topics, it is most likely that at least 25 questions will be asked. There are so many testable topics.

Candidates are allowed to move back and forth within a testlet.For example, you are allowed to move back and forth within testlet number 1, but you cannot proceed to testlet number 2, until you completely close problem 1. Each testlet must be complete before the examinee moves onto the next testlet. Once a candidate completes the multiple-choice testlet section, they are required to close out of this area before they move on to the simulations. They are not allowed to skip from a testlet to a simulation.

Time control is an issue. Recall, the FAR computer-based exam has been reduced from 4.5 hours to 4 hours and includes the entire new content area of governmental/nonprofit accounting. This could lead to time allocation issues, since it is not expected that any content has been cut from either the financial or the governmental/nonprofit area.

What a challenge – guide students to prepare for an exam that previously allowed one-third more time. New content has been added. The time has been cut by almost 1 and ½ hours – one hour cut from the old ARE section and ½ hour cut back from the paper-based exam. Add to that the change in testing medium (paper to computer) and I believe there could be some serious time problems. What will help?

As of the publication date of this supplement, we know that the examiners plan to include a timing device that shows the time remaining for the entire exam. Times are not subdivided among the three testlets. Times are also not subdivided between the testlets and the simulations. Most students will need some practice situations at controlling time. Students will benefit from practicing not only the technical content but also practicing allocating and controlling their time. See Chapter 6 for suggestions to control time during the testing process.

Multiple-choice Question Selection

How will the multiple-choice questions be selected? The answer to this question deals with two factors:

1. The content specification outlines (CSOs)

2. Multistage testing

Question Selection Linked to the CSOs

The question selection process is directly linked to the CSOs. According to the CSOs published on the cpa- website, there are five areas within the FAR exam:

I. Concepts and Standards for financial statements (17-23%)

II. Typical items: recognition, measurement, valuation, and presentation in financial statements in conformity with GAAP (27-33%)

III. Specific types of transactions and events: recognition, measurement, valuation, and presentation in financial statements in conformity with GAAP (27-33%)

IV. Accounting and reporting for governmental entities (8-12%)

V. Accounting and reporting for nongovernmental not-for-profit organizations (8-12%)

Each testlet contains questions from all five areas listed above. This information was disclosed to CPA review providers in a November 2002 meeting at the AICPA offices. I have since attempted to clarify the process. It seems curious to me that each testlet, consisting of 25 to 40 questions, would combine the financial accounting topics with the governmental/nonprofit topics. This is a big change from the former question selection process. For example, the Accounting and Reporting (ARE) paper-based exam did not combine the three areas of taxation, governmental/nonprofit, and managerial accounting. Separate, distinct multiple-choice questions were presented. Under the old exam format the questions were clearly divided among the subjects. The proposal for the CBT is to combine both the financial accounting area and the governmental/nonprofit area all in one testlet. Look for more information in the upcoming email updates.

The content of each testlet is spread across all five areas of the content specification outlines. For example, testlet number 1 could test any of the areas listed in the entire CSO. Testlet numbers 2 and 3 would do the same. Each testlet will attempt to test different topics at the same level and/or the same topics at a more advanced level. The AICPA has stated that it will not test a particular topic in both multiple-choice and simulation format. For example, if pension concepts are tested via multiple-choice, candidates will not see pension topics in the simulation section. The overlap is controlled by a software technique to avoid “test enemies.”

Test Enemies

“Test enemies” is a term used by the AICPA examinations team to describe the situation where the same content is tested in both the testlets (multiple-choice) and the simulations. The software is programmed to catch content crossover. This control feature helps to evenly allocate the coverage. When topics are tested via multiple-choice, the candidate can be assured that the same topics will not also be tested via the simulation format.

If the above process works as described, students might perceive the exam as more difficult. They are already concerned about mixing governmental accounting and financial accounting in the same test section. Now, the test enemy control feature should allow the examiners to test more content. Will this make the CBT more difficult? The answer to that question is – let’s wait and see.

Multistage Testing

The computer-based CPA exam is designed using a form of adaptive testing known as multistage testing. The test stages consist of individual testlets. All examinees receive the same testlet number 1. The level of difficulty of testlet numbers 2 and 3 varies depending upon individual candidate performance on testlet number 1.

There is no internal branching within a testlet. Once a candidate begins any testlet, the CBT has already developed the testlet so candidates can make back and forth at their will within a testlet.[2] This is quite different than what examinees encounter on other exams, such as the GMAT. The computer-based GMAT exam precludes a candidate from returning to a previous exam question within a testlet, as the GMAT utilizes internal branching.

On the CPA exam the second testlet may be of the same difficulty level as testlet number 1 or a higher level. The multiple-choice testlets are stored in the question database at three levels of difficulty.

Difficulty Levels of Multistage Testing

The CBT contains three levels of multiple-choice question difficulty:

1. Easy

2. Medium

3. Hard

All candidates receive the same first testlet. Based upon the results of the first testlet, the second testlet may be at an easy, medium, or hard difficulty level. This is very different from the paper-based exam.

Under the old format, all candidates were scored based upon the same question sets. While some pretest questions varied (see the discussion below about scoring the exam and pretest questions) the actual reported score for all 50,000 plus candidates taking the May exam would have been based upon the scores earned by answering the same questions that all other candidates answered. Similarly, the November exam scores would have also been based upon the results of all candidates answering the same questions on that exam.

To date, the AICPA has not provided a definition of what constitutes an easy, medium, or hard difficulty level. This is another wait and see situation. We must wait for further details. My guess is some subjects are more difficult than others. For example, bond, pension, lease, and deferred tax concepts have consistently been difficult for candidates. Simple classification of items on the statement of cash flows can range from the easy to medium to hard difficulty level. Candidates must perform additional analyses as the difficulty increases. Selecting the correct definition or example might be considered easy. For example a multiple-choice question asking candidates to select the example of an “underlying” as it pertains to derivatives is quite easy compared to a computation.

The AICPA assures us that the difficulty levels have been carefully tested, analyzed, and the proper weights have been applied. The team of psychometricians will see to that. For an excellent reference source about computer-based exams, see Computer-Based Testing, Building the Foundation for Future Assessments, edited by Craig N. Mills, Maria T. Potenza, John J. Fremer, and William C. Ward, published 2002 by Lawrence Erlbaum & Associates, Inc., Mahwah, New Jersey. Dr. Craig N. Mills is the Executive Director of the AICPA examinations team. Dr. Mills has extensive experience in the area of high-stakes computerized testing. The AICPA is clearly very cautious in releasing details about the difficulty levels.

Multiple-choice Scoring

We are told that there is a great deal of overlap among the easy, medium, and hard testlet. Exhibit 1 below shows us how the overlap might look. Note that the majority of the questions fall in the middle of the three categories.

Exhibit 1

Multiple-choice Levels of Difficulty

| EASY |

| MEDIUM |

| |

|HARD |

Difficulty Over-lap

Remember the first, and only the first testlet is assured to be at the easy difficulty level. Testlet numbers 2 and 3 vary based upon an individual candidate’s performance. It is possible that a candidate could get three testlets at the easy level. It is also possible that a candidate could jump from easy to hard and therefore be given one easy and two hard testlets. While there are many combinations, it is important to remember that the difficulty is not automatically progressive in nature, but rather based upon individual candidate performance.

No matter what the description – easy, medium, or hard – none of the questions will be give-aways. Technical knowledge remains the key factor to passing. Have CPA candidates had experience taking different exams? Yes, if they sat for other professional examinations such as the GMAT and LSAT at the same time as their friend or colleague, they worked different exam forms with different questions. Even on the paper-based CPA exam, candidates received different questions. This only occurred in the pretest question area.

Pretest Questions

The AICPA examinations team routinely creates potential CPA exam questions for the purpose of continuous test bank database expansion. These questions are known as pretest questions. Each CPA exam contains a sample of these questions. Pretest questions are not identifiable by the candidate. This encourages candidates to give equal attention to all questions, regardless of the question nature. Pretest responses are not included in the final score even if the candidate chose the correct answer. Pretest questions neither help nor penalize the candidate.

For the last five years, there have been as many as four different test versions used on each exam section. Each version includes different pretest items. Candidates may not have known it, but if they completed a different test version than their friend sitting in the same exam, they worked different pretest questions. The pretest concept will continue under the CBT format.

Multiple-choice Questions on the Computer – Concerns

Your students may be concerned about working different questions. However that is only one concern. How many multiple-choice tests have your students taken on a computer? They should definitely practice. Where will they get the needed practice? The 11th edition of Intermediate Accounting and Fundamentals of Intermediate Accounting are an excellent beginning. To pass, you must first learn the concepts. Then, you master the test-taking medium. An up-to-date technically sound textbook is essential. Your instruction is essential.

Recall, candidates use an on-screen calculator to perform the basic functions of add, subtract, multiply, and divide. The spreadsheet function is not open while the candidate works the multiple-choice. Will they be able to quickly and accurately use the on-screen calculator function?

Candidates are not allowed scrap paper. Many students are used to highlighting key points and writing computations in the margins of their exams. To be successful in a CBT environment, students must modify their test-taking habits.

Test-takers enter and exit the test center to sit for other professional exams. Will the center be too noisy? Will security and monitoring procedures work?

Candidates stare at a computer screen for 4 hours. Will they tire quickly? These are just some of the many concerns that students have shared with me over the past few months. Their concerns are valid. Once again, we must wait and see.

Multiple-choice – 80%

There are many concerns as well as many unanswered questions. But, the facts remain the same – the computerized exam is coming in 2004 and 80% of the financial and accounting examination is multiple-choice. The simulations are only 20%. The new exam is passable. With your guidance and instruction, your students will pass the new computer-based exam. They passed the old exam under the old format. They will pass the CBT under the new format. The central theme is – students who possess the necessary knowledge will pass!

-----------------------

[1] Page 96, Computer-Based Testing, edited by Craig N. Mills, Maria T. Potenza, John J. Fremer, and William C. Ward, published by Lawrence Erlbaum Associates, Inc. 2002

[2] Page 97, Computer-Based Testing, edited by Craig N. Mills, Maria T. Potenza, John J. Fremer, and William C. Ward, published by Lawrence Erlbaum Associates, Inc. 2002

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download