MOCA-Peds: Development of a New Assessment of Medical ...

MOCA-Peds: Development of a New Assessment of Medical Knowledge for Continuing Certification

Laurel K. Leslie, MD, MPH,a,b Murrey G. Olmsted, PhD,c Adam Turner, MPH,a Carol Carraccio, MD, MA,a Andrew Dwyer, PhD,a Linda Althouse, PhDa

The American Board of Pediatrics (ABP) certifies that general and subspecialty pediatricians meet standards of excellence established by their peers. Certification helps demonstrate that a general pediatrician or pediatric subspecialist has successfully completed accredited training and fulfills continuous certification requirements (Maintenance of Certification [MOC]). One current component of the MOC program is a closed-book examination administered at a secure testing center (ie, the MOC Part 3 examination). In this article, we describe the development of an alternative to this examination termed the "Maintenance of Certification Assessment for Pediatrics" (MOCA-Peds) during 2015?2016. MOCA-Peds was conceptualized as an online, summative (ie, pass/fail), continuous assessment of a pediatrician's knowledge that would also promote learning. The system would consist of a set number of multiple-choice questions delivered each quarter, with immediate feedback on questions, rationales clarifying correct and incorrect answers, references for further learning, and peer benchmarking. Questions would be delivered quarterly and taken at any time within the quarter in a setting with Internet connectivity and on any device. As part of the development process in 2015?2016, the ABP actively recruited pediatricians to serve as members of a yearlong user panel or single-session focus groups. Refinements to MOCA-Peds were made on the basis of their feedback. MOCA-Peds is being actively piloted with pediatricians in 2017?2018. The ABP anticipates an expected launch in January 2019 of MOCA-Peds for General Pediatrics, Pediatric Gastroenterology, Child Abuse, and Pediatric Infectious Diseases with launch dates for the remaining pediatric subspecialties between 2020 and 2022.

Since 1934, the American Board of Pediatrics (ABP) has certified general pediatricians and pediatric subspecialists on the basis of standards of excellence developed by their peers.1 Certification status has been used by various stakeholders (eg, the public, health care providers, hospital credentialers, licensing boards, and insurers) to determine if a pediatrician meets these standards. In 2000, recognizing the rapid change in medical knowledge, the American Board of

Medical Specialties (ABMS) and its 24-member board implemented the Maintenance of Certification (MOC) program. Its goal was to demonstrate that a physician was actively engaged in staying current with new knowledge and consequently making changes in their practice.2 Currently, this process includes a periodic assessment of a physician's medical knowledge via a proctored, multiple-choice examination administered at a secure testing center (ie, the MOC Part 3 examination).3

abstract

aThe American Board of Pediatrics, Chapel Hill, North Carolina; bSchool of Medicine, Tufts University, Boston, Massachusetts; and cRTI International, Durham, North Carolina

Dr Leslie contributed to the conception, design, and implementation of the Maintenance of Certification Assessment for Pediatrics (MOCA-Peds) and its formative evaluation and to the acquisition, analysis, and interpretation of the data, and she drafted the initial manuscript; Drs Althouse, Dwyer, Olmsted, and Mr Turner contributed to the conception, design, and implementation of MOCA-Peds and its formative evaluation and to the acquisition, analysis, and interpretation of the data, and they drafted components of the manuscript; Dr Carraccio contributed to the conception, design, and implementation of the MOCA-Peds program; and all authors reviewed and revised the manuscript, approved the final manuscript as submitted, and agree to be accountable for all aspects of the work.

DOI:

Accepted for publication Sep 11, 2018

Address correspondence to Laurel K. Leslie, MD, MPH, The American Board of Pediatrics, 111 Silver Cedar Court, Chapel Hill, NC 27514. E-mail: lleslie@

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

Copyright ? 2018 by the American Academy of Pediatrics

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

FUNDING: Funded by the American Board of Pediatrics (ABP) with the exception of the research and evaluation component, which was funded by the ABP Foundation, a nonprofit, supporting organization to the ABP.

To cite: Leslie LK, Olmsted MG, Turner A, et al. MOCA-Peds: Development of a New Assessment of Medical Knowledge for Continuing Certification. Pediatrics. 2018;142(6):e20181428

Downloaded from news by guest on October 6, 2021 PEDIATRICS Volume 142, number 6, December 2018:e20181428

SPECIAL ARTICLE

2015: AN INFLECTION POINT

In May 2015, the ABP invited >80 leaders in assessment, psychometrics, medical education, pediatric practice, and technology to participate in a conference focused on improving the MOC Part 3 examination.4 Similar to other certifying boards, the ABP visualized its examinations as summative (ie, pass or fail) assessments of a pediatrician's current knowledge and clinical judgment. However, the ABP was exploring multiple potential changes to the MOC examination, including the following 2 technological enhancements: (1) administration of the examination at home or work with remote proctoring through one's computer and (2) the incorporation of reference material to mimic access typically available in practice.

Evidence presented at the conference5 on the exponential increase in new medical knowledge,6 the loss of physician knowledge over time,7 ?9 contemporary theories regarding self-assessment10?12 and adult learning,1 3? 17 and technological advances in both assessment and education provided additional, compelling stimuli for change. Two presentations in particular helped to further shift the ABP's reconsideration of the proctored multiple-choice examination. In the first presentation, Cees van der Vleuten,1 8 an internationally recognized leader in medical education and assessment, challenged the ABP's assumption that examinations conducted by certifying organizations should only focus on the assessment of learning with a summative measure. Dr van der Vleuten18 encouraged the ABP and other ABMS member boards to consider whether examinations can also encourage learning. In a second presentation, members of the American Board of Anesthesiology (ABA) described their 2014 pilot of an innovative approach to their

MOC program. This pilot, termed MOCA-Minute, consisted of brief multiple-choice questions distributed via weekly e-mail with immediate feedback provided after each question was answered.19

These presentations initiated new conversations about alternative approaches to the ABP's MOC examination. In June 2015, the ABP Board of Directors unanimously agreed the ABP should develop and pilot an online, continuous assessment model that would consist of multiple-choice questions delivered on a periodic basis over a set interval of time, similar to MOCAMinute. The assessment, termed the Maintenance of Certification Assessment for Pediatrics (MOCAPeds), would eliminate the need to go to a testing center, permit the use of resources, and incorporate learning opportunities.

To translate this mission into an acceptable, sustainable, and psychometrically sound approach, the ABP established several teams, including (1) the MOCAPeds Executive Team, whose purpose was to make overarching business and policy decisions; (2) the MOCA-Peds Work Group Team, composed of several ABP departments who converted business and policy decisions into daily operations; (3) the MOCA-Peds Task Force, composed of primary care pediatricians who developed questions, wrote rationales, and identified references for questions; and (4) a research team who would lead the ABP's efforts to engage pediatricians who were maintaining certification in the development of MOCA-Peds.

A launch date for a MOCA-Peds pilot in general pediatrics was set for January 2017. To prepare for the pilot, the ABP planned to develop an initial model during Fall 2015 and engage pediatricians in refinement of the model and in the development of Web-based and

mobile platforms through a yearlong user panel and single-session focus groups in 2016 (see Table 1 for time line). The ABP contracted with an internationally recognized research organization, RTI International (RTI), to bring additional evaluative rigor to the engagement process with pediatricians and to proactively address any potential concerns about bias in data collection or analyses. In this article, we discuss the development of the MOCA-Peds model and platform in 2015?2016. Additional information about MOCAPeds is available at bp. org/mocapeds.

FALL 2015: CONCEPTUALIZING AN INITIAL MODEL FOR MOCA-PEDS

Capturing Potential Model Components in a Key Driver Diagram

From the outset, the ABP recognized the inherent difficulty in designing a single tool that functioned as both a summative assessment of general pediatric knowledge and an opportunity for learning. In addition, MOCA-Peds would need to be compatible with busy pediatricians' lives. The ABP chose to capture these 3 functions in a key driver diagram (KDD) to guide the development of MOCA-Peds (see Fig 1). KDDs20 are visual tools commonly employed in program evaluation and quality-improvement projects in which researchers demonstrate the relationships among (1) the overall aim(s) of a project, (2) the key drivers (domains) that are hypothesized to contribute directly to achieving the aim, (3) the secondary drivers (components) related to those domains, and (4) specific change strategies or activities linked to each component that the ABP would incorporate into MOCA-Peds and test during the 2017?2018 MOCA-Peds pilot. KDDs function as "living" documents that guide the work as a project is planned, implemented, and modified;

Downloaded from news by guest on October 6, 2021

2

LESLIE et al

TABLE 1 Time Line for the Development of the MOCA-Peds Model in 2015?2016

Activity

Time Line

Future of testing conference Board of directors decision to develop MOCA-Peds Public announcement to develop and pilot MOCA-Peds Internal planning regarding initial program model First iteration of KDD developed Approval for stakeholder engagement approach proposed by the

ABP-RTI Research Team IRB submission E-mail request for pediatrician participation to 72353

pediatricians System development Random selection of user panel and focus group participants from

>3000 respondents Face-to-face, daylong user panel meetings User panel virtual meetings Focus group meetings Review of communication materials by user panel and focus

groups -Testing of Web-based platform Registration and preparation for pilot launch Pilot launch

May 2015 June 2015 June 2015 June to December 2015 November 2015 September to November 2015

December 2015 January 2016

January to December 2016 January to February 2016

February to March 2016 April to November 2016 April to November 2016 July to November 2016

September to December 2016 September to December 2016 January 2017

IRB, institutional review board.

FIGURE 1 MOCA-Peds conceptual KDD. Change strategies are displayed in Table 2.

components and change strategies may evolve over time.

For MOCA-Peds, the overarching goal and aim were clear: to improve child health outcomes through the development of a high-quality alternative to the current MOC

examination. The following 3 key domains were identified in Fall 2015: (1) crafting a psychometrically sound, continuous assessment of medical knowledge; (2) incorporating opportunities for learning; and (3) improving the MOC experience. Often, KDDs incorporate

aims that specifically state a target audience, end date, and measure(s) of success. For the purposes of the development of the initial MOCAPeds model, the ABP chose to use the KDD as a conceptual tool, with formal aim statements crafted later as part of the evaluation.

To identify potential components and change strategies, the teams (1) consulted with experts in the medical education and assessment fields, (2) conducted a literature review on adult learning, (3) researched assessment approaches used in other professional disciplines and countries, and (4) closely examined the ABA's MOCA-Minute19 to consider the applicability to pediatrics of different elements of their program (eg, 1-minute questions, quarterly question delivery, 120 questions per year, rationale after each question, and scoring approach).

These efforts led to an initial conceptualization in Fall 2015 of the MOCA-Peds model, including its components and identified change strategies. Below we discuss the initial KDD conceptualized in 2015. We then describe the methodologies employed to gather pediatrician input and the resulting changes to the MOCA-Peds' components and change strategies. Table 2 delineates initial components and change strategies identified in 2015 as well as modifications based on pediatrician feedback in 2016.

Key Driver or Domain 1: Craft a Psychometrically Sound, Continuous Assessment of Medical Knowledge

Developing a psychometrically sound assessment of general pediatric knowledge is complex and multifaceted, and the ABP employs several strategies in pursuing that goal. For example, with general pediatrics, the ABP regularly conducts a rigorous practice analysis with practicing general pediatricians21 to update the

Downloaded from news by guest on October 6, 2021

PEDIATRICS Volume 142, number 6, December 2018

3

TABLE 2 Key Drivers (Domains), Secondary Drivers (Components), and Change Strategies in the Initial MOCA-Peds Model in 2015 and After Pediatrician Feedback in 2016

Primary Driver or Domain in Initial Model in 2015

Secondary Drivers or Components in Initial

Model in 2015

Change Strategies in Initial Model in 2015

Modifications or Additions to MOCA-Peds Based on Pediatrician Feedback in 2016

Craft a psychometrically Ensure the assessment Implement a continuous assessment that keeps pace NA

sound, continuous

is valid, relevant, and with rapid medical knowledge

assessment of medical reliable

Administer more questions over the lifetime of a

NA

knowledge

pediatrician's career

Provide opportunity for pediatricians to give feedback NA

on individual questions

Incorporate opportunities for learning

Ensure answers to questions reflect an individual pediatrician's knowledge and clinical judgment

Deter inappropriate test-taking behavior

Provide tools for selfreflection on personal knowledge gaps identified during the assessment

Require personal authentication during log-in Use MOCA-Peds code of conduct

Deliver questions in random order for different individuals using the platform

Establish different time limits for each question on the basis of difficulty

Conduct ongoing data analyses to identify potential security threats

Provide immediate feedback on answers to individual questions

Display personal performance on a dashboard Offer benchmark comparisons with peers Provide a question history page to review previous

questions Distribute similar questions derived from the same

learning objectives later in year to reinforce learning Provide opportunity to rate clinical relevance of each question to personal practice Ask pediatricians to rate their confidence in their answers

Dropped Modified: language clearer and more user-

friendly

NA

Modified: all questions have same 5-min time limit

NA

NA Modified: display enhanced NA Modified: display enhanced NA Added: present personalized aggregation of

relevance ratings Added: present personalized aggregation of

confidence ratings

Improve the MOC experience

Provide tools for keeping up-todate with medical knowledge

Incorporate flexibility and choice in the assessment process and how one learns

Permit access to references when answering questions

Include up-to-date rationale and references Deliver questions randomly from entire content outline

as opposed to questions grouped by content area Not considered in the 2015 initial model Not considered in the 2015 initial model

Deliver questions quarterly with choice within the quarter about when to answer questions

Provide Web-based and mobile platforms

NA NA NA Added: release learning objectives (ie,

question topics) for the upcoming year to permit review Added: include questions that incorporate practice recommendations from recently published guidelines and emerging public health topics in the 2017?2018 pilot NA; because of differences in opinions among focus group and user group participants regarding quarterly versus twice yearly or yearly delivery of questions, the ABP chose to maintain quarterly delivery Modified: display enhanced

Downloaded from news by guest on October 6, 2021

4

LESLIE et al

TABLE 2 Continued

Primary Driver or Domain in Initial Model in 2015

Secondary Drivers or Components in Initial

Model in 2015

Change Strategies in Initial Model in 2015

Offer choice questions weighted to inpatient, outpatient, or mixed settings

All pediatricians would participate in MOCA-Peds

NA, not applicable.

Combine continuous certification requirements

Accommodate for life events

Address consequence of loss of credentials with new system

Not considered in the 2015 initial model

Not considered in the 2015 initial model Not considered in the 2015 initial model Not considered in the 2015 initial model

Reduce anxiety about changing to a new summative assessment system

Not considered in the 2015 initial model

Modifications or Additions to MOCA-Peds Based on Pediatrician Feedback in 2016

NA

Continue to offer the standard proctored examination as an alternative to MOCAPeds

Added: recommend a 5-y cycle to align with MOC cycle as opposed to a 3-y summative assessment cycle as originally envisioned

Added: provide MOC Part 2 credit Added: drop 4 lowest quarters once MOCA-

Peds goes live in 2019 Added: MOCA-Peds questions delivered

in years 1?4; year 5 available to take proctored examination if the test taker does not meet passing standard Added: provide learning objectives (also above) and provide orientation materials (eg, videos and sample questions)

General Pediatrics Content Outline, a publicly available and comprehensive list of medical knowledge topics that board-certified pediatricians are expected to know.22 Practice analysis, which consists of surveys with practitioners about their practice patterns combined with panel review, is widely recognized as the primary methodology for establishing a certification examination's content validity and relevance.23?26 Second, examination questions are written by practicing pediatricians, each of which must align with the content outline. Third, questions are chosen to represent the various categories identified during the practice analysis and covered within the content outline. Fourth, committees of pediatricians review each question for accuracy and relevance. Fifth, before scoring an individual's examination, the psychometric properties of each question are examined, and questions that perform poorly are dropped. Last, the ABP works with a separate panel of practicing pediatricians in a well-established process known as standard setting27 to set the passing standard, or cut score, that reflects

the minimum level of knowledge required for board certification.

Because continuous assessments have not been commonly used by certifying organizations, the ABP committed to using this same rigorous approach for MOCAPeds. In addition, several change strategies were theorized to improve the validity, reliability, and relevance of MOCA-Peds compared with the examination, including (1) implementation of a continuous assessment of the general pediatrician that keeps pace with rapid medical knowledge expansion (ie, enhanced validity), (2) administration of questions over the lifetime of a pediatrician's career (ie, increased reliability), and (3) provision of opportunity for pediatricians to give feedback on individual questions (ie, improved relevance).

High-stakes certification examinations also have been typically proctored in secure testing environments to help address the other identified components in this domain, which are to ensure answers to questions reflect an individual

pediatrician's knowledge and clinical judgment and to deter inappropriate test-taking behavior. Requiring a continuous assessment of the general pediatrician to be administered in a proctored environment, even if remote, would help address these concerns but would be difficult to implement. The ABP identified the following 2 change strategies designed to authenticate the identity of the test taker: (1) require personal authentication during log-in and (2) use a MOCA-Peds code of conduct. To prevent examination content from being shared publicly, which would invalidate MOCA-Peds as a summative assessment of general pediatric knowledge, the following 3 additional change strategies were identified: (1) delivery of questions in random order, (2) set time limits for each question, and (3) conduct ongoing data analyses to identify security threats.

Key Driver or Domain 2: Incorporate Opportunities for Learning

The second domain is focused on the following 2 components: providing tools for self-reflection on personal

Downloaded from news by guest on October 6, 2021

PEDIATRICS Volume 142, number 6, December 2018

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download