Fielding the CAHPS Clinician & Group Survey

CAHPS? Clinician & Group Survey and Instructions

Fielding the CAHPS? Clinician & Group Survey

Table of Contents

Introduction ............................................................................ 1 Sampling Guidelines............................................................... 2

Defining the Sample Frame: Eligibility Guidelines .............................................. 2 Recommended Number of Completes...................................................................5 Calculating the Starting Sample Size....................................................................7 Preparing Sample Files for Data Collection........................................................10

Recommended Data Collection Modes ............................ ... 11

Mail Protocol......................................................................................................12 Telephone Protocol.............................................................................................13 Email Protocol ....................................................................................................15

Tracking Returned Questionnaires...................................... 17 Calculating the Response Rate............................................ 18 Appendixes

Appendix A: Justification for Recommendations Regarding Number of Completed Questionnaires at the Provider Level ................................................ 20 Appendix B: Determining Whether a Survey Response Is Complete.................. 23 Appendix C: Methods for Increasing the Number of Responses ........................ 26

Fielding the CAHPS Clinician & Group Survey Document No. 2033 Updated 6/12/2017

CAHPS? Clinician & Group Survey and Instructions

Documents Available for the CAHPS Clinician & Group Survey 3.0

This document is part of a comprehensive set of instructional materials that address implementing the Clinician & Group Survey, analyzing the data, and reporting the results. All documents are available on the Agency for Healthcare Research and Quality's Web site. For assistance in accessing these documents, please contact the CAHPS Help Line at 800-492-9261 or cahps1@.

For descriptions of these documents, refer to What's Available for the CAHPS Clinician & Group Survey 3.0.

Questionnaires ? CAHPS Clinician & Group Survey:

Overview of the Questionnaires ? Clinician & Group Survey 3.0 (Adult and

Child, English and Spanish)

Supplemental Items ? Supplemental Items for the Clinician &

Group Survey 3.0

Some supplemental items for this survey are intended to be administered together. Learn more about these item sets:

? Patient-Centered Medical Home ? Patient Narrative Elicitation Protocol ? Health Literacy ? Health Information Technology

Available for all CAHPS surveys

? Analyzing CAHPS Survey Data: Free programs for analyzing the data, guidance on preparing survey results for analysis, and instructions for using the CAHPS Analysis Program.

? Translating Surveys and Other Materials: Guidelines for translating surveys and selecting translators and translation reviewers.

Survey Administration Guidelines

? Preparing a Questionnaire Using the CAHPS Clinician & Group Survey

? Fielding the CAHPS Clinician & Group Survey

? Sample Notification Letters and Emails for the CAHPS Clinician & Group Survey

? Sample Telephone Script for the CAHPS Clinician & Group Survey

Reporting Measures and Guidelines

? Patient Experience Measures from the CAHPS Clinician & Group Survey

Fielding the CAHPS Clinician & Group Survey Document No. 2033

CAHPS? Clinician & Group Survey and Instructions

Introduction

This document explains how to field the CAHPS Clinician & Group Survey and gather the data needed for analysis and reporting. It provides instructions and advice related to the following topics:

? Constructing the sampling frame. ? Choosing the sample. ? Maintaining confidentiality. ? Collecting the data. ? Tracking returned questionnaires. ? Calculating the response rate. These instructions apply to the Adult and Child versions of this survey. The CAHPS Clinician & Group Surveys can be used to assess care at three levels: the individual provider, the practice site/clinic, or the medical group. As illustrated in the graphic below, a practice site/clinic is based on a single geographic location. A medical group may contain multiple practice sites/clinics and is defined by a specific list of providers.

Figure 1. Three Levels of Sampling

Fielding the CAHPS Clinician & Group Survey Document No. 2033

Page 1

CAHPS? Clinician & Group Survey and Instructions

Figure 2. Summary of Key Recommendations for Administering a Clinician & Group Survey

Administration Sampling frame Collection mode Sample size

Target response rate

To generate the standardized data necessary for valid comparisons, the survey should be conducted by a third-party vendor according to the CAHPS guidelines specified in this document.

Age (Adult: 18 and over; Child: 17 and under) and had a visit with an individual provider, practice or medical group.

Mail, telephone, email (with mail or telephone), or mixed mode protocols

Varies, depending on whether sampling is done at the level of the individual provider, practice, or medical group. In general, to produce statistically valid comparisons, the sample needs to be large enough to yield 50 completed questionnaires per provider or 300 completed questionnaires per medical group. The recommended sample size when sampling at the practice site level depends on the number of providers at each site; see Figure 4 for specific recommendations.

40 percent

Sampling Guidelines

These sampling guidelines will help you understand who is eligible to be included in the sample frame for a CAHPS Clinician & Group Survey. They also explain how to select a sample. By following these guidelines, you can be confident that your results will be comparable to those produced by other vendors and survey sponsors (organizations that fund or oversee the administration of the survey).

All users of the Clinician & Group Survey can submit their survey results to AHRQ's CAHPS Database and obtain comparative data reports. Adherence to the administration guidelines is particularly important if you want to submit your results to the CAHPS Database.

Defining the Sample Frame: Eligibility Guidelines

The sample will be drawn from a list of individuals (adults age 18 and older, or children 17 and younger) who have received care from a given provider, practice site, or medical group during the specified time interval. The list is called a sample frame.

The best source of sample information for a given survey sponsor depends on which data source has the most accurate and complete data. Health plans or purchasers of care may have administrative or billing data to identify individual patients. In some instances, the data to identify individual patients may be found only in the records of medical practices. It may be necessary to pull data from two or more sources in order

Fielding the CAHPS Clinician & Group Survey Document No. 2033

Page 2

CAHPS? Clinician & Group Survey and Instructions

to have both up-to-date contact information and the ability to connect the visit to a specific provider.

Connecting health care received to a specific provider is necessary even if you are only interested in assessing patients' experiences with a practice site, clinic, or medical group. This information is used in the first question on the survey to define "this provider" for the respondent to assess. If the source of the sampling frame does not accurately identify the provider that the patient saw, you may want to oversample to account for errors. Such errors can occur if, for example, you use administrative billing data for the sampling frame and visits with physician assistants or nurse practitioners are billed under the supervisory physician.

Please review these guidelines for determining whom to include in your sample frame:

? Include only patients who have had at least one visit to the selected provider/practice in the target time frame. The target time frame, or lookback period, for the Clinician & Group Survey 3.0 is 6 months. This target time frame is intended to make the sample frame as inclusive as possible and to standardize data collection for comparisons of results.

? To determine the look-back period for your sampling frame, use the anticipated start date of data collection. For example, if you are using a lookback period of 6 months and your anticipated start date is September 1, 2015, include all those who have had at least one visit since March 1, 2015.

? The sampling frame is a person-level list rather than a visit-level list. Therefore, patients should appear only once in the sampling frame regardless of how many visits they have had in the look-back period. Use their most recent visit for inclusion in the sampling frame.

? If you are administering the Adult Survey, include all adults 18 years or older.

? If you are administering the Child Survey, include all children 17 years or younger.

? Draw the sample irrespective of reason for visit and duration of patientprovider relationship so that the full range of patients is represented.

? Include all patients who meet the sampling criteria even if they are no longer currently receiving care from the practice site/clinic or provider. Similarly, it is not required that the patient have the same insurance coverage the patient had at the time of the visit, even if the health plan is providing data for the sampling frame. Therefore, a patient who has disenrolled from the health plan can be included in the sampling frame as long as the patient meets the other sampling criteria.

? Allow the sample frame to include multiple individuals from the same household, but do not include more than one person (adult or child) per household in the final sample for which the survey will be administered. In

Fielding the CAHPS Clinician & Group Survey Document No. 2033

Page 3

CAHPS? Clinician & Group Survey and Instructions

other words, be sure to de-duplicate the sample to ensure that only one person per household receives a survey.

To ensure that results are comparable, do not target specific patient populations, such as patients with particular conditions or experiences, when you draw the sample. All CAHPS survey items have been designed for the general population. The survey includes appropriate screening items for questions that are designed to assess a specific experience. If you need to analyze data results by a specific patient population and do not expect to get a sufficient number of responses for that subgroup, then you can supplement your general population sample with a targeted oversample. But you must pull a general population sample first, before any oversampling is done.

Figure 3. Sample Frame Elements

The following information (data elements) should be included in the sample frame that a survey sponsor provides to the vendor.

Adult Survey

Child Survey

Unique patient ID

Unique patient ID

Name of person (first and last names in separate fields)

Name of child (first and last names in separate fields)

Date of birth

Date of birth of child

Parent or guardian's name

Gender

Gender of parent or guardian

Complete address (includes street address, city, state, and ZIP Code each in a separate field)

Complete address of parent or guardian (includes street address, city, state, and ZIP Code each in a separate field)

Telephone number with area code (if available)

Parent or guardian's telephone number with area code (if available)

Email address (if available)

Parent or guardian's email address (if available)

Indicate if Spanish-language materials are Indicate if Spanish-language materials are

required (if known)

required (if known)

Name and unique ID of medical group

Name and unique ID of medical group

Name and unique ID of provider seen

Name and unique ID of provider seen

Name and unique ID of site where patient was seen

Name and unique ID of site where patient was seen

Fielding the CAHPS Clinician & Group Survey Document No. 2033

Page 4

CAHPS? Clinician & Group Survey and Instructions

Adult Survey

Indicate if the physician is the patient's assigned primary care provider (optional)

Date of most recent visit

Child Survey

Indicate if the physician is the child's assigned primary care provider (optional)

Date of most recent visit

Recommended Number of Completes

The level of sampling determines how many completed questionnaires are required to obtain usable information. ("Questionnaires" are the survey instruments that have been mailed or are administered by telephone or online. Appendix B explains how to determine whether the returned questionnaire is "complete.") A minimum number of completed questionnaires are required for each level to ensure that the results are statistically reliable:

? Individual providers: 50 completed questionnaires per provider. At least 50 completed questionnaires per provider are recommended if the survey will be used to report or assess performance for individual providers.

? Practice site or clinic: Varies by the number of providers. For the practice site or clinic level, the recommended number of completed questionnaires is based on the number of providers at the site.

Figure 4. Number of Completed Questionnaires Required for Practice Sites or Clinics

Number of Providers

Number of Completed Questionnaires

1

50

2

100

3

150

4-9

175

10-13

200

14-19

250

20 or more

300

? Medical group: 300 completed questionnaires. For applications of the survey intended to report or assess performance for a larger entity, such as a multi-site medical group, with no interest in assessing individual physicians or practice sites, a minimum of 300 completed questionnaires per medical group is recommended. This recommendation is

Fielding the CAHPS Clinician & Group Survey Document No. 2033

Page 5

CAHPS? Clinician & Group Survey and Instructions

based on analyses conducted for the CAHPS Group Practice Survey that preceded the CAHPS Clinician & Group Survey;1 these analyses were confirmed with data from the CAHPS Database.

The recommendations regarding the number of completed questionnaires per provider, practice site, and group apply to the survey with core items only. If your survey includes supplemental items, which often apply to a relatively small subset of the overall sample, a higher number of completed questionnaires may be needed to generate enough responses to those items for the purposes of analysis and reporting. Generally speaking, to yield a level of reliability for supplemental items that is consistent with that of the core items, at least 100 responses per item are needed at the group level and 50 responses per item are needed at the individual physician level.

Reasoning behind the recommendations. These recommendations are based on studies of the number of completed questionnaires necessary to achieve adequate provider-level, practice-level, or group-level reliability for a measure. That is, how many completed questionnaires are needed to reliably distinguish among different units of measurement (such as individual providers or practices)? To answer this question, the CAHPS team examined data from multiple field trials as well as data from the CAHPS Database.

The unit-level reliability coefficient indicates the extent to which the patients within that unit (e.g., the patients of an individual provider, practice, or group) agree with one another in terms of their reported experiences.2 This coefficient can take any value from 0.0 to 1.0, where 1.0 signifies a measure for which every patient (e.g., all patients surveyed for a given practice) reports an identical experience. High levels of reliability are ideal, but achieving higher levels requires more completed questionnaires. To balance the goal of reliability with the need for a feasible sample size, the CAHPS team adopted the widely accepted coefficient of 0.70 as the threshold. In particular, a reliability level of at least 0.70 is strongly recommended for "high stakes" purposes such as public reporting or payment incentives, given the larger errors around estimated scores below this threshold.

The number of completed questionnaires per sampling unit that is required to achieve this threshold will vary across areas and markets. In more than a dozen field tests of the CAHPS Clinician & Group Survey, all testing sites achieved or surpassed 0.70 reliability for the core composite measures with 50 completed questionnaires per provider. (See Appendix A for the data used to develop recommendations for provider-level sampling).

In some markets and provider populations, it has been possible to achieve 0.70 reliability with fewer responses. For example, in one market, completed questionnaires from as few as 20 patients per provider were adequate to achieve 0.70

1 Solomon LS, Hays RD, Zaslavsky AM, Ding L, Cleary PD. Psychometric properties of a group-level Consumer Assessment of Health Plans Study (CAHPS) instrument. Med Care. 2005 Jan;43(1):53-60.

2 Compared to the amount that practices differ from one another.

Fielding the CAHPS Clinician & Group Survey Document No. 2033

Page 6

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download